From Thorsten.Kersting at itp1.uni-stuttgart.de Tue Jul 1 03:17:28 2014 From: Thorsten.Kersting at itp1.uni-stuttgart.de (Thorsten Kersting) Date: Tue, 01 Jul 2014 10:17:28 +0200 Subject: [petsc-users] error in mpiaij with unkown soruce Message-ID: <53B26E98.1090901@itp1.uni-stuttgart.de> Hello everyone! I want so solve eigenvalues of a matrix, which has a complicated structure, with SLEPC. I get the following error while setting up the EPS Problem. But I dont understand the error, and how to fix it. Does anyone have an idea? Or do you need more information (and which information exactly)? [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 643 /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] MatAssemblyBegin line 4791 /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/src/mat/interface/matrix.c [0]PETSC ERROR: [0] MatAXPY_BasicWithPreallocation line 98 /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/src/mat/utils/axpy.c [0]PETSC ERROR: [0] MatAXPY_MPIAIJ line 2304 /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] MatAXPY line 29 /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/src/mat/utils/axpy.c [0]PETSC ERROR: [0] STSetUp_Precond line 69 /home/st/st_st/st_ac113559/petsc/slepc-3.4.2/src/st/impls/precond/precond.c [0]PETSC ERROR: [0] STSetUp line 263 /home/st/st_st/st_ac113559/petsc/slepc-3.4.2/src/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSSetUp_XD line 104 /home/st/st_st/st_ac113559/petsc/slepc-3.4.2/src/eps/impls/davidson/common/davidson.c [0]PETSC ERROR: [0] EPSSetUp_GD line 116 /home/st/st_st/st_ac113559/petsc/slepc-3.4.2/src/eps/impls/davidson/gd/gd.c [0]PETSC ERROR: [0] EPSSetUp line 60 /home/st/st_st/st_ac113559/petsc/slepc-3.4.2/src/eps/interface/setup.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./imfmchfrjc on a arch-linux2-c-real named uc1n164 by st_ac113559 Mon Jun 30 10:11:30 2014 [0]PETSC ERROR: Libraries linked from /home/st/st_st/st_ac113559/petsc/petsc-3.4.2/arch-linux2-c-real/lib [0]PETSC ERROR: Configure run at Tue Mar 11 13:58:34 2014 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-real -with-scalar-type=real [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file From jed at jedbrown.org Tue Jul 1 03:43:35 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 01 Jul 2014 10:43:35 +0200 Subject: [petsc-users] error in mpiaij with unkown soruce In-Reply-To: <53B26E98.1090901@itp1.uni-stuttgart.de> References: <53B26E98.1090901@itp1.uni-stuttgart.de> Message-ID: <87pphpo4eg.fsf@jedbrown.org> Thorsten Kersting writes: > Hello everyone! > > I want so solve eigenvalues of a matrix, which has a complicated > structure, with SLEPC. I get the following error while setting up the > EPS Problem. But I dont understand the error, and how to fix it. Does > anyone have an idea? Or do you need more information (and which > information exactly)? Is the error reproducible and if so, does it happen at the same place? Is it possible that you are setting many off-process values? You might need to assemble sooner with MAT_FLUSH_ASSEMBLY so that the stash does not grow too large for whatever resource limits may have been placed on your job. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Thorsten.Kersting at itp1.uni-stuttgart.de Tue Jul 1 04:00:28 2014 From: Thorsten.Kersting at itp1.uni-stuttgart.de (Thorsten Kersting) Date: Tue, 01 Jul 2014 11:00:28 +0200 Subject: [petsc-users] error in mpiaij with unkown soruce In-Reply-To: <87pphpo4eg.fsf@jedbrown.org> References: <53B26E98.1090901@itp1.uni-stuttgart.de> <87pphpo4eg.fsf@jedbrown.org> Message-ID: <53B278AC.20104@itp1.uni-stuttgart.de> Hello! thanks for the quick answer. Yes the error is reproducible and happens at the same place. I only set values on those processes, on which they are stored (I calculate them directly before setting them). And with every matrix I do this only once, so if i understand correctly, MAT_FINAL_ASSAMEBLY is sufficient. but i can try and see if it is different with MAT_FLUSH_ASSEMBLY. On 07/01/2014 10:43 AM, Jed Brown wrote: > Thorsten Kersting writes: > >> Hello everyone! >> >> I want so solve eigenvalues of a matrix, which has a complicated >> structure, with SLEPC. I get the following error while setting up the >> EPS Problem. But I dont understand the error, and how to fix it. Does >> anyone have an idea? Or do you need more information (and which >> information exactly)? > Is the error reproducible and if so, does it happen at the same place? > > Is it possible that you are setting many off-process values? You might > need to assemble sooner with MAT_FLUSH_ASSEMBLY so that the stash does > not grow too large for whatever resource limits may have been placed on > your job. From jed at jedbrown.org Tue Jul 1 04:08:09 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 01 Jul 2014 11:08:09 +0200 Subject: [petsc-users] error in mpiaij with unkown soruce In-Reply-To: <53B278AC.20104@itp1.uni-stuttgart.de> References: <53B26E98.1090901@itp1.uni-stuttgart.de> <87pphpo4eg.fsf@jedbrown.org> <53B278AC.20104@itp1.uni-stuttgart.de> Message-ID: <87mwcto39i.fsf@jedbrown.org> Thorsten Kersting writes: > Yes the error is reproducible and happens at the same place. > > I only set values on those processes, on which they are stored (I > calculate them directly before setting them). And with every matrix I do > this only once, so if i understand correctly, MAT_FINAL_ASSAMEBLY is > sufficient. but i can try and see if it is different with > MAT_FLUSH_ASSEMBLY. Sounds like a system problem. Can you try a smaller problem size and run in a debugger? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Tue Jul 1 04:38:17 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Jul 2014 11:38:17 +0200 Subject: [petsc-users] error in mpiaij with unkown soruce In-Reply-To: <53B278AC.20104@itp1.uni-stuttgart.de> References: <53B26E98.1090901@itp1.uni-stuttgart.de> <87pphpo4eg.fsf@jedbrown.org> <53B278AC.20104@itp1.uni-stuttgart.de> Message-ID: On Tue, Jul 1, 2014 at 11:00 AM, Thorsten Kersting < Thorsten.Kersting at itp1.uni-stuttgart.de> wrote: > Hello! > > thanks for the quick answer. > > Yes the error is reproducible and happens at the same place. > > I only set values on those processes, on which they are stored (I > calculate them directly before setting them). And with every matrix I do > this only once, so if i understand correctly, MAT_FINAL_ASSAMEBLY is > sufficient. but i can try and see if it is different with > MAT_FLUSH_ASSEMBLY. > I recommend using valgrind on this. Its possible that you have overwritten memory. Thanks, Matt > On 07/01/2014 10:43 AM, Jed Brown wrote: > >> Thorsten Kersting writes: >> >> Hello everyone! >>> >>> I want so solve eigenvalues of a matrix, which has a complicated >>> structure, with SLEPC. I get the following error while setting up the >>> EPS Problem. But I dont understand the error, and how to fix it. Does >>> anyone have an idea? Or do you need more information (and which >>> information exactly)? >>> >> Is the error reproducible and if so, does it happen at the same place? >> >> Is it possible that you are setting many off-process values? You might >> need to assemble sooner with MAT_FLUSH_ASSEMBLY so that the stash does >> not grow too large for whatever resource limits may have been placed on >> your job. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From akurlej at gmail.com Tue Jul 1 12:04:12 2014 From: akurlej at gmail.com (Arthur Kurlej) Date: Tue, 1 Jul 2014 12:04:12 -0500 Subject: [petsc-users] VecScatter Error w/Multiple Processors Message-ID: Hello, I am trying to use VecScatter to grab values from nonlocal processors, and I've run into the following error when I try to run for 8+ processors. The error is as follows: [0]PETSC ERROR: VecScatterCreate() line 1161 in src/vec/vec/utils/vscat.c And the code that causes it is Mat A MatGetOwnershipRange(A,&Istart,&Iend); for(rowcounter = Istart; rowcounter < Iend; rowcounter++){ ISCreateGeneral(PETSC_COMM_WORLD,lSize,vecIndices,PETSC_COPY_VALUES,&isGlobal); VecScatterCreate(gVec,isGlobal,lVec, NULL,&ctx); VecScatterBegin(ctx,gVec,lVec,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(ctx,gVec,lVec,INSERT_VALUES,SCATTER_FORWARD); //fill matrix } Any thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jul 1 12:55:31 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Jul 2014 12:55:31 -0500 Subject: [petsc-users] VecScatter Error w/Multiple Processors In-Reply-To: References: Message-ID: <112605A3-E7ED-4A73-9280-C859FCEBF1DD@mcs.anl.gov> There are several problems with this code. On Jul 1, 2014, at 12:04 PM, Arthur Kurlej wrote: > Hello, > > I am trying to use VecScatter to grab values from nonlocal processors, and I've run into the following error when I try to run for 8+ processors. The error is as follows: > > [0]PETSC ERROR: VecScatterCreate() line 1161 in src/vec/vec/utils/vscat.c > > And the code that causes it is > > Mat A > MatGetOwnershipRange(A,&Istart,&Iend); > > for(rowcounter = Istart; rowcounter < Iend; rowcounter++){ > > ISCreateGeneral(PETSC_COMM_WORLD,lSize,vecIndices,PETSC_COPY_VALUES,&isGlobal); ISCreateGeneral(PETSC_COMM_WORLD) is a collective operation, all processes in PETSC_COMM_WORLD must call this each time. But you have this inside a loop for the number of local rows; not all processes will have the same number of local rows > VecScatterCreate(gVec,isGlobal,lVec, NULL,&ctx); > VecScatterBegin(ctx,gVec,lVec,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,gVec,lVec,INSERT_VALUES,SCATTER_FORWARD); > > //fill matrix > } I am not sure what you want to accomplish but there is no way that doing a VecScatter for every row in a matrix is going to be scalable. are you trying to move values to other processes that you then plan to set into the matrix locally? If so just use MatSetValues() where each process sets the values it currently has. > > Any thoughts? > > From quecat001 at gmail.com Wed Jul 2 15:21:00 2014 From: quecat001 at gmail.com (Que Cat) Date: Wed, 2 Jul 2014 15:21:00 -0500 Subject: [petsc-users] Writing Vector to HDF5 file - Valgrind debug Message-ID: Dear Petsc-User, I used the following comment to write to a HDF5 file: PetscViewer H5viewer; PetscViewerHDF5Open(..., & H5viewer) PetscObjectSetName( (PetscObject) vector, Name); VecView(vector,H5viewer); PetscViewerDestroy(&H5viewer); As I debug using valgrind, I received the following error: ==29820== Use of uninitialised value of size 8 ==29820== at 0x828B87B: H5D__chunk_lookup (H5Dchunk.c:2274) ==29820== by 0x82ACEA3: H5D__link_chunk_collective_io (H5Dmpio.c:856) ==29820== by 0x82AC858: H5D__chunk_collective_io (H5Dmpio.c:696) ==29820== by 0x82ACC8B: H5D__chunk_collective_write (H5Dmpio.c:781) ==29820== by 0x82A8C47: H5D__write (H5Dio.c:672) ==29820== by 0x82A72E9: H5Dwrite (H5Dio.c:265) ==29820== by 0x53E6393: VecView_MPI_HDF5 (pdvec.c:786) ==29820== by 0x53E6E5A: VecView_MPI (pdvec.c:851) ==29820== by 0x5398A7C: VecView (vector.c:717) I have checked that my vector used in VecView(vector,H5viewer) was initialized before calling VecView(vector,H5viewer). Check the output hdf5 file, all the data was recorded correctly. Should I worry about the error detect by valgrind? Thanks. Que -------------- next part -------------- An HTML attachment was scrubbed... URL: From jifengzhao2015 at u.northwestern.edu Wed Jul 2 16:46:13 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Wed, 2 Jul 2014 16:46:13 -0500 Subject: [petsc-users] Preconditioners for a FEM model Message-ID: Hello all, I am working on solving a generalized eigenvalue problem with SLEPC and PETSC. *K* x = lamda *M* x I attached the sparsity pattern of matrix *M* (*K* is the same). It is a FEM model. It is so sparse is because of constraints. I have tried two things: 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in future). It works. But I am worrying that it is less parrallelable, when the problem contains millions of degree of freedom. 2. JD with Jacobi preconditioner. It could work, but a lot of tuning needs to be done in terms of RTOL, max_iteration_number. And sometimes I suffer from a stagnated solution, and can't obtain accurate result. Does anybody know that for my specific case of matrix sparsity, is there any thing I can do to speed up my direct solver (Krylov-Schur)? Is there any recommended preconditioners I could try on, for the case of JD? There are a lot of preconditioners in HYPRE library. Thank you in advance! [image: Inline image 1] -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From keceli at gmail.com Wed Jul 2 21:41:53 2014 From: keceli at gmail.com (=?UTF-8?Q?murat_ke=C3=A7eli?=) Date: Wed, 2 Jul 2014 21:41:53 -0500 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: References: Message-ID: Hi Jifeng, I think your application is suitable for the SIPs method, see attached paper. We have improved it recently, so that it can handle very large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current version of SIPs is basically adding a second layer of parallelism on top of SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of SIPs) know, if you would like to give it a try. Murat Keceli On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < jifengzhao2015 at u.northwestern.edu> wrote: > Hello all, > > I am working on solving a generalized eigenvalue problem with SLEPC and > PETSC. > > *K* x = lamda *M* x > > I attached the sparsity pattern of matrix *M* (*K* is the same). It is a > FEM model. It is so sparse is because of constraints. > > I have tried two things: > > 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in future). > It works. But I am worrying that it is less parrallelable, when the problem > contains millions of degree of freedom. > > 2. JD with Jacobi preconditioner. It could work, but a lot of tuning needs > to be done in terms of RTOL, max_iteration_number. And sometimes I suffer > from a stagnated solution, and can't obtain accurate result. > > Does anybody know that for my specific case of matrix sparsity, is there > any thing I can do to speed up my direct solver (Krylov-Schur)? > > Is there any recommended preconditioners I could try on, for the case of > JD? There are a lot of preconditioners in HYPRE library. > > Thank you in advance! > [image: Inline image 1] > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Zhang et al._2007.pdf Type: application/pdf Size: 523507 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Jul 2 21:42:45 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Jul 2014 21:42:45 -0500 Subject: [petsc-users] Writing Vector to HDF5 file - Valgrind debug In-Reply-To: References: Message-ID: Very likely this is an unimportant issue within the HDF5 code that you don?t need to worry about. We?ve found that many of the external packages we use in PETSc have places where they access uninitialized data; you might report it directly to the HDF5 team. Barry On Jul 2, 2014, at 3:21 PM, Que Cat wrote: > Dear Petsc-User, > > I used the following comment to write to a HDF5 file: > > PetscViewer H5viewer; > PetscViewerHDF5Open(..., & H5viewer) > PetscObjectSetName( (PetscObject) vector, Name); > VecView(vector,H5viewer); > PetscViewerDestroy(&H5viewer); > > As I debug using valgrind, I received the following error: > > ==29820== Use of uninitialised value of size 8 > ==29820== at 0x828B87B: H5D__chunk_lookup (H5Dchunk.c:2274) > ==29820== by 0x82ACEA3: H5D__link_chunk_collective_io (H5Dmpio.c:856) > ==29820== by 0x82AC858: H5D__chunk_collective_io (H5Dmpio.c:696) > ==29820== by 0x82ACC8B: H5D__chunk_collective_write (H5Dmpio.c:781) > ==29820== by 0x82A8C47: H5D__write (H5Dio.c:672) > ==29820== by 0x82A72E9: H5Dwrite (H5Dio.c:265) > ==29820== by 0x53E6393: VecView_MPI_HDF5 (pdvec.c:786) > ==29820== by 0x53E6E5A: VecView_MPI (pdvec.c:851) > ==29820== by 0x5398A7C: VecView (vector.c:717) > > I have checked that my vector used in VecView(vector,H5viewer) was initialized before calling VecView(vector,H5viewer). Check the output hdf5 file, all the data was recorded correctly. Should I worry about the error detect by valgrind? Thanks. > > Que From venkateshgk.j at gmail.com Wed Jul 2 22:50:33 2014 From: venkateshgk.j at gmail.com (venkatesh g) Date: Thu, 3 Jul 2014 09:20:33 +0530 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: References: Message-ID: Hi Murat, Is it possible to do a non-symmetric complex eigenvalue problem with this approach ? Best, Venkatesh On Thu, Jul 3, 2014 at 8:11 AM, murat ke?eli wrote: > Hi Jifeng, > > I think your application is suitable for the SIPs method, see attached > paper. We have improved it recently, so that it can handle very > large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current > version of SIPs is basically adding a second layer of parallelism on top of > SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of > SIPs) know, if you would like to give it a try. > > Murat Keceli > > > On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > >> Hello all, >> >> I am working on solving a generalized eigenvalue problem with SLEPC and >> PETSC. >> >> *K* x = lamda *M* x >> >> I attached the sparsity pattern of matrix *M* (*K* is the same). It is a >> FEM model. It is so sparse is because of constraints. >> >> I have tried two things: >> >> 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in future). >> It works. But I am worrying that it is less parrallelable, when the problem >> contains millions of degree of freedom. >> >> 2. JD with Jacobi preconditioner. It could work, but a lot of tuning >> needs to be done in terms of RTOL, max_iteration_number. And sometimes I >> suffer from a stagnated solution, and can't obtain accurate result. >> >> Does anybody know that for my specific case of matrix sparsity, is there >> any thing I can do to speed up my direct solver (Krylov-Schur)? >> >> Is there any recommended preconditioners I could try on, for the case of >> JD? There are a lot of preconditioners in HYPRE library. >> >> Thank you in advance! >> [image: Inline image 1] >> -- >> Jifeng Zhao >> PhD candidate at Northwestern University, US >> Theoretical and Applied Mechanics Program >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From jifengzhao2015 at u.northwestern.edu Thu Jul 3 00:18:26 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Thu, 3 Jul 2014 00:18:26 -0500 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: References: Message-ID: Hi Murat, Yes, that sounds great. I would like to have a try. Would you let me know how to use it on top of SLEPC and PETSC in more details? Cheers, Jifeng Zhao On Wed, Jul 2, 2014 at 9:41 PM, murat ke?eli wrote: > Hi Jifeng, > > I think your application is suitable for the SIPs method, see attached > paper. We have improved it recently, so that it can handle very > large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current > version of SIPs is basically adding a second layer of parallelism on top of > SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of > SIPs) know, if you would like to give it a try. > > Murat Keceli > > > On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > >> Hello all, >> >> I am working on solving a generalized eigenvalue problem with SLEPC and >> PETSC. >> >> *K* x = lamda *M* x >> >> I attached the sparsity pattern of matrix *M* (*K* is the same). It is a >> FEM model. It is so sparse is because of constraints. >> >> I have tried two things: >> >> 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in future). >> It works. But I am worrying that it is less parrallelable, when the problem >> contains millions of degree of freedom. >> >> 2. JD with Jacobi preconditioner. It could work, but a lot of tuning >> needs to be done in terms of RTOL, max_iteration_number. And sometimes I >> suffer from a stagnated solution, and can't obtain accurate result. >> >> Does anybody know that for my specific case of matrix sparsity, is there >> any thing I can do to speed up my direct solver (Krylov-Schur)? >> >> Is there any recommended preconditioners I could try on, for the case of >> JD? There are a lot of preconditioners in HYPRE library. >> >> Thank you in advance! >> [image: Inline image 1] >> -- >> Jifeng Zhao >> PhD candidate at Northwestern University, US >> Theoretical and Applied Mechanics Program >> > > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From hzhang at mcs.anl.gov Thu Jul 3 10:27:31 2014 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 3 Jul 2014 10:27:31 -0500 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: <7e6dad6a00a643168cf6746d1ef6edc8@GEORGE.anl.gov> References: <7e6dad6a00a643168cf6746d1ef6edc8@GEORGE.anl.gov> Message-ID: venkatesh: > > Is it possible to do a non-symmetric complex eigenvalue problem with this > approach ? > Unfortunately, no. Hong > > > > On Thu, Jul 3, 2014 at 8:11 AM, murat ke?eli wrote: > >> Hi Jifeng, >> >> I think your application is suitable for the SIPs method, see attached >> paper. We have improved it recently, so that it can handle very >> large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current >> version of SIPs is basically adding a second layer of parallelism on top of >> SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of >> SIPs) know, if you would like to give it a try. >> >> Murat Keceli >> >> >> On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < >> jifengzhao2015 at u.northwestern.edu> wrote: >> >>> Hello all, >>> >>> I am working on solving a generalized eigenvalue problem with SLEPC >>> and PETSC. >>> >>> *K* x = lamda *M* x >>> >>> I attached the sparsity pattern of matrix *M* (*K* is the same). It is >>> a FEM model. It is so sparse is because of constraints. >>> >>> I have tried two things: >>> >>> 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in >>> future). It works. But I am worrying that it is less parrallelable, when >>> the problem contains millions of degree of freedom. >>> >>> 2. JD with Jacobi preconditioner. It could work, but a lot of tuning >>> needs to be done in terms of RTOL, max_iteration_number. And sometimes I >>> suffer from a stagnated solution, and can't obtain accurate result. >>> >>> Does anybody know that for my specific case of matrix sparsity, is >>> there any thing I can do to speed up my direct solver (Krylov-Schur)? >>> >>> Is there any recommended preconditioners I could try on, for the case >>> of JD? There are a lot of preconditioners in HYPRE library. >>> >>> Thank you in advance! >>> [image: Inline image 1] >>> -- >>> Jifeng Zhao >>> PhD candidate at Northwestern University, US >>> Theoretical and Applied Mechanics Program >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From keceli at gmail.com Thu Jul 3 12:13:14 2014 From: keceli at gmail.com (=?UTF-8?Q?murat_ke=C3=A7eli?=) Date: Thu, 3 Jul 2014 12:13:14 -0500 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: References: Message-ID: Hi Jifeng, You can start with ex13 in SLEPc to see how it works for your case. You would need MUMPS as a direct solver. Section 3.4.5 is the relevant section in SLEPc manual (Campos and Roman 2012 paper would be very helpful if you need more details). Let me know if it works for you and we can continue from there. Thanks, Murat On Thu, Jul 3, 2014 at 12:18 AM, jifeng zhao < jifengzhao2015 at u.northwestern.edu> wrote: > Hi Murat, > > Yes, that sounds great. I would like to have a try. Would you let me know > how to use it on top of SLEPC and PETSC in more details? > > Cheers, > Jifeng Zhao > > > On Wed, Jul 2, 2014 at 9:41 PM, murat ke?eli wrote: > >> Hi Jifeng, >> >> I think your application is suitable for the SIPs method, see attached >> paper. We have improved it recently, so that it can handle very >> large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current >> version of SIPs is basically adding a second layer of parallelism on top of >> SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of >> SIPs) know, if you would like to give it a try. >> >> Murat Keceli >> >> >> On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < >> jifengzhao2015 at u.northwestern.edu> wrote: >> >>> Hello all, >>> >>> I am working on solving a generalized eigenvalue problem with SLEPC and >>> PETSC. >>> >>> *K* x = lamda *M* x >>> >>> I attached the sparsity pattern of matrix *M* (*K* is the same). It is >>> a FEM model. It is so sparse is because of constraints. >>> >>> I have tried two things: >>> >>> 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in >>> future). It works. But I am worrying that it is less parrallelable, when >>> the problem contains millions of degree of freedom. >>> >>> 2. JD with Jacobi preconditioner. It could work, but a lot of tuning >>> needs to be done in terms of RTOL, max_iteration_number. And sometimes I >>> suffer from a stagnated solution, and can't obtain accurate result. >>> >>> Does anybody know that for my specific case of matrix sparsity, is there >>> any thing I can do to speed up my direct solver (Krylov-Schur)? >>> >>> Is there any recommended preconditioners I could try on, for the case of >>> JD? There are a lot of preconditioners in HYPRE library. >>> >>> Thank you in advance! >>> [image: Inline image 1] >>> -- >>> Jifeng Zhao >>> PhD candidate at Northwestern University, US >>> Theoretical and Applied Mechanics Program >>> >> >> > > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From jifengzhao2015 at u.northwestern.edu Thu Jul 3 21:45:32 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Thu, 3 Jul 2014 21:45:32 -0500 Subject: [petsc-users] fill factor in factorization SLEPC Message-ID: Hi all, I am using SLEPC to solve an eigenvalue problem. I tried to add -info at run time. And I get instruction: MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 3.87526 or use MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,3.87526); MatLUFactorSymbolic_SeqAIJ(): for best performance. However, when I tried to add -*st_pc_factor_fill* 3.87526 at run time. It is not recognized at all! WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-st_pc_factor_fill value: 3.87526 Same things happened if I tried *pc_factor_fill* 3.87526 Does anybody know how to change the fill of factorization for SLEPC? Thanks, Jifeng Zhao -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jul 3 22:34:39 2014 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 3 Jul 2014 22:34:39 -0500 Subject: [petsc-users] fill factor in factorization SLEPC In-Reply-To: References: Message-ID: You may try -st_redundant_pc_factor_fill . Here is how I found the option using slepc-3.4.4/src/eps/examples/tutorials/ex13.c: 1) ./ex13 -info | |grep fill [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 2.95217 or use 2) ./ex13 -help |grep fill -st_redundant_pc_factor_fill <5>: Expected non-zeros in factored matrix (PCFactorSetFill) -st_redundant_pc_factor_reuse_fill: Use fill from previous factorization (PCFactorSetReuseFill) i.e., the default fill=5, sufficient for this case. There is no need to set as 2.95217. If you want to, do ./ex13 -st_redundant_pc_factor_fill 2.95217 -info ... [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 4 needed 2.95217 [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 2.95217 or use [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,2.95217); As you see, Reallocs 0, good performance. Hong On Thu, Jul 3, 2014 at 9:45 PM, jifeng zhao wrote: > Hi all, > > I am using SLEPC to solve an eigenvalue problem. > > I tried to add -info at run time. And I get instruction: > > MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 3.87526 or use > MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,3.87526); > MatLUFactorSymbolic_SeqAIJ(): for best performance. > > However, when I tried to add -st_pc_factor_fill 3.87526 at run time. It is > not recognized at all! > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-st_pc_factor_fill value: 3.87526 > > Same things happened if I tried pc_factor_fill 3.87526 > > Does anybody know how to change the fill of factorization for SLEPC? > > Thanks, > Jifeng Zhao > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program From jifengzhao2015 at u.northwestern.edu Fri Jul 4 10:11:59 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Fri, 4 Jul 2014 10:11:59 -0500 Subject: [petsc-users] Preconditioners for a FEM model In-Reply-To: References: Message-ID: Yes ex13 works for me. Could you let me know where can I download the program and how can I add options at the runtime? Thank you so much. Jifeng On Thu, Jul 3, 2014 at 12:13 PM, murat ke?eli wrote: > Hi Jifeng, > > You can start with ex13 in SLEPc to see how it works for your case. You > would need MUMPS as a direct solver. Section 3.4.5 is the relevant section > in SLEPc manual (Campos and Roman 2012 paper would be very helpful if you > need more details). Let me know if it works for you and we can continue > from there. > > Thanks, > Murat > > > On Thu, Jul 3, 2014 at 12:18 AM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > >> Hi Murat, >> >> Yes, that sounds great. I would like to have a try. Would you let me know >> how to use it on top of SLEPC and PETSC in more details? >> >> Cheers, >> Jifeng Zhao >> >> >> On Wed, Jul 2, 2014 at 9:41 PM, murat ke?eli wrote: >> >>> Hi Jifeng, >>> >>> I think your application is suitable for the SIPs method, see attached >>> paper. We have improved it recently, so that it can handle very >>> large (500k by 500k with more than 3.e7 nonzeros) sparse matrices.Current >>> version of SIPs is basically adding a second layer of parallelism on top of >>> SLEPc's shift and invert method. Let me or Hong Zhang (cc, the developer of >>> SIPs) know, if you would like to give it a try. >>> >>> Murat Keceli >>> >>> >>> On Wed, Jul 2, 2014 at 4:46 PM, jifeng zhao < >>> jifengzhao2015 at u.northwestern.edu> wrote: >>> >>>> Hello all, >>>> >>>> I am working on solving a generalized eigenvalue problem with SLEPC and >>>> PETSC. >>>> >>>> *K* x = lamda *M* x >>>> >>>> I attached the sparsity pattern of matrix *M* (*K* is the same). It is >>>> a FEM model. It is so sparse is because of constraints. >>>> >>>> I have tried two things: >>>> >>>> 1. Krylov-Schur and exact shift-and-invert (I will try MUMPS in >>>> future). It works. But I am worrying that it is less parrallelable, when >>>> the problem contains millions of degree of freedom. >>>> >>>> 2. JD with Jacobi preconditioner. It could work, but a lot of tuning >>>> needs to be done in terms of RTOL, max_iteration_number. And sometimes I >>>> suffer from a stagnated solution, and can't obtain accurate result. >>>> >>>> Does anybody know that for my specific case of matrix sparsity, is >>>> there any thing I can do to speed up my direct solver (Krylov-Schur)? >>>> >>>> Is there any recommended preconditioners I could try on, for the case >>>> of JD? There are a lot of preconditioners in HYPRE library. >>>> >>>> Thank you in advance! >>>> [image: Inline image 1] >>>> -- >>>> Jifeng Zhao >>>> PhD candidate at Northwestern University, US >>>> Theoretical and Applied Mechanics Program >>>> >>> >>> >> >> >> -- >> Jifeng Zhao >> PhD candidate at Northwestern University, US >> Theoretical and Applied Mechanics Program >> > > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MassFullwithdiag.png Type: image/png Size: 10077 bytes Desc: not available URL: From jifengzhao2015 at u.northwestern.edu Fri Jul 4 10:13:28 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Fri, 4 Jul 2014 10:13:28 -0500 Subject: [petsc-users] fill factor in factorization SLEPC In-Reply-To: References: Message-ID: Hi Hong, Thanks a lot. -st_redundant_pc_factor_fill does work for me. And thank you for presenting how you found the solution. It is helpful. Cheers, Jifeng On Thu, Jul 3, 2014 at 10:34 PM, Hong Zhang wrote: > You may try -st_redundant_pc_factor_fill . > Here is how I found the option using > slepc-3.4.4/src/eps/examples/tutorials/ex13.c: > 1) > ./ex13 -info | |grep fill > [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 2.95217 or use > > 2) > ./ex13 -help |grep fill > -st_redundant_pc_factor_fill <5>: Expected non-zeros in factored > matrix (PCFactorSetFill) > -st_redundant_pc_factor_reuse_fill: Use fill from previous > factorization (PCFactorSetReuseFill) > > i.e., the default fill=5, sufficient for this case. There is no need > to set as 2.95217. If you want to, do > ./ex13 -st_redundant_pc_factor_fill 2.95217 -info > ... > [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 4 needed > 2.95217 > [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 2.95217 or use > [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,2.95217); > > As you see, Reallocs 0, good performance. > > Hong > > On Thu, Jul 3, 2014 at 9:45 PM, jifeng zhao > wrote: > > Hi all, > > > > I am using SLEPC to solve an eigenvalue problem. > > > > I tried to add -info at run time. And I get instruction: > > > > MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 3.87526 or use > > MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,3.87526); > > MatLUFactorSymbolic_SeqAIJ(): for best performance. > > > > However, when I tried to add -st_pc_factor_fill 3.87526 at run time. It > is > > not recognized at all! > > > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-st_pc_factor_fill value: 3.87526 > > > > Same things happened if I tried pc_factor_fill 3.87526 > > > > Does anybody know how to change the fill of factorization for SLEPC? > > > > Thanks, > > Jifeng Zhao > > > > -- > > Jifeng Zhao > > PhD candidate at Northwestern University, US > > Theoretical and Applied Mechanics Program > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Sat Jul 5 03:09:53 2014 From: fd.kong at siat.ac.cn (fd.kong at siat.ac.cn) Date: Sat, 5 Jul 2014 16:09:53 +0800 (GMT+08:00) Subject: [petsc-users] can not configure pets-3.5.0 on mac 10.9.2 Message-ID: <1e56ec9.10185.14705928240.Coremail.fd.kong@siat.ac.cn> Hi all, I can not configure pets-3.5.0 on mac with options: --with-clanguage=cxx --with-shared-libraries=1 --download-fblaslapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-hdf5=1 --with-mpi-dir=/Users/livia/math/mpich-3.1_instal The configure log file is attached. Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configurelog.zip Type: application/x-zip-compressed Size: 367134 bytes Desc: not available URL: From balay at mcs.anl.gov Sat Jul 5 09:04:00 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 5 Jul 2014 09:04:00 -0500 Subject: [petsc-users] can not configure pets-3.5.0 on mac 10.9.2 In-Reply-To: <1e56ec9.10185.14705928240.Coremail.fd.kong@siat.ac.cn> References: <1e56ec9.10185.14705928240.Coremail.fd.kong@siat.ac.cn> Message-ID: Looks like the error is in --download-fblaslapack=1 [we'll have to check on this]. You should not need --download-fblaslapack=1 on Mac - system blas/lapack should work. Some of the externalpackages don't support --with-64-bit-indices option. Satish On Sat, 5 Jul 2014, fd.kong at siat.ac.cn wrote: > Hi all, > > > I can not configure pets-3.5.0 on mac with options: --with-clanguage=cxx --with-shared-libraries=1 --download-fblaslapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-hdf5=1 --with-mpi-dir=/Users/livia/math/mpich-3.1_instal > > > The configure log file is attached. > > > Fande, > > > From balay at mcs.anl.gov Sat Jul 5 09:07:00 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 5 Jul 2014 09:07:00 -0500 Subject: [petsc-users] can not configure pets-3.5.0 on mac 10.9.2 In-Reply-To: References: <1e56ec9.10185.14705928240.Coremail.fd.kong@siat.ac.cn> Message-ID: >>>> f951: internal compiler error: Segmentation fault: 11 Please submit a full bug report, with preprocessed source if appropriate. See for instructions. <<<<< $ grep 'f951: internal compiler error' configure.log f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 f951: internal compiler error: Segmentation fault: 11 Perhaps you need a different gfortran compiler. Satish On Sat, 5 Jul 2014, Satish Balay wrote: > Looks like the error is in --download-fblaslapack=1 [we'll have to > check on this]. > > You should not need --download-fblaslapack=1 on Mac - system > blas/lapack should work. > > Some of the externalpackages don't support --with-64-bit-indices > option. > > Satish > > On Sat, 5 Jul 2014, fd.kong at siat.ac.cn wrote: > > > Hi all, > > > > > > I can not configure pets-3.5.0 on mac with options: --with-clanguage=cxx --with-shared-libraries=1 --download-fblaslapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-hdf5=1 --with-mpi-dir=/Users/livia/math/mpich-3.1_instal > > > > > > The configure log file is attached. > > > > > > Fande, > > > > > > > > From tlk0812 at hotmail.com Sat Jul 5 18:18:45 2014 From: tlk0812 at hotmail.com (LikunTan) Date: Sun, 6 Jul 2014 07:18:45 +0800 Subject: [petsc-users] FFTW convolution in Petsc Message-ID: Hi all, I am testing /petsc-3.4.3/src/mat/examples/tests/ex121.c for FFTW convolution in Petsc. I simplified the example code by considering the tangent function of the real space vector only, and got the following error when executing it: [0]PETSC ERROR: --------------------- Error Message ------------------------------------[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type:see http://www.mcs.anl.gov/petsc/documentation/installation.html#external![0]PETSC ERROR: Unknown Mat type given: fftw![0]PETSC ERROR: ------------------------------------------------------------------------[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates.[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.[0]PETSC ERROR: See docs/index.html for manual pages.[0]PETSC ERROR: ------------------------------------------------------------------------[0]PETSC ERROR: ./ex121 on a arch-linux2-c-debug named garuda.ce.caltech.edu by likunt Sat Jul 5 16:07:13 2014[0]PETSC ERROR: Libraries linked from /home/likunt/petsc-3.4.3/arch-linux2-c-debug/lib[0]PETSC ERROR: Configure run at Sat Jul 5 13:01:11 2014[0]PETSC ERROR: Configure options --with-scale-type=complex[0]PETSC ERROR: ------------------------------------------------------------------------[0]PETSC ERROR: MatSetType() line 63 in src/mat/interface/matreg.c[0]PETSC ERROR: MatCreateFFT() line 76 in src/mat/impls/fft/fft.c[0]PETSC ERROR: main() line 59 in src/mat/examples/tests/ex121.c The error message shows lack of FFTW, but I am not sure how to include FFTW in the external package. Your help is well appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jul 5 18:20:10 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 6 Jul 2014 01:20:10 +0200 Subject: [petsc-users] FFTW convolution in Petsc In-Reply-To: References: Message-ID: On Sun, Jul 6, 2014 at 1:18 AM, LikunTan wrote: > Hi all, > > I am testing /petsc-3.4.3/src/mat/examples/tests/ex121.c for FFTW > convolution in Petsc. > You need to configure using --download-fftw Matt > I simplified the example code by considering the tangent function of the > real space vector only, and got the following error when executing it: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type: > see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > [0]PETSC ERROR: Unknown Mat type given: fftw! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex121 on a arch-linux2-c-debug named > garuda.ce.caltech.edu by likunt Sat Jul 5 16:07:13 2014 > [0]PETSC ERROR: Libraries linked from > /home/likunt/petsc-3.4.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Sat Jul 5 13:01:11 2014 > [0]PETSC ERROR: Configure options --with-scale-type=complex > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetType() line 63 in src/mat/interface/matreg.c > [0]PETSC ERROR: MatCreateFFT() line 76 in src/mat/impls/fft/fft.c > [0]PETSC ERROR: main() line 59 in src/mat/examples/tests/ex121.c > > The error message shows lack of FFTW, but I am not sure how to include > FFTW in the external package. Your help is well appreciated. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Jul 7 06:47:06 2014 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 7 Jul 2014 11:47:06 +0000 Subject: [petsc-users] MatNestGetISs in fortran In-Reply-To: References: <64c4658aeb7441abbe20e4aa252554a2@MAR190N1.marin.local>, , Message-ID: <67a2f55136204b4987ae899b3d0a9064@MAR190n2.marin.local> Satish, Thanks for your reply! I've been on holiday for some time, do you still want me to test this or has it been fixed with the 3.5 release? Chris ________________________________________ From: Satish Balay Sent: Monday, June 16, 2014 5:41 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] MatNestGetISs in fortran perhaps this routine does not need custom fortran interface. Does the attached src/mat/impls/nest/ftn-auto/matnestf.c work [with petsc-3.4]? If so - I'll add this to petsc dev [master] thanks, Satish On Fri, 13 Jun 2014, Klaij, Christiaan wrote: > Perhaps this message from May 27 "slipped through the email cracks" as Matt puts it? > > Chris > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > ________________________________________ > From: Klaij, Christiaan > Sent: Monday, June 02, 2014 9:54 AM > To: petsc-users at mcs.anl.gov > Subject: RE: MatNestGetISs in fortran > > Just a reminder. Could you please add fortran support for MatNestGetISs? > ________________________________________ > From: Klaij, Christiaan > Sent: Tuesday, May 27, 2014 3:47 PM > To: petsc-users at mcs.anl.gov > Subject: MatNestGetISs in fortran > > I'm trying to use MatNestGetISs in a fortran program but it seems to be missing from the fortran include file (PETSc 3.4). > > From tisaac at ices.utexas.edu Mon Jul 7 10:36:52 2014 From: tisaac at ices.utexas.edu (Tobin Isaac) Date: Mon, 7 Jul 2014 10:36:52 -0500 Subject: [petsc-users] Jacobian construction, DA vs Plex Message-ID: <20140707153652.GA10150@ices.utexas.com> Hi, If I have a pointwise Jacobian function f, I know that I can call DMGetDS() and pass f to PetscDSSetJacobian(), and that f will be used by PetscFEIntegrate() and thus by DMPlexSNESComputeJacobianFEM(). It looks like PetscFEIntegrate() is only used by plex and not da. Is there anyway that I can use f to construct a Jacobian with a da object? Toby -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From knepley at gmail.com Mon Jul 7 10:43:48 2014 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Jul 2014 17:43:48 +0200 Subject: [petsc-users] Jacobian construction, DA vs Plex In-Reply-To: <20140707153652.GA10150@ices.utexas.com> References: <20140707153652.GA10150@ices.utexas.com> Message-ID: On Mon, Jul 7, 2014 at 5:36 PM, Tobin Isaac wrote: > > Hi, > > If I have a pointwise Jacobian function f, I know that I can call > DMGetDS() and pass f to PetscDSSetJacobian(), and that f will be used > by PetscFEIntegrate() and thus by DMPlexSNESComputeJacobianFEM(). It > looks like PetscFEIntegrate() is only used by plex and not da. Is > there anyway that I can use f to construct a Jacobian with a da > object? > There is a completely untested way. My idea is that the DA is able to give back cell closures the same way that Plex does, making all the FEM code work just as it does in the unstructured case. There are a few toy tests of this functionality in src/dm/impls/plex/examples/tests/ex3.c. I have not made a thoroughgoing example which switches between Plex and DA, but it should be possible. Let me know if it fails. All this DA code which does not assume a vertex discretization is activated by giving it a default section. Thanks, Matt > Toby > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jul 7 11:37:57 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 7 Jul 2014 11:37:57 -0500 Subject: [petsc-users] MatNestGetISs in fortran In-Reply-To: <67a2f55136204b4987ae899b3d0a9064@MAR190n2.marin.local> References: <64c4658aeb7441abbe20e4aa252554a2@MAR190N1.marin.local>, , <67a2f55136204b4987ae899b3d0a9064@MAR190n2.marin.local> Message-ID: My change is in petsc-3.5. You can upgrade and see if it works for you.. Satish On Mon, 7 Jul 2014, Klaij, Christiaan wrote: > Satish, > > Thanks for your reply! I've been on holiday for some time, do you still want me to test this or has it been fixed with the 3.5 release? > > Chris > > ________________________________________ > From: Satish Balay > Sent: Monday, June 16, 2014 5:41 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] MatNestGetISs in fortran > > perhaps this routine does not need custom fortran interface. > > Does the attached src/mat/impls/nest/ftn-auto/matnestf.c work [with petsc-3.4]? > > If so - I'll add this to petsc dev [master] > > thanks, > Satish > > On Fri, 13 Jun 2014, Klaij, Christiaan wrote: > > > Perhaps this message from May 27 "slipped through the email cracks" as Matt puts it? > > > > Chris > > > > > > dr. ir. Christiaan Klaij > > CFD Researcher > > Research & Development > > E mailto:C.Klaij at marin.nl > > T +31 317 49 33 44 > > > > > > MARIN > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > > > ________________________________________ > > From: Klaij, Christiaan > > Sent: Monday, June 02, 2014 9:54 AM > > To: petsc-users at mcs.anl.gov > > Subject: RE: MatNestGetISs in fortran > > > > Just a reminder. Could you please add fortran support for MatNestGetISs? > > ________________________________________ > > From: Klaij, Christiaan > > Sent: Tuesday, May 27, 2014 3:47 PM > > To: petsc-users at mcs.anl.gov > > Subject: MatNestGetISs in fortran > > > > I'm trying to use MatNestGetISs in a fortran program but it seems to be missing from the fortran include file (PETSc 3.4). > > > > > From tisaac at ices.utexas.edu Mon Jul 7 13:28:04 2014 From: tisaac at ices.utexas.edu (Tobin Isaac) Date: Mon, 7 Jul 2014 13:28:04 -0500 Subject: [petsc-users] Jacobian construction, DA vs Plex In-Reply-To: References: <20140707153652.GA10150@ices.utexas.com> Message-ID: <20140707182803.GA21666@ices.utexas.com> On Mon, Jul 07, 2014 at 05:43:48PM +0200, Matthew Knepley wrote: > On Mon, Jul 7, 2014 at 5:36 PM, Tobin Isaac wrote: > > > > > Hi, > > > > If I have a pointwise Jacobian function f, I know that I can call > > DMGetDS() and pass f to PetscDSSetJacobian(), and that f will be used > > by PetscFEIntegrate() and thus by DMPlexSNESComputeJacobianFEM(). It > > looks like PetscFEIntegrate() is only used by plex and not da. Is > > there anyway that I can use f to construct a Jacobian with a da > > object? > > > > There is a completely untested way. My idea is that the DA is able to give > back > cell closures the same way that Plex does, making all the FEM code work just > as it does in the unstructured case. There are a few toy tests of this > functionality > in src/dm/impls/plex/examples/tests/ex3.c. I have not made a thoroughgoing > example which switches between Plex and DA, but it should be possible. Let > me know if it fails. > > All this DA code which does not assume a vertex discretization is activated > by > giving it a default section. > > Thanks, > > Matt Thanks, Matt. I see the cell loop in DMDAComputeL2Diff(), and it looks similar enough to the plexfem.c cell loops that I could adapt DMPlexSNESComputeJacobianFEM() if I wanted to. Toby > > > > Toby > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From ashwinsrnth at gmail.com Mon Jul 7 20:32:46 2014 From: ashwinsrnth at gmail.com (Ashwin Srinath) Date: Mon, 7 Jul 2014 21:32:46 -0400 Subject: [petsc-users] [petsc4py] vec.getCUSPArray for cusp vecs Message-ID: Hello, petsc-users, In PETSc, we have VecCUSPGetArray to access the underlying array from a Vec of type 'cusp'. Does petsc4py offer something similar? If not, I'm considering using petsc4py and pyCUDA together in some way, and I'd be extremely grateful for any advice on doing that. Thanks so much, Ashwin -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrewdalecramer at gmail.com Tue Jul 8 01:28:04 2014 From: andrewdalecramer at gmail.com (Andrew Cramer) Date: Tue, 8 Jul 2014 16:28:04 +1000 Subject: [petsc-users] Matlab gets a different solution to Petsc Message-ID: Doing a basic cantilever bending problem to test my code which results in the linear system Au = b. - Using DMDA for the domain and KSPSetComputeOperators - Solving it using various methods in petsc gets similar (within 1%) solutions - Even using -pc_type lu - Using KSPGetOperators and KSPGetRhs to export to matlab Exporting the matrix and and the rhs, importing them into matlab and solving with backslash gives a solution which matches the Euler-Bernoulli beam model much closer (0.4% error vs 9.6%). Calculating the residual of petsc's solution using matlab ( norm(A*u-b)/norm(b)) I get 0.3 having solved with -pc_type lu. Is there a way I could have accidentally made petsc solve a different problem to Ax=b? I've been looking at this code for a while now (days) and can't seem to figure out what is wrong. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Tue Jul 8 02:55:42 2014 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Tue, 8 Jul 2014 09:55:42 +0200 Subject: [petsc-users] Matlab gets a different solution to Petsc In-Reply-To: References: Message-ID: <53BBA3FE.6080704@iue.tuwien.ac.at> Hi Andrew, > Doing a basic cantilever bending problem to test my code which results > in the linear system Au = b. > > - Using DMDA for the domain and KSPSetComputeOperators > - Solving it using various methods in petsc gets similar (within 1%) > solutions > - Even using -pc_type lu > - Using KSPGetOperators and KSPGetRhs to export to matlab > > Exporting the matrix and and the rhs, importing them into matlab and > solving with backslash gives a solution which matches the > Euler-Bernoulli beam model much closer (0.4% error vs 9.6%). > > Calculating the residual of petsc's solution using matlab > (norm(A*u-b)/norm(b)) I get 0.3 having solved with -pc_type lu. > > Is there a way I could have accidentally made petsc solve a different > problem to Ax=b? I've been looking at this code for a while now (days) > and can't seem to figure out what is wrong. one possible cause: How many digits to you export per floating point value? If you only export a small number of significant digits (~4) and your system doesn't have a very good condition number, then this is the effect of truncation errors. Best regards, Karli From knepley at gmail.com Tue Jul 8 03:44:46 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 8 Jul 2014 10:44:46 +0200 Subject: [petsc-users] Matlab gets a different solution to Petsc In-Reply-To: References: Message-ID: On Tue, Jul 8, 2014 at 8:28 AM, Andrew Cramer wrote: > Doing a basic cantilever bending problem to test my code which results in > the linear system Au = b. > > - Using DMDA for the domain and KSPSetComputeOperators > - Solving it using various methods in petsc gets similar (within 1%) > solutions > - Even using -pc_type lu > - Using KSPGetOperators and KSPGetRhs to export to matlab > > Exporting the matrix and and the rhs, importing them into matlab and > solving with backslash gives a solution which matches the Euler-Bernoulli > beam model much closer (0.4% error vs 9.6%). > As Karl says, this is probably truncation error, but more to the point this is accidental. You would not expect a solution that is more accurate than the discretization error. If you are comparing a discrete solution to a continuous solution, you want to look at "mesh convergence" meaning that you run on a series of refined meshes, calculate the error, and usually plot log(error) against mesh size h. Matt > Calculating the residual of petsc's solution using matlab ( > norm(A*u-b)/norm(b)) I get 0.3 having solved with -pc_type lu. > > Is there a way I could have accidentally made petsc solve a different > problem to Ax=b? I've been looking at this code for a while now (days) and > can't seem to figure out what is wrong. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Tue Jul 8 07:39:56 2014 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Tue, 08 Jul 2014 14:39:56 +0200 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg Message-ID: <53BBE69C.4000609@physik.uni-muenchen.de> Hi, I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. The code to setup the matrix is like this: call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) call DMCreateMatrix(C%da, A, ierr) call MatSetFromOptions(A,ierr) call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) insert matrix values & assemble then solve. If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. However if I use gamg I get an allocation error when it tries to create a coarse grid. If I use the DMDA preallocation, it works but uses way more memory... Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? Is that even the problem or am I missing something? Thank you so very much. Sincerely, Fabian From Fabian.Jakub at physik.uni-muenchen.de Tue Jul 8 07:46:01 2014 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Tue, 08 Jul 2014 14:46:01 +0200 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <53BBE69C.4000609@physik.uni-muenchen.de> References: <53BBE69C.4000609@physik.uni-muenchen.de> Message-ID: <53BBE809.5020106@physik.uni-muenchen.de> I'm sorry, I just remembered... this may be important to know... call KSPCreate(imp_comm,ksp,ierr) call KSPSetOperators(ksp,A,A,ierr) call KSPSetDM(ksp,C%da,ierr) call KSPSetDMActive(ksp,PETSC_FALSE,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSetUp(ksp,ierr) Thanks again... Am 08.07.2014 14:39, schrieb Fabian.Jakub: > Hi, > I have a question regarding gamg where I get a wrong preallocation > i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. > > I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on > a star stencil. > > The code to setup the matrix is like this: > > call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) > call DMCreateMatrix(C%da, A, ierr) > call MatSetFromOptions(A,ierr) > > call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, > PETSC_NULL_INTEGER, o_nnz, ierr) > > insert matrix values & assemble > > then solve. > > If I solve the system with any ''normal'' KSP/PC there is no problem > and matrix info actually confirms that the preallocation is good. > However if I use gamg I get an allocation error when it tries to > create a coarse grid. > > If I use the DMDA preallocation, it works but uses way more memory... > > Is there a possibility to use custom preallocation and at the same > time let gamg create the coarse grid? > Is that even the problem or am I missing something? > > Thank you so very much. > > Sincerely, > Fabian > > > > > > > From dominik at itis.ethz.ch Tue Jul 8 09:58:43 2014 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Tue, 8 Jul 2014 16:58:43 +0200 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ Message-ID: Suddenly (with no apparent change in the code) I am bombarded with messages like: ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: __FUNCT__="SetupStiffnessMatrix" does not agree with __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: __FUNCT__="SetupStiffnessMatrix" does not agree with __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: __FUNCT__="SetupStiffnessMatrix" does not agree with __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: __FUNCT__="SetupStiffnessMatrix" does not agree with __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" Can you please advise how to fix it? Many thanks Dominik -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jul 8 10:07:33 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 8 Jul 2014 10:07:33 -0500 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: Message-ID: On Tue, 8 Jul 2014, Dominik Szczerba wrote: > Suddenly (with no apparent change in the code) I am bombarded with messages > like: > > ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > __FUNCT__="SetupStiffnessMatrix" does not agree with > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: Use __FUNCT__="ScalarSolver::SetupStiffnessMatrix" ? __FUNCTION__ is automatically generated by some compilers - but we are still keeping __FUNCT__ arround - just to support compilers that dont do this automatically. And with compilers that do generate __FUNCTION__ string automatically - we are able to do this check and print a warning.. Satish > __FUNCT__="SetupStiffnessMatrix" does not agree with > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > __FUNCT__="SetupStiffnessMatrix" does not agree with > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > __FUNCT__="SetupStiffnessMatrix" does not agree with > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > > > Can you please advise how to fix it? > > Many thanks > Dominik > From bsmith at mcs.anl.gov Tue Jul 8 10:14:27 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 8 Jul 2014 10:14:27 -0500 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <53BBE69C.4000609@physik.uni-muenchen.de> References: <53BBE69C.4000609@physik.uni-muenchen.de> Message-ID: <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. Barry On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: > Hi, > I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. > > I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. > > The code to setup the matrix is like this: > > call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) > call DMCreateMatrix(C%da, A, ierr) > call MatSetFromOptions(A,ierr) > > call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) > > insert matrix values & assemble > > then solve. > > If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. > However if I use gamg I get an allocation error when it tries to create a coarse grid. > > If I use the DMDA preallocation, it works but uses way more memory... > > Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? > Is that even the problem or am I missing something? > > Thank you so very much. > > Sincerely, > Fabian > > > > > > > From Fabian.Jakub at physik.uni-muenchen.de Tue Jul 8 10:51:17 2014 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Tue, 08 Jul 2014 17:51:17 +0200 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> Message-ID: <53BC1375.9070701@physik.uni-muenchen.de> Thank you very much for the swift reply! I am running git describe: v3.5-25-g0ace994 and configured with ./configure \ --with-cc=$(which mpicc) \ --with-fc=$(which mpif90) \ --with-cxx=$(which mpic++) \ --with-fortran \ --with-shared-libraries \ --with-hdf5 \ --with-hdf5-dir="$HDF5_DIR" \ --with-cmake=$(which cmake) \ \ I run the model with: mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info Should I append the output when creating the DMDA? For the KSP setup it says: [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter [0] VecScatterCreate(): General case: MPI to Seq [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 Setup KSP [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Argument out of range [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 63. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. ------------------------------------------------------------------------- Am 08.07.2014 17:14, schrieb Barry Smith: > GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. > > Barry > > > > On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: > >> Hi, >> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. >> >> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. >> >> The code to setup the matrix is like this: >> >> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) >> call DMCreateMatrix(C%da, A, ierr) >> call MatSetFromOptions(A,ierr) >> >> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) >> >> insert matrix values & assemble >> >> then solve. >> >> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. >> However if I use gamg I get an allocation error when it tries to create a coarse grid. >> >> If I use the DMDA preallocation, it works but uses way more memory... >> >> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? >> Is that even the problem or am I missing something? >> >> Thank you so very much. >> >> Sincerely, >> Fabian >> >> >> >> >> >> >> From andrewdalecramer at gmail.com Tue Jul 8 18:34:32 2014 From: andrewdalecramer at gmail.com (Andrew Cramer) Date: Wed, 9 Jul 2014 09:34:32 +1000 Subject: [petsc-users] Matlab gets a different solution to Petsc In-Reply-To: References: Message-ID: I get *significantly *different results from backslash and petsc using the *same matrix*. Deflection |Au-b|/|b| Euler-Bernoulli: 4000 Petsc: 4386 0.3 Matlab: 4013 1.6e-10 I'm exporting to 6 significant figures. Condition number is rather poor (~2e7) but I get similar or worse results with problems having lower conditions numbers (~1e3). On 8 July 2014 18:44, Matthew Knepley wrote: > On Tue, Jul 8, 2014 at 8:28 AM, Andrew Cramer > wrote: > >> Doing a basic cantilever bending problem to test my code which results in >> the linear system Au = b. >> >> - Using DMDA for the domain and KSPSetComputeOperators >> - Solving it using various methods in petsc gets similar (within 1%) >> solutions >> - Even using -pc_type lu >> - Using KSPGetOperators and KSPGetRhs to export to matlab >> >> Exporting the matrix and and the rhs, importing them into matlab and >> solving with backslash gives a solution which matches the Euler-Bernoulli >> beam model much closer (0.4% error vs 9.6%). >> > > As Karl says, this is probably truncation error, but more to the point > this is accidental. You would not expect > a solution that is more accurate than the discretization error. If you are > comparing a discrete solution to a > continuous solution, you want to look at "mesh convergence" meaning that > you run on a series of refined > meshes, calculate the error, and usually plot log(error) against mesh size > h. > > Matt > > >> Calculating the residual of petsc's solution using matlab ( >> norm(A*u-b)/norm(b)) I get 0.3 having solved with -pc_type lu. >> >> Is there a way I could have accidentally made petsc solve a different >> problem to Ax=b? I've been looking at this code for a while now (days) and >> can't seem to figure out what is wrong. >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 8 18:41:45 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 08 Jul 2014 18:41:45 -0500 Subject: [petsc-users] Matlab gets a different solution to Petsc In-Reply-To: References: Message-ID: <87egxvfmfa.fsf@jedbrown.org> Andrew Cramer writes: > I get *significantly *different results from backslash and petsc using > the *same > matrix*. > > Deflection |Au-b|/|b| > Euler-Bernoulli: 4000 > Petsc: 4386 0.3 > Matlab: 4013 1.6e-10 > > I'm exporting to 6 significant figures. Condition number is rather poor > (~2e7) but I get similar or worse results with problems having lower > conditions numbers (~1e3). Are you sure the matrices are the same? Use "-ksp_view_mat binary", then in MATLAB, A = PetscBinaryRead('binaryoutput') if you want to get both matrices in the same environment. Or explicitly compute the norms in PETSc. To debug, start with the simplest and most well-conditioned problem that exhibits the issue. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From andrewdalecramer at gmail.com Tue Jul 8 19:10:59 2014 From: andrewdalecramer at gmail.com (Andrew Cramer) Date: Wed, 9 Jul 2014 10:10:59 +1000 Subject: [petsc-users] Matlab gets a different solution to Petsc In-Reply-To: <87egxvfmfa.fsf@jedbrown.org> References: <87egxvfmfa.fsf@jedbrown.org> Message-ID: Seems those last few significant figures were significant after all. Now to figure out why getting rid of them gets a better answer than leaving them in... Thanks for the help, I'll root around a bit more. On 9 July 2014 09:41, Jed Brown wrote: > Andrew Cramer writes: > > > I get *significantly *different results from backslash and petsc using > > the *same > > matrix*. > > > > Deflection |Au-b|/|b| > > Euler-Bernoulli: 4000 > > Petsc: 4386 0.3 > > Matlab: 4013 1.6e-10 > > > > I'm exporting to 6 significant figures. Condition number is rather poor > > (~2e7) but I get similar or worse results with problems having lower > > conditions numbers (~1e3). > > Are you sure the matrices are the same? Use "-ksp_view_mat binary", > then in MATLAB, A = PetscBinaryRead('binaryoutput') if you want to get > both matrices in the same environment. Or explicitly compute the norms > in PETSc. > > To debug, start with the simplest and most well-conditioned problem > that exhibits the issue. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 8 19:13:44 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 08 Jul 2014 19:13:44 -0500 Subject: [petsc-users] [petsc4py] vec.getCUSPArray for cusp vecs In-Reply-To: References: Message-ID: <87a98jfkxz.fsf@jedbrown.org> r<#secure method=pgpmime mode=sign> Ashwin Srinath writes: > Hello, petsc-users, > > In PETSc, we have VecCUSPGetArray to access the underlying array from a Vec > of type 'cusp'. Does petsc4py offer something similar? If not, I'm > considering using petsc4py and pyCUDA together in some way, and I'd be > extremely grateful for any advice on doing that. It does not currently exist. What type would you like to see returned? If we can find a sufficiently generic approach, it can be added. From ashwinsrnth at gmail.com Tue Jul 8 19:30:06 2014 From: ashwinsrnth at gmail.com (Ashwin Srinath) Date: Tue, 8 Jul 2014 20:30:06 -0400 Subject: [petsc-users] [petsc4py] vec.getCUSPArray for cusp vecs In-Reply-To: <87a98jfkxz.fsf@jedbrown.org> References: <87a98jfkxz.fsf@jedbrown.org> Message-ID: The vec.getArray() functions returns a NumPy array. I think it'd be great to see a vec.getGPUArray() that returns a PyCUDA GPUArray . Thoughts? Thanks! Ashwin On Tue, Jul 8, 2014 at 8:13 PM, Jed Brown wrote: > r<#secure method=pgpmime mode=sign> > Ashwin Srinath writes: > > > Hello, petsc-users, > > > > In PETSc, we have VecCUSPGetArray to access the underlying array from a > Vec > > of type 'cusp'. Does petsc4py offer something similar? If not, I'm > > considering using petsc4py and pyCUDA together in some way, and I'd be > > extremely grateful for any advice on doing that. > > It does not currently exist. What type would you like to see returned? > If we can find a sufficiently generic approach, it can be added. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From milo at zeusdvds.com Tue Jul 8 19:32:06 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Tue, 08 Jul 2014 17:32:06 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 8, 2014 Message-ID: <53bc8d868aad3_301933fea058053d9@prd-nresque01.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQyNTg_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJLX1ZOWlFBUGtGNUlDanVBbU9rS2RwZmRuNVE2VEgyQkElM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpReU5UZ19ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGZXh6REZoY3R6eHFWazFvWWlrb0RZdyUzRCUzRA==/nSWOjOBwx81T8NKSm5-K7w==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs The Law's Lash (1928) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWxhd3MtbGFzaC0xOTI4LWR2ZC8=/IIDblb_M1WUO7d-fcJXJrA== )South of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/wZ0serfa9wZQGqzowBlB5g== )Diary of a Hitman (1991) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZGlhcnktb2YtYS1oaXRtYW4tMTk5MS1kdmQv/j7EBuHVnmOVO4va0yqOv9g== )Razzia sur le chnouf (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcmF6emlhLXN1ci1sYS1jaG5vdWYtMTk1NS1kdmQv/b8xI-_inOBIwhKclBRA4jg== )Let's Get Lost (1988) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGV0cy1nZXQtbG9zdC0xOTg4LWR2ZC8=/g8zgyzKOL-BLK6svFNZ57g== )The Showdown (1950) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNob3dkb3duLTE5NTAtZHZkLw==/Upd_toFkLSCDzB11bCcsrQ== )The Spoilers (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNwb2lsZXJzLTE5NTUtZHZkLw==/8uwyNMgk4Txuw9Ycs7zfCw== )Escort Girl (1941) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZXNjb3J0LWdpcmwtMTk0MS1kdmQv/wAqqaMUC7vqPRnutmtP4KA== )Fast Forward (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZmFzdC1mb3J3YXJkLTE5ODUtZHZkLw==/4w13MgzvSiZ6eN2qzvE6oA== )Rocky IV (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcm9ja3ktaXYtMTk4NS1kdmQv/QKdYJiJkSq35y7oFeIryOQ== ) Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/kXcFWX7NoTIsFC1dR78Zfw== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/0P72etsQrM2OfcbExPsxaw== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/ygOSkiHEPr-LFUsQw_Zy0g== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/2Z9k30KzVw34OZtsLUUXPg== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/r_9NtuFa_WIW_cubk3umfA== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/IPKFxAnwXABQLvEIpqGKjg== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/V_LhmIJJrFPEURRJeqNdeg== )? Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/ECDvk-cjzZUpob03B7-feA==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774258&sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQyNTg%3D%2FADT27paxrhUp3_FRmokFiQ%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBK_VNZQAPkF5ICjuAmOkKdpfdn5Q6TH2BA=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0MjU4/ZeUIYucr-UiBu9sKfk8pYg==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774258&utm_content=CamID6597069774258 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrewdalecramer at gmail.com Tue Jul 8 20:18:56 2014 From: andrewdalecramer at gmail.com (Andrew Cramer) Date: Wed, 9 Jul 2014 11:18:56 +1000 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: Message-ID: Rather than maintain them manually one option is to put #undef __FUNCT__ #define __FUNCT__ __func__ at the head of your source files which handles any namespace mangling and ensures that you will always be correct. __func__ is standard for C99 and well supported. On 9 July 2014 01:07, Satish Balay wrote: > On Tue, 8 Jul 2014, Dominik Szczerba wrote: > > > Suddenly (with no apparent change in the code) I am bombarded with > messages > > like: > > > > ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > > __FUNCT__="SetupStiffnessMatrix" does not agree with > > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > > > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > > Use __FUNCT__="ScalarSolver::SetupStiffnessMatrix" ? > > __FUNCTION__ is automatically generated by some compilers - but we are > still keeping __FUNCT__ arround - just to support compilers that dont > do this automatically. > > And with compilers that do generate __FUNCTION__ string automatically - we > are able > to do this check and print a warning.. > > Satish > > > > __FUNCT__="SetupStiffnessMatrix" does not agree with > > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > > > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > > __FUNCT__="SetupStiffnessMatrix" does not agree with > > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > > > "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > > __FUNCT__="SetupStiffnessMatrix" does not agree with > > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > > > > > > Can you please advise how to fix it? > > > > Many thanks > > Dominik > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 8 20:27:24 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 9 Jul 2014 03:27:24 +0200 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: Message-ID: On Wed, Jul 9, 2014 at 3:18 AM, Andrew Cramer wrote: > Rather than maintain them manually one option is to put > > #undef __FUNCT__ > #define __FUNCT__ __func__ > > at the head of your source files which handles any namespace mangling and > ensures that you will always be correct. __func__ is standard for C99 and > well supported. > We cannot count on C99. For example, it is not support by MS. Matt > > On 9 July 2014 01:07, Satish Balay wrote: > >> On Tue, 8 Jul 2014, Dominik Szczerba wrote: >> >> > Suddenly (with no apparent change in the code) I am bombarded with >> messages >> > like: >> > >> > ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: >> > __FUNCT__="SetupStiffnessMatrix" does not agree with >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" >> > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: >> >> Use __FUNCT__="ScalarSolver::SetupStiffnessMatrix" ? >> >> __FUNCTION__ is automatically generated by some compilers - but we are >> still keeping __FUNCT__ arround - just to support compilers that dont >> do this automatically. >> >> And with compilers that do generate __FUNCTION__ string automatically - >> we are able >> to do this check and print a warning.. >> >> Satish >> >> >> > __FUNCT__="SetupStiffnessMatrix" does not agree with >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" >> > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: >> > __FUNCT__="SetupStiffnessMatrix" does not agree with >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" >> > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: >> > __FUNCT__="SetupStiffnessMatrix" does not agree with >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" >> > >> > >> > Can you please advise how to fix it? >> > >> > Many thanks >> > Dominik >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jul 8 21:58:31 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 8 Jul 2014 21:58:31 -0500 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: Message-ID: One can skip __FUNCT__ alltogether in user sourcefiles [assuming you have a c99 compiler] - as __func__ would automatically be used by the petsc macros [in includes] Satish On Tue, 8 Jul 2014, Matthew Knepley wrote: > On Wed, Jul 9, 2014 at 3:18 AM, Andrew Cramer > wrote: > > > Rather than maintain them manually one option is to put > > > > #undef __FUNCT__ > > #define __FUNCT__ __func__ > > > > at the head of your source files which handles any namespace mangling and > > ensures that you will always be correct. __func__ is standard for C99 and > > well supported. > > > > We cannot count on C99. For example, it is not support by MS. > > Matt > > > > > > On 9 July 2014 01:07, Satish Balay wrote: > > > >> On Tue, 8 Jul 2014, Dominik Szczerba wrote: > >> > >> > Suddenly (with no apparent change in the code) I am bombarded with > >> messages > >> > like: > >> > > >> > ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > >> > __FUNCT__="SetupStiffnessMatrix" does not agree with > >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > >> > > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > >> > >> Use __FUNCT__="ScalarSolver::SetupStiffnessMatrix" ? > >> > >> __FUNCTION__ is automatically generated by some compilers - but we are > >> still keeping __FUNCT__ arround - just to support compilers that dont > >> do this automatically. > >> > >> And with compilers that do generate __FUNCTION__ string automatically - > >> we are able > >> to do this check and print a warning.. > >> > >> Satish > >> > >> > >> > __FUNCT__="SetupStiffnessMatrix" does not agree with > >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > >> > > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > >> > __FUNCT__="SetupStiffnessMatrix" does not agree with > >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > >> > > >> "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp:660: > >> > __FUNCT__="SetupStiffnessMatrix" does not agree with > >> > __FUNCTION__="ScalarSolver::SetupStiffnessMatrix" > >> > > >> > > >> > Can you please advise how to fix it? > >> > > >> > Many thanks > >> > Dominik > >> > > >> > >> > > > > > From jed at jedbrown.org Tue Jul 8 22:21:12 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 08 Jul 2014 22:21:12 -0500 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: Message-ID: <8761j7fc9j.fsf@jedbrown.org> Satish Balay writes: > One can skip __FUNCT__ alltogether in user sourcefiles [assuming you > have a c99 compiler] - as __func__ would automatically be used > by the petsc macros [in includes] You can literally skip it everywhere, C99 or not, and PETSc will not issue these warnings. If __func__ is not available, traces will say "User defined function". -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Jul 9 00:37:07 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 9 Jul 2014 00:37:07 -0500 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <53BC1375.9070701@physik.uni-muenchen.de> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> <53BC1375.9070701@physik.uni-muenchen.de> Message-ID: <40960837-6B62-4F8B-952F-54B548FD454B@mcs.anl.gov> Can you run with -ksp_view_pre binary and email the file binaryoutput With this we should be able to reproduce the problem and debug it. Thanks Barry On Jul 8, 2014, at 10:51 AM, Fabian.Jakub wrote: > Thank you very much for the swift reply! > > I am running > git describe: v3.5-25-g0ace994 > and configured with > > ./configure \ > --with-cc=$(which mpicc) \ > --with-fc=$(which mpif90) \ > --with-cxx=$(which mpic++) \ > --with-fortran \ > --with-shared-libraries \ > --with-hdf5 \ > --with-hdf5-dir="$HDF5_DIR" \ > --with-cmake=$(which cmake) \ > \ > > I run the model with: > mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info > > Should I append the output when creating the DMDA? > For the KSP setup it says: > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 > Setup KSP > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 > [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Argument out of range > [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 63. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > ------------------------------------------------------------------------- > > Am 08.07.2014 17:14, schrieb Barry Smith: >> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. >> >> Barry >> >> >> >> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: >> >>> Hi, >>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. >>> >>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. >>> >>> The code to setup the matrix is like this: >>> >>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) >>> call DMCreateMatrix(C%da, A, ierr) >>> call MatSetFromOptions(A,ierr) >>> >>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) >>> >>> insert matrix values & assemble >>> >>> then solve. >>> >>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. >>> However if I use gamg I get an allocation error when it tries to create a coarse grid. >>> >>> If I use the DMDA preallocation, it works but uses way more memory... >>> >>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? >>> Is that even the problem or am I missing something? >>> >>> Thank you so very much. >>> >>> Sincerely, >>> Fabian >>> >>> >>> >>> >>> >>> >>> > From Fabian.Jakub at physik.uni-muenchen.de Wed Jul 9 06:06:30 2014 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Wed, 09 Jul 2014 13:06:30 +0200 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <40960837-6B62-4F8B-952F-54B548FD454B@mcs.anl.gov> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> <53BC1375.9070701@physik.uni-muenchen.de> <40960837-6B62-4F8B-952F-54B548FD454B@mcs.anl.gov> Message-ID: <53BD2236.7030100@physik.uni-muenchen.de> If I do, the generated file is empty and the just states the solvers... cat binaryoutput: {Wgmres{Vilu Do I have an error in my Petscbuild? This by the way happens with all matrices, even tried a couple of examples from petsc src. Anyway, I uploaded the matrix dump (-mat_view binary) I hope this helps. To be thorough, I further put up a second file, with 2 matrices, The second matrix is the same as before (the one that is not working with gamg). The first one with dof 8 is solved and the solution is somewhat the source term for the second. I just thought I would append this one too because it does indeed work with gamg. Maybe that gives any hints. I really appreciate you looking into this. Thank you! Sincerely, Fabian Am 09.07.2014 07:37, schrieb Barry Smith: > Can you run with -ksp_view_pre binary and email the file binaryoutput > > With this we should be able to reproduce the problem and debug it. > > Thanks > > > Barry > > On Jul 8, 2014, at 10:51 AM, Fabian.Jakub wrote: > >> Thank you very much for the swift reply! >> >> I am running >> git describe: v3.5-25-g0ace994 >> and configured with >> >> ./configure \ >> --with-cc=$(which mpicc) \ >> --with-fc=$(which mpif90) \ >> --with-cxx=$(which mpic++) \ >> --with-fortran \ >> --with-shared-libraries \ >> --with-hdf5 \ >> --with-hdf5-dir="$HDF5_DIR" \ >> --with-cmake=$(which cmake) \ >> \ >> >> I run the model with: >> mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info >> >> Should I append the output when creating the DMDA? >> For the KSP setup it says: >> >> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter >> [0] VecScatterCreate(): General case: MPI to Seq >> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used >> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 >> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. >> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used >> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 >> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. >> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 >> Setup KSP >> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 >> [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 >> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 >> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 >> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [1]PETSC ERROR: Argument out of range >> [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc >> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 >> [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 >> [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake >> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c >> [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c >> [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c >> [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c >> [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c >> [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc >> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 >> [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 >> [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake >> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c >> [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c >> [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c >> [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c >> [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 63. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> ------------------------------------------------------------------------- >> >> Am 08.07.2014 17:14, schrieb Barry Smith: >>> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. >>> >>> Barry >>> >>> >>> >>> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: >>> >>>> Hi, >>>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. >>>> >>>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. >>>> >>>> The code to setup the matrix is like this: >>>> >>>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) >>>> call DMCreateMatrix(C%da, A, ierr) >>>> call MatSetFromOptions(A,ierr) >>>> >>>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) >>>> >>>> insert matrix values & assemble >>>> >>>> then solve. >>>> >>>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. >>>> However if I use gamg I get an allocation error when it tries to create a coarse grid. >>>> >>>> If I use the DMDA preallocation, it works but uses way more memory... >>>> >>>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? >>>> Is that even the problem or am I missing something? >>>> >>>> Thank you so very much. >>>> >>>> Sincerely, >>>> Fabian >>>> >>>> >>>> >>>> >>>> >>>> >>>> From Shuangshuang.Jin at pnnl.gov Wed Jul 9 18:37:19 2014 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 9 Jul 2014 23:37:19 +0000 Subject: [petsc-users] How to set KSP Solver Options at Runtime? Message-ID: <71FF54182841B443932BB8F835FD98537D6DC7@EX10MBOX02.pnnl.gov> Hi, I'm using the KSP solver as following to solve a linear system Ax=b, where A is a 16767x16767 square matrix, b is a 16767 length vector. EXTERN_C_BEGIN extern PetscErrorCode PCCreate_Jacobi(PC); EXTERN_C_END static PetscErrorCode solvingAxb(Mat A, Vec b, PetscInt nbus, Vec &x, const int me) { PetscErrorCode ierr; KSP ksp; // linear solver context Vec u; // exact solution PetscInt its; PetscReal norm; // norm of solution error PetscLogDouble t1, t2; PetscViewer viewer; PC pc; // preconditioner context PetscInt Istart, Iend; ierr = VecDuplicate(b, &u); CHKERRQ(ierr); ierr = VecDuplicate(b, &x); CHKERRQ(ierr); ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); CHKERRQ(ierr); ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN); CHKERRQ(ierr); PCRegister("ourjacobi", PCCreate_Jacobi); KSPGetPC(ksp, &pc); PCSetType(pc, "ourjacobi"); ierr = KSPSetFromOptions(ksp); CHKERRQ(ierr); ierr = KSPSolve(ksp, b, x); CHKERRQ(ierr); ierr = MatMult(A,x,u); CHKERRQ(ierr); ierr = VecAXPY(u, -1.0, b); CHKERRQ(ierr); ierr = VecNorm(u, NORM_2, &norm); CHKERRQ(ierr); ierr = KSPGetIterationNumber(ksp, &its); CHKERRQ(ierr); ierr = VecDestroy(&b); CHKERRQ(ierr); ierr = VecDestroy(&u); CHKERRQ(ierr); ierr = KSPDestroy(&ksp);CHKERRQ(ierr); PetscFunctionReturn(0); } I would like to know how to set the solver options at runtime to make it run faster, such as ksp_type, pc_type, and etc? It takes very long time to solve the system if I use no options. Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 9 18:53:21 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 10 Jul 2014 01:53:21 +0200 Subject: [petsc-users] How to set KSP Solver Options at Runtime? In-Reply-To: <71FF54182841B443932BB8F835FD98537D6DC7@EX10MBOX02.pnnl.gov> References: <71FF54182841B443932BB8F835FD98537D6DC7@EX10MBOX02.pnnl.gov> Message-ID: On Thu, Jul 10, 2014 at 1:37 AM, Jin, Shuangshuang < Shuangshuang.Jin at pnnl.gov> wrote: > Hi, I?m using the KSP solver as following to solve a linear system Ax=b, > where A is a 16767x16767 square matrix, b is a 16767 length vector. > > EXTERN_C_BEGIN > extern PetscErrorCode PCCreate_Jacobi(PC); > EXTERN_C_END > > static PetscErrorCode solvingAxb(Mat A, Vec b, PetscInt nbus, Vec &x, > const int me) > { > PetscErrorCode ierr; > > KSP ksp; // linear solver context > Vec u; // exact solution > PetscInt its; > PetscReal norm; // norm of solution error > PetscLogDouble t1, t2; > PetscViewer viewer; > PC pc; // preconditioner context > PetscInt Istart, Iend; > > ierr = VecDuplicate(b, &u); CHKERRQ(ierr); > ierr = VecDuplicate(b, &x); CHKERRQ(ierr); > > ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); CHKERRQ(ierr); > > ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN); > CHKERRQ(ierr); > > PCRegister("ourjacobi", PCCreate_Jacobi); > > KSPGetPC(ksp, &pc); > PCSetType(pc, "ourjacobi"); > > ierr = KSPSetFromOptions(ksp); CHKERRQ(ierr); > > ierr = KSPSolve(ksp, b, x); CHKERRQ(ierr); > > ierr = MatMult(A,x,u); CHKERRQ(ierr); > ierr = VecAXPY(u, -1.0, b); CHKERRQ(ierr); > ierr = VecNorm(u, NORM_2, &norm); CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp, &its); CHKERRQ(ierr); > > ierr = VecDestroy(&b); CHKERRQ(ierr); > ierr = VecDestroy(&u); CHKERRQ(ierr); > > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > > PetscFunctionReturn(0); > } > > I would like to know how to set the solver options at runtime to make it > run faster, such as ksp_type, pc_type, and etc? It takes very long time to > solve the system if I use no options. > The preconditioner always depends on the particular problem being solved. You will have to tell us more about it. Thanks, Matt > Thanks, > Shuangshuang > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From milo at zeusdvds.com Wed Jul 9 22:02:05 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Wed, 09 Jul 2014 20:02:05 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 9, 2014 Message-ID: <53be022d85ce2_5d79899ea07647909@prd-nresque02.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQzMjI_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJLX0RkWlFBUGtGNVBseHNFTk4zcDU5dnh6cEl6Mkt3NzQlM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpRek1qSV9ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGSDdLcnd6VFZTaXRRRTBxbFZEVnFCdyUzRCUzRA==/CdIqWNiEJej7XWXL99gkEQ==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs The Law's Lash (1928) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWxhd3MtbGFzaC0xOTI4LWR2ZC8=/s9jeB6iaMVsYWL5EMbaqDg== )South of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/JIJb6gMd3TGJu6hIgRaWDg== )Diary of a Hitman (1991) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZGlhcnktb2YtYS1oaXRtYW4tMTk5MS1kdmQv/UHmirDr0Srmivg-F3OTqwA== )Razzia sur le chnouf (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcmF6emlhLXN1ci1sYS1jaG5vdWYtMTk1NS1kdmQv/ig1jZtORQ4K3mkg_FbMTJA== )Let's Get Lost (1988) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGV0cy1nZXQtbG9zdC0xOTg4LWR2ZC8=/lPFCMiXOIhCJ2e9dNyqI_g== )The Showdown (1950) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNob3dkb3duLTE5NTAtZHZkLw==/DwIOEIg-oFnOGg5-VdCbxQ== )The Spoilers (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNwb2lsZXJzLTE5NTUtZHZkLw==/9jFWtCx9KnhanGIbBfKBnw== )Escort Girl (1941) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZXNjb3J0LWdpcmwtMTk0MS1kdmQv/mBdzvXfUUBaHBVKHWhN2ZA== )Fast Forward (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZmFzdC1mb3J3YXJkLTE5ODUtZHZkLw==/uf9ocA82w_2x1LTIwluEuQ== )Rocky IV (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcm9ja3ktaXYtMTk4NS1kdmQv/OaHx8WnWpHpCN94j-9HkZA== ) Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/qqA8ft3oR3JTob-v760Bkw== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/JtORe3wr1MAG-9sl1v8C1w== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/04-_ZcDKAMe5ziiNVhqQHA== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/oCjkXmYIW1aQkJQTlhe8Pw== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/w8WT8eqkuuLBTy53jXEPjg== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/LXHq_yvJcBJn5t6oEC6G5A== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/M6hI5mwYXF8AIhXVrl8BOA== )? All orders ship within 24 hours and all shipping is FREE. Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/yhCWn7_FdnyPgPYSQ1bLOA==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774322&sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQzMjI%3D%2FeA1GDvL7xVrWZVxccgj5fA%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBK_DdZQAPkF5PlxsENN3p59vxzpIz2Kw74=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0MzIy/zb8a0eHs0a9Udmzz_dJPSw==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774322&utm_content=CamID6597069774322 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Jul 9 23:04:30 2014 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Thu, 10 Jul 2014 04:04:30 +0000 Subject: [petsc-users] How to set KSP Solver Options at Runtime? In-Reply-To: <71FF54182841B443932BB8F835FD98537D6DC7@EX10MBOX02.pnnl.gov> Message-ID: Shuangshuang, You don't need PCRegister("ourjacobi", PCCreate_Jacobi); KSPGetPC(ksp, &pc); PCSetType(pc, "ourjacobi"); in your code if you want to use PETSc's native Jacobi preconditioner. All you have to do is set it at run-time -pc_type jacobi. In PETSc, XXXSetFromOptions routine checks if a particular solver is specified via a run-time option. If it is then that solver is set otherwise it uses the default. KSPSetFromOptions sets the linear solver and the preconditioner (runtime options -ksp_type and -pc_type), SNESSetFromOptions the nonlinear solver (runtime option -snes_type), and TSSetFromOptions the time-stepping method (runtime option -ts_type). There are other bunch of options for tuning preconditioners. Run your code with -help option to see the list of options available. Shri From: "Jin, Shuangshuang" Date: Wed, 9 Jul 2014 23:37:19 +0000 To: "petsc-users at mcs.anl.gov" Subject: [petsc-users] How to set KSP Solver Options at Runtime? >Hi, I?m using the KSP solver as following to solve a linear system Ax=b, >where A is a 16767x16767 square matrix, b is a 16767 length vector. > >EXTERN_C_BEGIN >extern PetscErrorCode PCCreate_Jacobi(PC); >EXTERN_C_END > >static PetscErrorCode solvingAxb(Mat A, Vec b, PetscInt nbus, Vec &x, >const int me) >{ > PetscErrorCode ierr; > > KSP ksp; // linear solver context > Vec u; // exact solution > PetscInt its; > PetscReal norm; // norm of solution error > PetscLogDouble t1, t2; > PetscViewer viewer; > PC pc; // preconditioner context > PetscInt Istart, Iend; > > ierr = VecDuplicate(b, &u); CHKERRQ(ierr); > ierr = VecDuplicate(b, &x); CHKERRQ(ierr); > > ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); CHKERRQ(ierr); > > ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN); >CHKERRQ(ierr); > > PCRegister("ourjacobi", PCCreate_Jacobi); > > KSPGetPC(ksp, &pc); > PCSetType(pc, "ourjacobi"); > > ierr = KSPSetFromOptions(ksp); CHKERRQ(ierr); > > ierr = KSPSolve(ksp, b, x); CHKERRQ(ierr); > > ierr = MatMult(A,x,u); CHKERRQ(ierr); > ierr = VecAXPY(u, -1.0, b); CHKERRQ(ierr); > ierr = VecNorm(u, NORM_2, &norm); CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp, &its); CHKERRQ(ierr); > > ierr = VecDestroy(&b); CHKERRQ(ierr); > ierr = VecDestroy(&u); CHKERRQ(ierr); > > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > > PetscFunctionReturn(0); >} > >I would like to know how to set the solver options at runtime to make it >run faster, such as ksp_type, pc_type, and etc? It takes very long time >to solve the system if I use no options. > >Thanks, >Shuangshuang > From Vincent.De-Groof at uibk.ac.at Thu Jul 10 07:41:23 2014 From: Vincent.De-Groof at uibk.ac.at (De Groof, Vincent Frans Maria) Date: Thu, 10 Jul 2014 12:41:23 +0000 Subject: [petsc-users] User-defined KSP Message-ID: <17A78B9D13564547AC894B88C15967472039D630@XMBX4.uibk.ac.at> Hi, I want to do some experiments with my own iterative solvers. Is there any alternative for a user-defined KSP similar to PCSHELL? And a question on the side. Are the orthogonalization methods ( (M)GS ) readily available? Or are they private for GMRES? thanks, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jul 10 07:57:51 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 10 Jul 2014 07:57:51 -0500 Subject: [petsc-users] User-defined KSP In-Reply-To: <17A78B9D13564547AC894B88C15967472039D630@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C15967472039D630@XMBX4.uibk.ac.at> Message-ID: <87bnsxcqwg.fsf@jedbrown.org> "De Groof, Vincent Frans Maria" writes: > I want to do some experiments with my own iterative solvers. Is there > any alternative for a user-defined KSP similar to PCSHELL? There is not a KSPSHELL, but you can create and register new implementations from user code. I recommend copying an existing implementation most like what you want to implement, or start from something simple like src/ksp/ksp/impls/cr/cr.c. Namespace the implementation, compile it, and call KSPRegister() somewhere in your application. If it turns out to work better than other options for some problem of interest, please consider contributing to PETSc. > And a question on the side. Are the orthogonalization methods ( (M)GS > ) readily available? Or are they private for GMRES? The GMRES implementation is incremental. If you have all the vectors at once, it's just a couple lines of code using VecMDot and VecMAXPY. We may add a general-purpose "TSQR" in the near future. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From milo at zeusdvds.com Thu Jul 10 14:45:55 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Thu, 10 Jul 2014 12:45:55 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 10, 2014 Message-ID: <53beed73f1a64_326f11fde9c42920018@prd-nresque01.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQzNzQ_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJJNGlOZFFBUGtGNU1vaGZmRWJER1ZqTGJ3eUVhbWpXUE0lM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpRek56UV9ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGTHJ4bWV4TldVR09WeG9sbnZVdUtsQSUzRCUzRA==/h_G14HtcbyIPjlx0p_jh-A==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs All orders ship FREE and all orders ship within 24 hours. New Releases Feudin', Fussin' and a-Fightin' (1948) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZmV1ZGluLWZ1c3Npbi1hbmQtYS1maWdodGluLTE5NDgtZHZkLw==/IrWdtgsGufvyIDqHuNRtZA== )Two Latins From Manhattan (1941) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdHdvLWxhdGlucy1mcm9tLW1hbmhhdHRhbi0xOTQxLWR2ZC8=/FeOVU0r3CVvKeyRbDhQbNA== )Start Cheering (1938) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc3RhcnQtY2hlZXJpbmctMTkzOC1kdmQv/jbfcbojRQ8yg78iCmA0ZvA== )Fish Don't Blink (2002) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZmlzaC1kb250LWJsaW5rLTIwMDItZHZkLw==/eb5ns7C4L6J1g42JucQ0SA== )Futureworld (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZnV0dXJld29ybGQtMTk3Ni1kdmQv/v-FqW78y-k-zJE5Gg_49cQ== )Cthulhu Mansion (1992) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vY3RodWxodS1tYW5zaW9uLTE5OTItZHZkLw==/xe9NvoVwrIPyKSp8oxZKyw== )Transatlantic (1931) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdHJhbnNhdGxhbnRpYy0xOTMxLWR2ZC8=/co_Zp-k1e3fgo4A4QmTvcw== )Ghost of Hidden Valley (1946) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZ2hvc3Qtb2YtaGlkZGVuLXZhbGxleS0xOTQ2LWR2ZC8=/OGDek_ZhW53hHkTZQkhSPg== )The Thin Red Line (1964) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXRoaW4tcmVkLWxpbmUtMTk2NC1kdmQv/aoRc6Iplu6ugAOJLluBpmA== )The Two-Headed Spy (1958) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXR3by1oZWFkZWQtc3B5LTE5NTgtZHZkLw==/MLJJLxVg4qIPTQKnEVlO9g== ) Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/WSjGj1vbR58MVZXVE7mvAQ== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/q_4dN3oOgbRINYJT57sSUA== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/nrxOZanWID3NTJ439R1hrg== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/aZi57647qz388A_BPKXKJA== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/LX-oXD__uX6erL6Hdcz5mQ== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/zjOzSphMOT4Dv9yTygF-OQ== )S ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/zjOzSphMOT4Dv9yTygF-OQ== )outh of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/l9c2lpl6tulnLD5_feGQKw== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/mQ7nbifEv8WxZ_ME_qIP6g== )? Quick LinksComedy ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vY29tZWR5Lw==/_cYlTQyiPE3RPSwv3F9fXg== )\Drama ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZHJhbWEv/oMMGQEk6IVeYOrou5dACRg== )Western ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vd2VzdGVybi8=/809_Mx5wPNm_WI0VOGHZTQ== )Musical ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbXVzaWNhbC8=/QjW3ICUCYhuTjRJG24ah8Q== )Adventure ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYWR2ZW50dXJlLw==/7rYWEcU2gTCfiDcnUUo0bQ== )Documentary ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZG9jdW1lbnRhcnkv/IGecTRluAbuUWhp7RHYBCg== )Animation ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYW5pbWF0aW9uLw==/kZ1vq8gIpV9wqCDZJpj4_A== )TV Shows ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdHYtc2hvd3Mv/CkYwJBx94vcSbIlyKVP-RA== ) All orders ship within 24 hours and all shipping is FREE. Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/1DdU-q-5BVkA6MaTD6SjaQ==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774374&sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQzNzQ%3D%2FNp57DIbHY4rRzNxDUI9FTA%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBI4iNdQAPkF5MohffEbDGVjLbwyEamjWPM=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0Mzc0/98Zr9ilff8cSViFFEL5DFQ==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774374&utm_content=CamID6597069774374 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Thu Jul 10 19:49:42 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 00:49:42 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 10 19:54:52 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 19:54:52 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind What happens with -ksp_monitor on both cases? Same convergence history or different? Barry On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. > > If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. From hus003 at ucsd.edu Thu Jul 10 20:07:38 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 01:07:38 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU> If I run: ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason For DMDA_STENCIL_BOX, I get: 0 KSP Residual norm 17.5431 1 KSP Residual norm 3.23193 2 KSP Residual norm 0.399307 3 KSP Residual norm 0.0103842 4 KSP Residual norm 0.000124391 Linear solve converged due to CONVERGED_RTOL iterations 4 For DMDA_STENCIL_STAR, I get: 0 KSP Residual norm 60.7996 1 KSP Residual norm 13.7279 2 KSP Residual norm 4.15583 3 KSP Residual norm 0.430568 4 KSP Residual norm 0.0999957 5 KSP Residual norm 0.00260724 6 KSP Residual norm 0.00191392 7 KSP Residual norm 5.4681e-05 Linear solve converged due to CONVERGED_RTOL iterations 7 The history seems different. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 5:54 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind What happens with -ksp_monitor on both cases? Same convergence history or different? Barry On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. > > If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. From bsmith at mcs.anl.gov Thu Jul 10 20:14:18 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 20:14:18 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. What is exactly your finite difference stencil? How many degrees of freedom per point? Barry On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: > If I run: > ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason > > For DMDA_STENCIL_BOX, I get: > > 0 KSP Residual norm 17.5431 > 1 KSP Residual norm 3.23193 > 2 KSP Residual norm 0.399307 > 3 KSP Residual norm 0.0103842 > 4 KSP Residual norm 0.000124391 > Linear solve converged due to CONVERGED_RTOL iterations 4 > > For DMDA_STENCIL_STAR, I get: > > 0 KSP Residual norm 60.7996 > 1 KSP Residual norm 13.7279 > 2 KSP Residual norm 4.15583 > 3 KSP Residual norm 0.430568 > 4 KSP Residual norm 0.0999957 > 5 KSP Residual norm 0.00260724 > 6 KSP Residual norm 0.00191392 > 7 KSP Residual norm 5.4681e-05 > Linear solve converged due to CONVERGED_RTOL iterations 7 > > The history seems different. > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 5:54 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > What happens with -ksp_monitor on both cases? Same convergence history or different? > > Barry > > > > On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > >> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >> >> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >> >> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. > From hus003 at ucsd.edu Thu Jul 10 20:16:43 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 01:16:43 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CADB6@XMAIL-MBX-BH1.AD.UCSD.EDU> I don't know how to use valgrind. I followed your link to add these lines into the makefile: runtest29: -@${MPIEXEC} -valgrind -n 1 test_29 -malloc off -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason However when I run make runtest29, it gives me errors: [mpiexec at Huis-MacBook-Pro.local] match_arg (./utils/args/args.c:160): unrecognized argument valgrind [mpiexec at Huis-MacBook-Pro.local] HYDU_parse_array (./utils/args/args.c:175): argument matching returned error [mpiexec at Huis-MacBook-Pro.local] parse_args (./ui/mpich/utils.c:1609): error parsing input array [mpiexec at Huis-MacBook-Pro.local] HYD_uii_mpx_get_parameters (./ui/mpich/utils.c:1660): unable to parse user arguments [mpiexec at Huis-MacBook-Pro.local] main (./ui/mpich/mpiexec.c:153): error parsing parameters make: [runtest29] Error 255 (ignored) ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 5:54 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind What happens with -ksp_monitor on both cases? Same convergence history or different? Barry On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. > > If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. From bsmith at mcs.anl.gov Thu Jul 10 20:23:43 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 20:23:43 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CADB6@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADB6@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <5D4730FC-4964-49C7-82BF-E9E07DCF30D0@mcs.anl.gov> The directions are there ? run with valgrind using: ${PETSC_DIR}/bin/petscmpiexec -valgrind -n NPROC PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS ? or invoke valgrind directly with: mpiexec -n NPROC valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS you used 1/2 of the first direction and 1/2 of the second direction. So Use either the first or the second, don?t mix them together. Note the first one has ${PETSC_DIR}/bin/petscmpiexec instead of mpiexec. Barry On Jul 10, 2014, at 8:16 PM, Sun, Hui wrote: > I don't know how to use valgrind. I followed your link to add these lines into the makefile: > runtest29: > -@${MPIEXEC} -valgrind -n 1 test_29 -malloc off -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason > > However when I run make runtest29, it gives me errors: > [mpiexec at Huis-MacBook-Pro.local] match_arg (./utils/args/args.c:160): unrecognized argument valgrind > [mpiexec at Huis-MacBook-Pro.local] HYDU_parse_array (./utils/args/args.c:175): argument matching returned error > [mpiexec at Huis-MacBook-Pro.local] parse_args (./ui/mpich/utils.c:1609): error parsing input array > [mpiexec at Huis-MacBook-Pro.local] HYD_uii_mpx_get_parameters (./ui/mpich/utils.c:1660): unable to parse user arguments > [mpiexec at Huis-MacBook-Pro.local] main (./ui/mpich/mpiexec.c:153): error parsing parameters > make: [runtest29] Error 255 (ignored) > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 5:54 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > What happens with -ksp_monitor on both cases? Same convergence history or different? > > Barry > > > > On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > >> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >> >> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >> >> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. > From hus003 at ucsd.edu Thu Jul 10 20:46:37 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 01:46:37 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 6:14 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. What is exactly your finite difference stencil? How many degrees of freedom per point? Barry On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: > If I run: > ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason > > For DMDA_STENCIL_BOX, I get: > > 0 KSP Residual norm 17.5431 > 1 KSP Residual norm 3.23193 > 2 KSP Residual norm 0.399307 > 3 KSP Residual norm 0.0103842 > 4 KSP Residual norm 0.000124391 > Linear solve converged due to CONVERGED_RTOL iterations 4 > > For DMDA_STENCIL_STAR, I get: > > 0 KSP Residual norm 60.7996 > 1 KSP Residual norm 13.7279 > 2 KSP Residual norm 4.15583 > 3 KSP Residual norm 0.430568 > 4 KSP Residual norm 0.0999957 > 5 KSP Residual norm 0.00260724 > 6 KSP Residual norm 0.00191392 > 7 KSP Residual norm 5.4681e-05 > Linear solve converged due to CONVERGED_RTOL iterations 7 > > The history seems different. > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 5:54 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > What happens with -ksp_monitor on both cases? Same convergence history or different? > > Barry > > > > On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: > >> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >> >> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >> >> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. > -------------- next part -------------- A non-text attachment was scrubbed... Name: test_29_cp.c Type: application/octet-stream Size: 19911 bytes Desc: test_29_cp.c URL: From bsmith at mcs.anl.gov Thu Jul 10 21:46:14 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 21:46:14 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally >>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. Barry On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 6:14 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. > > What is exactly your finite difference stencil? How many degrees of freedom per point? > > > Barry > > On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: > >> If I run: >> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >> >> For DMDA_STENCIL_BOX, I get: >> >> 0 KSP Residual norm 17.5431 >> 1 KSP Residual norm 3.23193 >> 2 KSP Residual norm 0.399307 >> 3 KSP Residual norm 0.0103842 >> 4 KSP Residual norm 0.000124391 >> Linear solve converged due to CONVERGED_RTOL iterations 4 >> >> For DMDA_STENCIL_STAR, I get: >> >> 0 KSP Residual norm 60.7996 >> 1 KSP Residual norm 13.7279 >> 2 KSP Residual norm 4.15583 >> 3 KSP Residual norm 0.430568 >> 4 KSP Residual norm 0.0999957 >> 5 KSP Residual norm 0.00260724 >> 6 KSP Residual norm 0.00191392 >> 7 KSP Residual norm 5.4681e-05 >> Linear solve converged due to CONVERGED_RTOL iterations 7 >> >> The history seems different. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 5:54 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> What happens with -ksp_monitor on both cases? Same convergence history or different? >> >> Barry >> >> >> >> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >> >>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>> >>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>> >>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >> > > From hus003 at ucsd.edu Thu Jul 10 22:05:06 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 03:05:06 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CADE1@XMAIL-MBX-BH1.AD.UCSD.EDU> If you run this: ./test_29_cp -da_grid_x 41 -da_grid_y 41 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason Then DMDA_STENTIL_STAR gives you convergence to the correct solution in 45 steps, while DMDA_STENTIL_BOX does not converge to the correct solution even after 1000 steps. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 7:46 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally >>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. Barry On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 6:14 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. > > What is exactly your finite difference stencil? How many degrees of freedom per point? > > > Barry > > On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: > >> If I run: >> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >> >> For DMDA_STENCIL_BOX, I get: >> >> 0 KSP Residual norm 17.5431 >> 1 KSP Residual norm 3.23193 >> 2 KSP Residual norm 0.399307 >> 3 KSP Residual norm 0.0103842 >> 4 KSP Residual norm 0.000124391 >> Linear solve converged due to CONVERGED_RTOL iterations 4 >> >> For DMDA_STENCIL_STAR, I get: >> >> 0 KSP Residual norm 60.7996 >> 1 KSP Residual norm 13.7279 >> 2 KSP Residual norm 4.15583 >> 3 KSP Residual norm 0.430568 >> 4 KSP Residual norm 0.0999957 >> 5 KSP Residual norm 0.00260724 >> 6 KSP Residual norm 0.00191392 >> 7 KSP Residual norm 5.4681e-05 >> Linear solve converged due to CONVERGED_RTOL iterations 7 >> >> The history seems different. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 5:54 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> What happens with -ksp_monitor on both cases? Same convergence history or different? >> >> Barry >> >> >> >> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >> >>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>> >>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>> >>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >> > > From hus003 at ucsd.edu Thu Jul 10 22:07:31 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 03:07:31 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CADEC@XMAIL-MBX-BH1.AD.UCSD.EDU> I have also noticed those unnecessary zeros, in both cases DMDA_STENTIL_BOX and DMDA_STENTIL_STAR there are many unnecessary zeros. Do you have any idea how to get rid of those entries? ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 7:46 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally >>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. Barry On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 6:14 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. > > What is exactly your finite difference stencil? How many degrees of freedom per point? > > > Barry > > On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: > >> If I run: >> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >> >> For DMDA_STENCIL_BOX, I get: >> >> 0 KSP Residual norm 17.5431 >> 1 KSP Residual norm 3.23193 >> 2 KSP Residual norm 0.399307 >> 3 KSP Residual norm 0.0103842 >> 4 KSP Residual norm 0.000124391 >> Linear solve converged due to CONVERGED_RTOL iterations 4 >> >> For DMDA_STENCIL_STAR, I get: >> >> 0 KSP Residual norm 60.7996 >> 1 KSP Residual norm 13.7279 >> 2 KSP Residual norm 4.15583 >> 3 KSP Residual norm 0.430568 >> 4 KSP Residual norm 0.0999957 >> 5 KSP Residual norm 0.00260724 >> 6 KSP Residual norm 0.00191392 >> 7 KSP Residual norm 5.4681e-05 >> Linear solve converged due to CONVERGED_RTOL iterations 7 >> >> The history seems different. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 5:54 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> What happens with -ksp_monitor on both cases? Same convergence history or different? >> >> Barry >> >> >> >> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >> >>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>> >>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>> >>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >> > > From bsmith at mcs.anl.gov Thu Jul 10 22:14:35 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 22:14:35 -0500 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <53BD2236.7030100@physik.uni-muenchen.de> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> <53BC1375.9070701@physik.uni-muenchen.de> <40960837-6B62-4F8B-952F-54B548FD454B@mcs.anl.gov> <53BD2236.7030100@physik.uni-muenchen.de> Message-ID: On Jul 9, 2014, at 6:06 AM, Fabian.Jakub wrote: > If I do, the generated file is empty and the just states the solvers... > > cat binaryoutput: > {Wgmres{Vilu > > Do I have an error in my Petscbuild? No this was my mistake I told you the wrong directions > This by the way happens with all matrices, even tried a couple of examples from petsc src. > > Anyway, I uploaded the matrix dump (-mat_view binary) > I hope this helps. > Thanks. I have reproduced the problem with your matrix. I will work on figuring out how to fix it. From the code /* very pessimistic but could be too low in theory */ so likely you just provided a matrix that Mark?s original algorithm could not handle. Barry > > To be thorough, I further put up a second file, with 2 matrices, > The second matrix is the same as before (the one that is not working with gamg). > The first one with dof 8 is solved and the solution is somewhat the source term for the second. > I just thought I would append this one too because it does indeed work with gamg. > Maybe that gives any hints. > > I really appreciate you looking into this. > > Thank you! > > Sincerely, > Fabian > > > > Am 09.07.2014 07:37, schrieb Barry Smith: >> Can you run with -ksp_view_pre binary and email the file binaryoutput >> >> With this we should be able to reproduce the problem and debug it. >> >> Thanks >> >> >> Barry >> >> On Jul 8, 2014, at 10:51 AM, Fabian.Jakub wrote: >> >>> Thank you very much for the swift reply! >>> >>> I am running >>> git describe: v3.5-25-g0ace994 >>> and configured with >>> >>> ./configure \ >>> --with-cc=$(which mpicc) \ >>> --with-fc=$(which mpif90) \ >>> --with-cxx=$(which mpic++) \ >>> --with-fortran \ >>> --with-shared-libraries \ >>> --with-hdf5 \ >>> --with-hdf5-dir="$HDF5_DIR" \ >>> --with-cmake=$(which cmake) \ >>> \ >>> >>> I run the model with: >>> mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info >>> >>> Should I append the output when creating the DMDA? >>> For the KSP setup it says: >>> >>> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter >>> [0] VecScatterCreate(): General case: MPI to Seq >>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used >>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 >>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 >>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. >>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 >>> Setup KSP >>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 >>> [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 >>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 >>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 >>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 >>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 >>> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [1]PETSC ERROR: Argument out of range >>> [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc >>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 >>> [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 >>> [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake >>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c >>> [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c >>> [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c >>> [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c >>> [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c >>> [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc >>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 >>> [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 >>> [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake >>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c >>> [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c >>> [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 63. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> ------------------------------------------------------------------------- >>> >>> Am 08.07.2014 17:14, schrieb Barry Smith: >>>> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. >>>> >>>> Barry >>>> >>>> >>>> >>>> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: >>>> >>>>> Hi, >>>>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. >>>>> >>>>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. >>>>> >>>>> The code to setup the matrix is like this: >>>>> >>>>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) >>>>> call DMCreateMatrix(C%da, A, ierr) >>>>> call MatSetFromOptions(A,ierr) >>>>> >>>>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) >>>>> >>>>> insert matrix values & assemble >>>>> >>>>> then solve. >>>>> >>>>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. >>>>> However if I use gamg I get an allocation error when it tries to create a coarse grid. >>>>> >>>>> If I use the DMDA preallocation, it works but uses way more memory... >>>>> >>>>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? >>>>> Is that even the problem or am I missing something? >>>>> >>>>> Thank you so very much. >>>>> >>>>> Sincerely, >>>>> Fabian >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> > From bsmith at mcs.anl.gov Thu Jul 10 22:34:53 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 22:34:53 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CADEC@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADEC@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Jul 10, 2014, at 10:07 PM, Sun, Hui wrote: > I have also noticed those unnecessary zeros, in both cases DMDA_STENTIL_BOX and DMDA_STENTIL_STAR there are many unnecessary zeros. Do you have any idea how to get rid of those entries? These zeros are for all the locations allowed by the stencil (assuming a full stencil for every variable). The only way to get rid of them is to use a custom matrix allocation. The box stencil will have many extra of these zeros since it includes a lot of ?coupling? in the matrix that doesn?t exist. Barry > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 7:46 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally > >>>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. > > Barry > > On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > >> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 6:14 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. >> >> What is exactly your finite difference stencil? How many degrees of freedom per point? >> >> >> Barry >> >> On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: >> >>> If I run: >>> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >>> >>> For DMDA_STENCIL_BOX, I get: >>> >>> 0 KSP Residual norm 17.5431 >>> 1 KSP Residual norm 3.23193 >>> 2 KSP Residual norm 0.399307 >>> 3 KSP Residual norm 0.0103842 >>> 4 KSP Residual norm 0.000124391 >>> Linear solve converged due to CONVERGED_RTOL iterations 4 >>> >>> For DMDA_STENCIL_STAR, I get: >>> >>> 0 KSP Residual norm 60.7996 >>> 1 KSP Residual norm 13.7279 >>> 2 KSP Residual norm 4.15583 >>> 3 KSP Residual norm 0.430568 >>> 4 KSP Residual norm 0.0999957 >>> 5 KSP Residual norm 0.00260724 >>> 6 KSP Residual norm 0.00191392 >>> 7 KSP Residual norm 5.4681e-05 >>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>> >>> The history seems different. >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Thursday, July 10, 2014 5:54 PM >>> To: Sun, Hui >>> Cc: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>> >>> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> What happens with -ksp_monitor on both cases? Same convergence history or different? >>> >>> Barry >>> >>> >>> >>> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >>> >>>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>>> >>>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>>> >>>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >>> >> >> > From bsmith at mcs.anl.gov Thu Jul 10 22:41:40 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 22:41:40 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CADE1@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADE1@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <9352CD49-3F04-4C59-8F7C-83AEB28B2E8E@mcs.anl.gov> On Jul 10, 2014, at 10:05 PM, Sun, Hui wrote: > If you run this: > ./test_29_cp -da_grid_x 41 -da_grid_y 41 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason > > Then DMDA_STENTIL_STAR gives you convergence to the correct solution in 45 steps, while DMDA_STENTIL_BOX does not converge to the correct solution even after 1000 steps. I saved the matrices and right hand sides for both of these runs (box and star). They are exactly the same. So you are doing everything right. I also ran both box and star with -pc_type jacobi and -ksp_type gmres and got essentially the exact same convergence (as one would expect in exact arithmetic). The combination of bcgs and ilu(0) is just messing you up for box stencil. Intuitively one would expect that putting extra zeros into the matrix would result in a ?better? ilu(0) preconditioner, but it need not be the case! I would recommend you avoid ilu as the preconditioner for your problem. Likely you should use PCFIELDSPLIT Barry > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 7:46 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally > >>>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. > > Barry > > On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > >> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 6:14 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. >> >> What is exactly your finite difference stencil? How many degrees of freedom per point? >> >> >> Barry >> >> On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: >> >>> If I run: >>> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >>> >>> For DMDA_STENCIL_BOX, I get: >>> >>> 0 KSP Residual norm 17.5431 >>> 1 KSP Residual norm 3.23193 >>> 2 KSP Residual norm 0.399307 >>> 3 KSP Residual norm 0.0103842 >>> 4 KSP Residual norm 0.000124391 >>> Linear solve converged due to CONVERGED_RTOL iterations 4 >>> >>> For DMDA_STENCIL_STAR, I get: >>> >>> 0 KSP Residual norm 60.7996 >>> 1 KSP Residual norm 13.7279 >>> 2 KSP Residual norm 4.15583 >>> 3 KSP Residual norm 0.430568 >>> 4 KSP Residual norm 0.0999957 >>> 5 KSP Residual norm 0.00260724 >>> 6 KSP Residual norm 0.00191392 >>> 7 KSP Residual norm 5.4681e-05 >>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>> >>> The history seems different. >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Thursday, July 10, 2014 5:54 PM >>> To: Sun, Hui >>> Cc: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>> >>> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> What happens with -ksp_monitor on both cases? Same convergence history or different? >>> >>> Barry >>> >>> >>> >>> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >>> >>>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>>> >>>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>>> >>>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >>> >> >> > From hus003 at ucsd.edu Thu Jul 10 22:48:29 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 03:48:29 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADEC@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CAE08@XMAIL-MBX-BH1.AD.UCSD.EDU> I see. Should I use the command MatMPIAIJSetPreallocation to preallocate the memory? By the way, it seems that using DMDA_STENTIL_BOX with stencil width 3 will have 4 times as much memory allocation than using DMDA_STENTIL_STAR, is that supposed to be the problem? However, here is another code, 2D poisson, with 3 degrees of freedom on each node. I tried the same thing, both DMDA_STENTIL_BOX and DMDA_STENTIL_STAR with stencil width 3. But this one doesn't seem to produce the same kind of problem. Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 8:34 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX On Jul 10, 2014, at 10:07 PM, Sun, Hui wrote: > I have also noticed those unnecessary zeros, in both cases DMDA_STENTIL_BOX and DMDA_STENTIL_STAR there are many unnecessary zeros. Do you have any idea how to get rid of those entries? These zeros are for all the locations allowed by the stencil (assuming a full stencil for every variable). The only way to get rid of them is to use a custom matrix allocation. The box stencil will have many extra of these zeros since it includes a lot of ?coupling? in the matrix that doesn?t exist. Barry > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 7:46 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally > >>>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. > > Barry > > On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > >> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 6:14 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. >> >> What is exactly your finite difference stencil? How many degrees of freedom per point? >> >> >> Barry >> >> On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: >> >>> If I run: >>> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >>> >>> For DMDA_STENCIL_BOX, I get: >>> >>> 0 KSP Residual norm 17.5431 >>> 1 KSP Residual norm 3.23193 >>> 2 KSP Residual norm 0.399307 >>> 3 KSP Residual norm 0.0103842 >>> 4 KSP Residual norm 0.000124391 >>> Linear solve converged due to CONVERGED_RTOL iterations 4 >>> >>> For DMDA_STENCIL_STAR, I get: >>> >>> 0 KSP Residual norm 60.7996 >>> 1 KSP Residual norm 13.7279 >>> 2 KSP Residual norm 4.15583 >>> 3 KSP Residual norm 0.430568 >>> 4 KSP Residual norm 0.0999957 >>> 5 KSP Residual norm 0.00260724 >>> 6 KSP Residual norm 0.00191392 >>> 7 KSP Residual norm 5.4681e-05 >>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>> >>> The history seems different. >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Thursday, July 10, 2014 5:54 PM >>> To: Sun, Hui >>> Cc: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>> >>> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> What happens with -ksp_monitor on both cases? Same convergence history or different? >>> >>> Barry >>> >>> >>> >>> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >>> >>>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>>> >>>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>>> >>>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >>> >> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: 2D_poisson_3dof.c Type: application/octet-stream Size: 7698 bytes Desc: 2D_poisson_3dof.c URL: From hus003 at ucsd.edu Thu Jul 10 23:01:11 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 11 Jul 2014 04:01:11 +0000 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <9352CD49-3F04-4C59-8F7C-83AEB28B2E8E@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADE1@XMAIL-MBX-BH1.AD.UCSD.EDU>, <9352CD49-3F04-4C59-8F7C-83AEB28B2E8E@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CAE1E@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you very much for your time, Barry. I really appreciate it! Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, July 10, 2014 8:41 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX On Jul 10, 2014, at 10:05 PM, Sun, Hui wrote: > If you run this: > ./test_29_cp -da_grid_x 41 -da_grid_y 41 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason > > Then DMDA_STENTIL_STAR gives you convergence to the correct solution in 45 steps, while DMDA_STENTIL_BOX does not converge to the correct solution even after 1000 steps. I saved the matrices and right hand sides for both of these runs (box and star). They are exactly the same. So you are doing everything right. I also ran both box and star with -pc_type jacobi and -ksp_type gmres and got essentially the exact same convergence (as one would expect in exact arithmetic). The combination of bcgs and ilu(0) is just messing you up for box stencil. Intuitively one would expect that putting extra zeros into the matrix would result in a ?better? ilu(0) preconditioner, but it need not be the case! I would recommend you avoid ilu as the preconditioner for your problem. Likely you should use PCFIELDSPLIT Barry > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 7:46 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally > >>>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. > > can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. > > Barry > > On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: > >> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 6:14 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. >> >> What is exactly your finite difference stencil? How many degrees of freedom per point? >> >> >> Barry >> >> On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: >> >>> If I run: >>> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >>> >>> For DMDA_STENCIL_BOX, I get: >>> >>> 0 KSP Residual norm 17.5431 >>> 1 KSP Residual norm 3.23193 >>> 2 KSP Residual norm 0.399307 >>> 3 KSP Residual norm 0.0103842 >>> 4 KSP Residual norm 0.000124391 >>> Linear solve converged due to CONVERGED_RTOL iterations 4 >>> >>> For DMDA_STENCIL_STAR, I get: >>> >>> 0 KSP Residual norm 60.7996 >>> 1 KSP Residual norm 13.7279 >>> 2 KSP Residual norm 4.15583 >>> 3 KSP Residual norm 0.430568 >>> 4 KSP Residual norm 0.0999957 >>> 5 KSP Residual norm 0.00260724 >>> 6 KSP Residual norm 0.00191392 >>> 7 KSP Residual norm 5.4681e-05 >>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>> >>> The history seems different. >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Thursday, July 10, 2014 5:54 PM >>> To: Sun, Hui >>> Cc: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>> >>> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> What happens with -ksp_monitor on both cases? Same convergence history or different? >>> >>> Barry >>> >>> >>> >>> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >>> >>>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>>> >>>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>>> >>>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >>> >> >> > From bsmith at mcs.anl.gov Thu Jul 10 23:09:39 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Jul 2014 23:09:39 -0500 Subject: [petsc-users] a strange bug on using DMDA_STENCIL_BOX In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CAE08@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6C8D91@XMAIL-MBX-BH1.AD.UCSD.EDU>, <58A30DC2-4163-4EB6-A208-EBB3F6359382@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6C9DA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, <35AEA6CF-C118-4737-A72F-2465D41DB676@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADCC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <42E9664F-962E-4EE6-BB21-73C8E3B1CE17@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB6CADEC@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB6CAE08@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <67644E5D-17E9-4316-BEBF-36E5A7ECFD8C@mcs.anl.gov> On Jul 10, 2014, at 10:48 PM, Sun, Hui wrote: > I see. Should I use the command MatMPIAIJSetPreallocation to preallocate the memory? > > By the way, it seems that using DMDA_STENTIL_BOX with stencil width 3 will have 4 times as much memory allocation than using DMDA_STENTIL_STAR, is that supposed to be the problem? It is not really a ?problem?. Since the box stencil includes the corners it naturally will use many more locations in the vector. You should just use the stencil that matches the stencil you are using so if you are using a stencil that only sticks out directly on each side and doesn?t use the corners you should use the star stencil. It uses much less memory. If your discretization requires using the corners then you need the box stencil. > > However, here is another code, 2D poisson, with 3 degrees of freedom on each node. I tried the same thing, both DMDA_STENTIL_BOX and DMDA_STENTIL_STAR with stencil width 3. But this one doesn't seem to produce the same kind of problem. Yes ILU(0) is very problem dependent, it is mostly used for Poisson type problems (though it is much slower for large problems than using good algorithms like multigrid). Barry > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Thursday, July 10, 2014 8:34 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX > > On Jul 10, 2014, at 10:07 PM, Sun, Hui wrote: > >> I have also noticed those unnecessary zeros, in both cases DMDA_STENTIL_BOX and DMDA_STENTIL_STAR there are many unnecessary zeros. Do you have any idea how to get rid of those entries? > > These zeros are for all the locations allowed by the stencil (assuming a full stencil for every variable). The only way to get rid of them is to use a custom matrix allocation. > > The box stencil will have many extra of these zeros since it includes a lot of ?coupling? in the matrix that doesn?t exist. > > Barry > >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Thursday, July 10, 2014 7:46 PM >> To: Sun, Hui >> Cc: petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >> >> Hmm. Because the box stencil puts extra zeros in the original matrix (due to the ?extra? stencil locations) ilu(0) will converge differently with the box stencil than the star stencil with the linear solver. Thus one will actually get a different convergence history. With Jacobi (which doesn?t depend on the non-zero structure of the matrix) the convergence won?t change. I checked with Jacobi and -da_refine 1 and they converge the same. Are you sure there is an issue with box stencil? You wrote originally >> >>>>> However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >> >> can you explain to me how to reproduce this ?does not converge, the residue \|Ax-b\|_2 remains large.? business. >> >> Barry >> >> On Jul 10, 2014, at 8:46 PM, Sun, Hui wrote: >> >>> Thanks for the hint. That is a possible bug. However, I still haven't caught the bug after rethinking about it. I've got three degrees of freedom on each node, MAC grid, and std 5 points laplacian, central differencing for derivatives. Here I've also attached my code. >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Thursday, July 10, 2014 6:14 PM >>> To: Sun, Hui >>> Cc: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>> >>> Perhaps the code is not correct for the star stencil? That is perhaps you are actually setting values in the ?corner? locations that are not being communicated with stencil but are communicated with star? If your finite difference stencil involves the ?corner? locations then you need to always use the box stencil type. >>> >>> What is exactly your finite difference stencil? How many degrees of freedom per point? >>> >>> >>> Barry >>> >>> On Jul 10, 2014, at 8:07 PM, Sun, Hui wrote: >>> >>>> If I run: >>>> ./test_29 -da_grid_x 6 -da_grid_y 6 -ksp_type bcgs -ksp_monitor_short -ksp_converged_reason >>>> >>>> For DMDA_STENCIL_BOX, I get: >>>> >>>> 0 KSP Residual norm 17.5431 >>>> 1 KSP Residual norm 3.23193 >>>> 2 KSP Residual norm 0.399307 >>>> 3 KSP Residual norm 0.0103842 >>>> 4 KSP Residual norm 0.000124391 >>>> Linear solve converged due to CONVERGED_RTOL iterations 4 >>>> >>>> For DMDA_STENCIL_STAR, I get: >>>> >>>> 0 KSP Residual norm 60.7996 >>>> 1 KSP Residual norm 13.7279 >>>> 2 KSP Residual norm 4.15583 >>>> 3 KSP Residual norm 0.430568 >>>> 4 KSP Residual norm 0.0999957 >>>> 5 KSP Residual norm 0.00260724 >>>> 6 KSP Residual norm 0.00191392 >>>> 7 KSP Residual norm 5.4681e-05 >>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>> >>>> The history seems different. >>>> >>>> >>>> ________________________________________ >>>> From: Barry Smith [bsmith at mcs.anl.gov] >>>> Sent: Thursday, July 10, 2014 5:54 PM >>>> To: Sun, Hui >>>> Cc: petsc-users at mcs.anl.gov >>>> Subject: Re: [petsc-users] a strange bug on using DMDA_STENCIL_BOX >>>> >>>> Your interpretation is correct. If DMDA_STENTIL_STAR works then one absolutely expects that DMDA_STENTIL_BOX would produce the same answer. If you are running with linux could you run with valgrind to see if there is some strange memory corruption or lack of initialization http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>> >>>> What happens with -ksp_monitor on both cases? Same convergence history or different? >>>> >>>> Barry >>>> >>>> >>>> >>>> On Jul 10, 2014, at 7:49 PM, Sun, Hui wrote: >>>> >>>>> I'm using KSP for 2D Navier Stokes, finite difference. I have DM object created with DMDA_STENTIL_STAR. I'm only using one single process to run the code. It works fine, the linear system converges to the correct solution with correct order, as expected. However, if I change it to DMDA_STENTIL_BOX, it does not converge, the residue \|Ax-b\|_2 remains large. >>>>> >>>>> I have checked that the rhs for both cases are exactly the same, and the matrices also seem to be the same because I have checked the output of A*y, where y is the analytic solution, and both cases give exactly the same answer. If the linear system is exactly the same, then it seems very strange to me that DMDA_STENTIL_STAR works while DMDA_STENTIL_BOX does not work, for I use the same solver, there shouldn't be any difference. >>>>> >>>>> If you happen to encounter this kind of bug before and have some idea what's going on, please let me know. Thank you very much! If you want to look at the code I've written, feel free to ask. >>>> >>> >>> >> > > <2D_poisson_3dof.c> From milo at zeusdvds.com Fri Jul 11 08:36:28 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Fri, 11 Jul 2014 06:36:28 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 11, 2014 Message-ID: <53bfe85cdcb0_326f11fde9c577760cc@prd-nresque01.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQ0NDE_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJLT0c5ZFFBUGtGNUNzQ3pDaWxwbUFCYUItS3FBRUpCWW8lM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpRME5ERV9ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGNlp4OWhkaFpFcGlpbldlUTJiaGxQZyUzRCUzRA==/o4Ov8h1vjRryxWjW_CJzBw==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs The Law's Lash (1928) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWxhd3MtbGFzaC0xOTI4LWR2ZC8=/7eeiS-a9gUK6xmAUgohUCA== )South of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/ibDbtY5PmBuhVeo3htBpnA== )Diary of a Hitman (1991) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZGlhcnktb2YtYS1oaXRtYW4tMTk5MS1kdmQv/AULxKiafBGQJrW7fV94pIg== )Razzia sur le chnouf (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcmF6emlhLXN1ci1sYS1jaG5vdWYtMTk1NS1kdmQv/IdTh071BdFhWXxG_56og9Q== )Let's Get Lost (1988) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGV0cy1nZXQtbG9zdC0xOTg4LWR2ZC8=/KKE8XsEuCJH59v13wrTBNQ== )The Showdown (1950) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNob3dkb3duLTE5NTAtZHZkLw==/4sBcfqer4jeNqpeWKyo4rg== )The Spoilers (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXNwb2lsZXJzLTE5NTUtZHZkLw==/fFZYy6XlVsluicPx38xdLA== )Escort Girl (1941) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZXNjb3J0LWdpcmwtMTk0MS1kdmQv/rqgcM7QFEXYfg7dRw_yoIQ== )Fast Forward (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZmFzdC1mb3J3YXJkLTE5ODUtZHZkLw==/TpVRAviZ7DqZjIuH1B2u8g== )Rocky IV (1985) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcm9ja3ktaXYtMTk4NS1kdmQv/P69GE_z91lDY18D7Gl1lpQ== ) Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/lfUHVkMOodySX_irH16BQw== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/aS6MnteNUY7wDljtKeRYPg== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/xfpktBvlxjOqP1aIDeTGQQ== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/qgfWC464Qqda0AsbbojrFQ== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/5EwyF_XSQ_f_EMzpF-5h2w== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/l3ir07XyjONEu2sCJ-hsQA== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/sDCFMWiTilMjj-l9Cw4OaA== )? All orders ship within 24 hours and all shipping is FREE. Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/zAVrumWKG2zgQqWZRGdALg==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774441&sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQ0NDE%3D%2FJynM2AeBa8CP4OfuJ4UOXw%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBKOG9dQAPkF5CsCzCilpmABaB-KqAEJBYo=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0NDQx/Fsikj3aJsc3DxMXcgY80oA==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774441&utm_content=CamID6597069774441 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.miah at qmul.ac.uk Fri Jul 11 09:20:18 2014 From: w.miah at qmul.ac.uk (Wadud Miah) Date: Fri, 11 Jul 2014 14:20:18 +0000 Subject: [petsc-users] PETSc 3.5.0 build with Hypre Message-ID: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Hello, I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: ./configure \ --with-hypre=1 \ --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ --with-fortran-interfaces=1 \ --prefix=$HOME/petsc-3.5.0 \ --with-shared-libraries=1 \ --with-mpi=1 \ --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so But I keep getting the following error message during configure: UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. Regards, ------------------------------------------- Wadud Miah Research Computing Services (HPC) 020 7882 8393 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 11 09:26:50 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 11 Jul 2014 16:26:50 +0200 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: On Fri, Jul 11, 2014 at 4:20 PM, Wadud Miah wrote: > Hello, > > > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following > configure options: > > > > ./configure \ > > --with-hypre=1 \ > > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ > This argument is not the directory, but an actual list of libraries. I think you really want --with-hypre-dir=/data/home/eew918/hypre-2.9.0b instead of the include/dir options. Thanks Matt > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 > -L/data/home/eew918/hypre-2.9.0b/lib" \ > > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > --with-fortran-interfaces=1 \ > > --prefix=$HOME/petsc-3.5.0 \ > > --with-shared-libraries=1 \ > > --with-mpi=1 \ > > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so > > > > But I keep getting the following error message during configure: > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > > ------------------------------------------------------------------------------- > > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not > work > > > > I see the compilation command with the correct library path > (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l > switch. > > > > Regards, > > > > ------------------------------------------- > > Wadud Miah > > Research Computing Services (HPC) > > 020 7882 8393 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jul 11 09:28:55 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 11 Jul 2014 09:28:55 -0500 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: On Fri, 11 Jul 2014, Wadud Miah wrote: > Hello, > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: > > ./configure \ > --with-hypre=1 \ > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib -lHYPRE" You are probably better off using --download-hypre instead > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ You can use --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead of the above 3 options > --with-fortran-interfaces=1 \ > --prefix=$HOME/petsc-3.5.0 \ > --with-shared-libraries=1 \ > --with-mpi=1 \ > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 [instead of the above 3 options Satish > > But I keep getting the following error message during configure: > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work > > I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. > > Regards, > > ------------------------------------------- > Wadud Miah > Research Computing Services (HPC) > 020 7882 8393 > > From knepley at gmail.com Fri Jul 11 09:38:31 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 11 Jul 2014 16:38:31 +0200 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: On Fri, Jul 11, 2014 at 4:28 PM, Satish Balay wrote: > On Fri, 11 Jul 2014, Wadud Miah wrote: > > > Hello, > > > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following > configure options: > > > > ./configure \ > > --with-hypre=1 \ > > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ > > This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib > -lHYPRE" > > You are probably better off using --download-hypre instead > > > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 > -L/data/home/eew918/hypre-2.9.0b/lib" \ > > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > You can use > --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead > of the above 3 options > It should be -dir, not -lib. Matt > > --with-fortran-interfaces=1 \ > > --prefix=$HOME/petsc-3.5.0 \ > > --with-shared-libraries=1 \ > > --with-mpi=1 \ > > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so > > You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 > [instead of the above 3 options > > Satish > > > > > But I keep getting the following error message during configure: > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > > ------------------------------------------------------------------------------- > > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not > work > > > > I see the compilation command with the correct library path > (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l > switch. > > > > Regards, > > > > ------------------------------------------- > > Wadud Miah > > Research Computing Services (HPC) > > 020 7882 8393 > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.miah at qmul.ac.uk Fri Jul 11 09:44:33 2014 From: w.miah at qmul.ac.uk (Wadud Miah) Date: Fri, 11 Jul 2014 14:44:33 +0000 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: Hello Satish, Thanks for the quick reply. I am also getting the following error when I use the --with-fortran-datatypes=1 flag: Cannot use generated fortran interface definitions with fortran datatypes Any help will be greatly appreciated. Regards, Wadud. -----Original Message----- From: Satish Balay [mailto:balay at mcs.anl.gov] Sent: 11 July 2014 15:29 To: Wadud Miah Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre On Fri, 11 Jul 2014, Wadud Miah wrote: > Hello, > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: > > ./configure \ > --with-hypre=1 \ > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib -lHYPRE" You are probably better off using --download-hypre instead > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ You can use --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead of the above 3 options > --with-fortran-interfaces=1 \ > --prefix=$HOME/petsc-3.5.0 \ > --with-shared-libraries=1 \ > --with-mpi=1 \ > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 [instead of the above 3 options Satish > > But I keep getting the following error message during configure: > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work > > I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. > > Regards, > > ------------------------------------------- > Wadud Miah > Research Computing Services (HPC) > 020 7882 8393 > > From balay at mcs.anl.gov Fri Jul 11 10:05:40 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 11 Jul 2014 10:05:40 -0500 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: On Fri, 11 Jul 2014, Wadud Miah wrote: > Hello Satish, > > Thanks for the quick reply. I am also getting the following error when I use the > --with-fortran-datatypes=1 flag: > > Cannot use generated fortran interface definitions with fortran datatypes > > Any help will be greatly appreciated. As the message says - you cannot use --with-fortran-datatypes=1 with --with-fortran-interfaces=1 Are you sure you need either of these options? Satish > > Regards, > Wadud. > > -----Original Message----- > From: Satish Balay [mailto:balay at mcs.anl.gov] > Sent: 11 July 2014 15:29 > To: Wadud Miah > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre > > On Fri, 11 Jul 2014, Wadud Miah wrote: > > > Hello, > > > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: > > > > ./configure \ > > --with-hypre=1 \ > > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ > > This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib -lHYPRE" > > You are probably better off using --download-hypre instead > > > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ > > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > You can use --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead of the above 3 options > > > --with-fortran-interfaces=1 \ > > --prefix=$HOME/petsc-3.5.0 \ > > --with-shared-libraries=1 \ > > --with-mpi=1 \ > > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so > > You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 [instead of the above 3 options > > Satish > > > > > But I keep getting the following error message during configure: > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work > > > > I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. > > > > Regards, > > > > ------------------------------------------- > > Wadud Miah > > Research Computing Services (HPC) > > 020 7882 8393 > > > > > > From w.miah at qmul.ac.uk Fri Jul 11 10:12:21 2014 From: w.miah at qmul.ac.uk (Wadud Miah) Date: Fri, 11 Jul 2014 15:12:21 +0000 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: <8420c12f65f54b4a99383457fb0d0c51@AM3PR07MB436.eurprd07.prod.outlook.com> Hi Satish, The description of --with-fortran-interfaces is: --with-fortran-interfaces= Generate Fortran interface definitions for PETSc functions current: 0 I would like to use PETSc in Fortran 90 programs, so I thought this switch will create Fortran wrappers. Is this the case or have I misunderstood it? Regards, Wadud. -----Original Message----- From: Satish Balay [mailto:balay at mcs.anl.gov] Sent: 11 July 2014 16:06 To: Wadud Miah Cc: petsc-users Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre On Fri, 11 Jul 2014, Wadud Miah wrote: > Hello Satish, > > Thanks for the quick reply. I am also getting the following error when I use the > --with-fortran-datatypes=1 flag: > > Cannot use generated fortran interface definitions with fortran datatypes > > Any help will be greatly appreciated. As the message says - you cannot use --with-fortran-datatypes=1 with --with-fortran-interfaces=1 Are you sure you need either of these options? Satish > > Regards, > Wadud. > > -----Original Message----- > From: Satish Balay [mailto:balay at mcs.anl.gov] > Sent: 11 July 2014 15:29 > To: Wadud Miah > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre > > On Fri, 11 Jul 2014, Wadud Miah wrote: > > > Hello, > > > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: > > > > ./configure \ > > --with-hypre=1 \ > > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ > > This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib -lHYPRE" > > You are probably better off using --download-hypre instead > > > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ > > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > You can use --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead of the above 3 options > > > --with-fortran-interfaces=1 \ > > --prefix=$HOME/petsc-3.5.0 \ > > --with-shared-libraries=1 \ > > --with-mpi=1 \ > > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so > > You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 [instead of the above 3 options > > Satish > > > > > But I keep getting the following error message during configure: > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work > > > > I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. > > > > Regards, > > > > ------------------------------------------- > > Wadud Miah > > Research Computing Services (HPC) > > 020 7882 8393 > > > > > > From balay at mcs.anl.gov Fri Jul 11 10:25:46 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 11 Jul 2014 10:25:46 -0500 Subject: [petsc-users] PETSc 3.5.0 build with Hypre In-Reply-To: <8420c12f65f54b4a99383457fb0d0c51@AM3PR07MB436.eurprd07.prod.outlook.com> References: <2b58ca521c454c5cade548e9b2695afa@AM3PR07MB436.eurprd07.prod.outlook.com> <8420c12f65f54b4a99383457fb0d0c51@AM3PR07MB436.eurprd07.prod.outlook.com> Message-ID: On Fri, 11 Jul 2014, Wadud Miah wrote: > Hi Satish, > > The description of --with-fortran-interfaces is: > > --with-fortran-interfaces= > Generate Fortran interface definitions for PETSc functions current: 0 > > I would like to use PETSc in Fortran 90 programs, so I thought this switch will create Fortran wrappers. Is this the case or have I misunderstood it? The fortran wrappers are always built [if configured with a fortran compiler] 'Fortran interface definitions' gives you extra type checking for the subroutine calls - when using a f90 compiler. However this code can have rough edges - so its not enabled by default. You can use it - and report if you encounter issues. >>>>>>> --with-fortran-datatypes= Declare PETSc objects in Fortran like type(Vec) instead of Vec current: 0 <<<<<<< This is experimental - to provide a bit more native f90 coding support [instead of preprocessing]. And it does not work with --with-fortran-interfaces=1 Satish > > Regards, > Wadud. > > -----Original Message----- > From: Satish Balay [mailto:balay at mcs.anl.gov] > Sent: 11 July 2014 16:06 > To: Wadud Miah > Cc: petsc-users > Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre > > On Fri, 11 Jul 2014, Wadud Miah wrote: > > > Hello Satish, > > > > Thanks for the quick reply. I am also getting the following error when I use the > > --with-fortran-datatypes=1 flag: > > > > Cannot use generated fortran interface definitions with fortran datatypes > > > > Any help will be greatly appreciated. > > As the message says - you cannot use --with-fortran-datatypes=1 with --with-fortran-interfaces=1 > > Are you sure you need either of these options? > > Satish > > > > > Regards, > > Wadud. > > > > -----Original Message----- > > From: Satish Balay [mailto:balay at mcs.anl.gov] > > Sent: 11 July 2014 15:29 > > To: Wadud Miah > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] PETSc 3.5.0 build with Hypre > > > > On Fri, 11 Jul 2014, Wadud Miah wrote: > > > > > Hello, > > > > > > I am trying to build PETSc 3.5.0 with Hypre 2.9.0b with the following configure options: > > > > > > ./configure \ > > > --with-hypre=1 \ > > > --with-hypre-lib=/data/home/eew918/hypre-2.9.0b/lib \ > > > > This should be: --with-hypre-lib="-L/data/home/eew918/hypre-2.9.0b/lib -lHYPRE" > > > > You are probably better off using --download-hypre instead > > > > > --with-hypre-include=/data/home/eew918/hypre-2.9.0b/include \ > > > --LDFLAGS="-L/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64 -L/data/home/eew918/hypre-2.9.0b/lib" \ > > > --with-blas-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > > --with-lapack-lib="mkl_intel_lp64 mkl_sequential mkl_core" \ > > > > You can use --with-blas-lapack-lib=/opt/intel/composer_xe_2013_sp1.2.144/mkl [instead of the above 3 options > > > > > --with-fortran-interfaces=1 \ > > > --prefix=$HOME/petsc-3.5.0 \ > > > --with-shared-libraries=1 \ > > > --with-mpi=1 \ > > > --with-mpi-include=/opt/openmpi/1.6.5/intel/13.1/include \ > > > --with-mpi-lib=/opt/openmpi/1.6.5/intel/13.1/lib/libmpi.so > > > > You are better off using --with-mpi-dir=/opt/openmpi/1.6.5/intel/13.1 [instead of the above 3 options > > > > Satish > > > > > > > > But I keep getting the following error message during configure: > > > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > > ------------------------------------------------------------------------------- > > > --with-hypre-lib=['/data/home/eew918/hypre-2.9.0b/lib'] and > > > --with-hypre-include=['/data/home/eew918/hypre-2.9.0b/include'] did not work > > > > > > I see the compilation command with the correct library path (-L/data/home/eew918/hypre-2.9.0b/lib) but no library name with the -l switch. > > > > > > Regards, > > > > > > ------------------------------------------- > > > Wadud Miah > > > Research Computing Services (HPC) > > > 020 7882 8393 > > > > > > > > > > > > From tlk0812 at hotmail.com Fri Jul 11 12:05:57 2014 From: tlk0812 at hotmail.com (LikunTan) Date: Sat, 12 Jul 2014 01:05:57 +0800 Subject: [petsc-users] fftw in parallel Message-ID: Hi all, I would like to use fft to calculate discrete convolution. I am referring to /src/mat/examples/tests/ex121 as a model example. But I am not sure how to implement it in parallel, it seems to me not possible. Would you please give me any guide on this? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 11 12:12:20 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 11 Jul 2014 12:12:20 -0500 Subject: [petsc-users] Computing time derivatives in the residual for TS In-Reply-To: References: Message-ID: <87wqbjakgb.fsf@jedbrown.org> Mani, I apologize for my slow reply. Mani Chandra writes: > Hi, > > Suppose the following is the residual function that TS needs: > > void residualFunction(TS ts, PetscScalar t, Vec X, Vec dX_dt, Vec F, void > *ptr) > > and this returns the following finite volume residual at each grid point > > dU_dt + gradF + sourceTerms = 0 > > where dU_dt are the time derivatives of the conserved variables. > > The value of dU_dt needs to be computed from the values of the time > derivatives of the primitive variables given in dX_dt, i.e. dU_dt is an > analytic function of dX_dt. I recommend using this analytic relationship, i.e., dU_dt = dU/dX dX_dt This can be done by algorithmic differentiation or differencing, if you prefer. > But is it possible to compute dU_dt numerically using the vector X and > it's value at a previous time step like the following? > > dU_dt = (U(X) - U(X_old) )/dt -- (1) > > (instead of analytically computing dU_dt which is a function of the vectors > X and dX_dt which is hard to do if the function is very complicated) > > So, is computing dU_dt using (1) permissible using TS? The jacobian will be > correctly assembled for implicit methods? No, your equation is a very inaccurate time derivative and will spoil the accuracy and stability properties of the high-order time integrators. You could add U as an algebraic constraint: U - eq_state(X) = 0 if you want. This equation is trivial to solve, but it requires that you use methods capable of solving DAEs. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Jul 11 14:10:21 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Jul 2014 14:10:21 -0500 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <53BC1375.9070701@physik.uni-muenchen.de> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> <53BC1375.9070701@physik.uni-muenchen.de> Message-ID: <7B801110-E807-45D5-A1B5-11E96293F102@mcs.anl.gov> Fabian, I have fixed this in the branch barry/fix-gamg-preallocation-block and next. It worked for the matrix you provided after my fix. Please let me know if you have any additional problems Barry On Jul 8, 2014, at 10:51 AM, Fabian.Jakub wrote: > Thank you very much for the swift reply! > > I am running > git describe: v3.5-25-g0ace994 > and configured with > > ./configure \ > --with-cc=$(which mpicc) \ > --with-fc=$(which mpif90) \ > --with-cxx=$(which mpic++) \ > --with-fortran \ > --with-shared-libraries \ > --with-hdf5 \ > --with-hdf5-dir="$HDF5_DIR" \ > --with-cmake=$(which cmake) \ > \ > > I run the model with: > mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info > > Should I append the output when creating the DMDA? > For the KSP setup it says: > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 > Setup KSP > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 > [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Argument out of range > [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 63. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > ------------------------------------------------------------------------- > > Am 08.07.2014 17:14, schrieb Barry Smith: >> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. >> >> Barry >> >> >> >> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: >> >>> Hi, >>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. >>> >>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. >>> >>> The code to setup the matrix is like this: >>> >>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) >>> call DMCreateMatrix(C%da, A, ierr) >>> call MatSetFromOptions(A,ierr) >>> >>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) >>> >>> insert matrix values & assemble >>> >>> then solve. >>> >>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. >>> However if I use gamg I get an allocation error when it tries to create a coarse grid. >>> >>> If I use the DMDA preallocation, it works but uses way more memory... >>> >>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? >>> Is that even the problem or am I missing something? >>> >>> Thank you so very much. >>> >>> Sincerely, >>> Fabian >>> >>> >>> >>> >>> >>> >>> > From invite+AcGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y at facebookmail.com Sat Jul 12 03:13:39 2014 From: invite+AcGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y at facebookmail.com (Facebook) Date: Sat, 12 Jul 2014 01:13:39 -0700 Subject: [petsc-users] Check out Srimurni's photos on Facebook. Message-ID: Check out Srimurni's photos on Facebook. If you sign up for Facebook, you'll be able to stay connected with friends by seeing their photos and videos, staying up to date with their latest status updates, exchanging messages and more. Join Srimurni on Facebook https://www.facebook.com/p.php?i=100006938825435&k=AQDpToFuozvDto88ZOjf_5QaUAl124DZqTPgJ1Ey5SUF8Qm0rGMnJwofPWZTCHXK3vHv_-6_r20evwW8UsmXOPOvU20&r&app_id=350685531728 Already have an account? Add this email address to your account: https://www.facebook.com/merge_accounts.php?e=petsc-users%40mcs.anl.gov&c=AQADT3WWgc9fK_VT_4SzlH711aU6AdR4uCcUzAgwqG7raA&source=unknown ======================================= This message was sent to petsc-users at mcs.anl.gov. If you don't want to receive these emails from Facebook in the future or have your email address used for friend suggestions, please follow the link below to unsubscribe. https://www.facebook.com/o.php?k=AS1a_XCeXZAax872&e=petsc-users%40mcs.anl.gov&mid=HMTcwNTg0OTI5OnBldHNjLXVzZXJzQG1jcy5hbmwuZ292Ojg Facebook, Inc., Attention: Department 415, PO Box 10005, Palo Alto, CA 94303 -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Sun Jul 13 12:49:43 2014 From: epscodes at gmail.com (Xiangdong) Date: Sun, 13 Jul 2014 13:49:43 -0400 Subject: [petsc-users] MatGetValuesStencil? Message-ID: Hello everyone, I am just wondering whether there is a function doing the opposite of MatSetValuesStencil. I tried to find a MatGetValues equivalent for using structured grid indexing, but it seems that MatGetValuesStencil does not exist. I have a sparse matrix A built on DMDA. If I want to use A's entries to build another DMDA matrices B, what is the best way to retrieve A's entries? Is it possible to use grid indexing? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From wumeng07maths at qq.com Sun Jul 13 13:22:47 2014 From: wumeng07maths at qq.com (=?ISO-8859-1?B?T28gICAgICA=?=) Date: Mon, 14 Jul 2014 02:22:47 +0800 Subject: [petsc-users] Save Mat as TxT file Message-ID: Dear All, May I ask how to save a Mat object as a txt file? Best, :) Meng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Jul 13 13:31:31 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 13 Jul 2014 12:31:31 -0600 Subject: [petsc-users] Save Mat as TxT file In-Reply-To: References: Message-ID: <8738e5hzzw.fsf@jedbrown.org> Oo writes: > Dear All, > > > May I ask how to save a Mat object as a txt file? You can MatView to an ASCII viewer, but please use the binary viewer for anything in which you would like performance or accuracy. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Jul 13 14:45:08 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 13 Jul 2014 14:45:08 -0500 Subject: [petsc-users] MatGetValuesStencil? In-Reply-To: References: Message-ID: We don?t currently have such a routine. You could copy MatSetValuesStencil() and MatSetValuesLocal() and produce MatGetValuesStencil() and MatGetValuesLocal() with very minor changes and get what you want. You could consider contributing these back to PETSc with a pull request see https://bitbucket.org/petsc/petsc/wiki/Home#markdown-header-contributing-a-larger-piece-of-functionality Barry On Jul 13, 2014, at 12:49 PM, Xiangdong wrote: > Hello everyone, > > I am just wondering whether there is a function doing the opposite of MatSetValuesStencil. I tried to find a MatGetValues equivalent for using structured grid indexing, but it seems that MatGetValuesStencil does not exist. > > I have a sparse matrix A built on DMDA. If I want to use A's entries to build another DMDA matrices B, what is the best way to retrieve A's entries? Is it possible to use grid indexing? > > Thank you. > > Best, > Xiangdong From Fabian.Jakub at physik.uni-muenchen.de Mon Jul 14 02:45:50 2014 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian Jakub) Date: Mon, 14 Jul 2014 09:45:50 +0200 Subject: [petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg In-Reply-To: <7B801110-E807-45D5-A1B5-11E96293F102@mcs.anl.gov> References: <53BBE69C.4000609@physik.uni-muenchen.de> <637AF1E7-E893-41E5-8340-3D0B83A64D70@mcs.anl.gov> <53BC1375.9070701@physik.uni-muenchen.de> <7B801110-E807-45D5-A1B5-11E96293F102@mcs.anl.gov> Message-ID: <1405323950.28204.0.camel@howser> This works. Thanks again for your help, Fabian Am Freitag, den 11.07.2014, 14:10 -0500 schrieb Barry Smith: > Fabian, > > I have fixed this in the branch barry/fix-gamg-preallocation-block and next. It worked for the matrix you provided after my fix. > > Please let me know if you have any additional problems > > Barry > > > On Jul 8, 2014, at 10:51 AM, Fabian.Jakub wrote: > > > Thank you very much for the swift reply! > > > > I am running > > git describe: v3.5-25-g0ace994 > > and configured with > > > > ./configure \ > > --with-cc=$(which mpicc) \ > > --with-fc=$(which mpif90) \ > > --with-cxx=$(which mpic++) \ > > --with-fortran \ > > --with-shared-libraries \ > > --with-hdf5 \ > > --with-hdf5-dir="$HDF5_DIR" \ > > --with-cmake=$(which cmake) \ > > \ > > > > I run the model with: > > mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info > > > > Should I append the output when creating the DMDA? > > For the KSP setup it says: > > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > [0] VecScatterCreate(): General case: MPI to Seq > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0 unneeded,420864 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 15967792 > > Setup KSP > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 42057616 > > [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 16179040 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 41749680 > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [1]PETSC ERROR: Argument out of range > > [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > > [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > > [1]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > > [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > > [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > > [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > > [1]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > > [1]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 > > [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 17:47:31 2014 > > [0]PETSC ERROR: Configure options --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ --with-fortran --with-shared-libraries --with-hdf5 --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ --with-cmake=/usr/bin/cmake > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #2 MatSetValues() line 1136 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > > [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > > [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > > [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > > [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 305 in /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 63. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > ------------------------------------------------------------------------- > > > > Am 08.07.2014 17:14, schrieb Barry Smith: > >> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you?ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error. > >> > >> Barry > >> > >> > >> > >> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub wrote: > >> > >>> Hi, > >>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **. > >>> > >>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil. > >>> > >>> The code to setup the matrix is like this: > >>> > >>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) > >>> call DMCreateMatrix(C%da, A, ierr) > >>> call MatSetFromOptions(A,ierr) > >>> > >>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr) > >>> > >>> insert matrix values & assemble > >>> > >>> then solve. > >>> > >>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good. > >>> However if I use gamg I get an allocation error when it tries to create a coarse grid. > >>> > >>> If I use the DMDA preallocation, it works but uses way more memory... > >>> > >>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid? > >>> Is that even the problem or am I missing something? > >>> > >>> Thank you so very much. > >>> > >>> Sincerely, > >>> Fabian > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > > From milo at zeusdvds.com Mon Jul 14 14:29:44 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Mon, 14 Jul 2014 12:29:44 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 14, 2014 Message-ID: <53c42fa8e52ee_32b2dbbe9c47067973@prd-nresque01.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQ1OTA_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJMMFc5QlFBUGtGNU5OQl9HaE1lc1VZZ3Z4aTFuN2l4NDglM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpRMU9UQV9ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGNUxJeXNGVENMczVYWk54WmpXaTRnUSUzRCUzRA==/Apm3NGsPfm-cZHPJq87v_w==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs ALL ORDERS SHIP FREE AND ALL ORDERS SHIP WITHIN 24 HOURS New Releases My Darling Clementing (1946) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbXktZGFybGluZy1jbGVtZW50aW5lLTE5NDYtZHZkLw==/Yf5Jhbk_fu6VEjoMtGWwog== )March Or Die (1977) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbWFyY2gtb3ItZGllLTE5NzctZHZkLw==/ANeRJtxSw_fe1MSKaI2VKg== )Home Is Where The Hart Is (1987) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vaG9tZS1pcy13aGVyZS10aGUtaGFydC1pcy0xOTg3LWR2ZC8=/gvJq-ccUZII4Zz-oQX6kOw== )Josh And S.A.M. (1993) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vam9zaC1hbmQtcy1hLW0tMTk5My1kdmQv/MEZ6BhG9YbRVt9Qz0FqIrw== )Lost Lagoon (1958) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbG9zdC1sYWdvb24tMTk1OC1kdmQv/rjG1NZFe0I8X5YgVzAJXNg== )Los aventuras de Joselito y pulgarcito (1960) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGFzLWF2ZW50dXJhcy1kZS1qb3NlbGl0by15LXB1bGdhcmNpdG8tMTk2MC1kdmQv/NMsw06zwakr0YiyoyrBvcg== )La Mugrosita (1982) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGEtbXVncm9zaXRhLTE5ODIv/TfOaYYxS8KCmvJXYUp4VEQ== )Thanks A Million (1935) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhhbmtzLWEtbWlsbGlvbi0xOTM1LWR2ZC8=/q1JXEfhPl5CoIF-P8nxnMg== )Tombola (1962) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdG9tYm9sYS0xOTYyLWR2ZC8=/84EP6lKq1e875h1UvFv0Gg== )Louisiana Hayride (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbG91aXNpYW5hLWhheXJpZGUtMTk0NC1kdmQv/euWGLw1FUz1h9UpJ2H1pPQ== )Love Thy Neightbor (1984) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbG92ZS10aHktbmVpZ2hib3ItMTk4NC1kdmQv/p3s6UC_doFUrv9PTQ38YBA== ) ALL ORDERS SHIP FREE AND ALL ORDERS SHIP WITHIN 24 HOURS Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/ZB_KRyAncoVzfL7unfA6JA== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/K-hCVy1UGsBKVzb-awExyQ== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/T_6vfSg5mPFKGT7DgHBcmg== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/6Oh00JijDx89vlpNgdSapQ== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/a8E1ZFrtUKQMyajx1CZYTQ== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/k0graH5UQv2dQs__odOEPw== )S ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/k0graH5UQv2dQs__odOEPw== )outh of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/dsys3nWINo4kr7FW8xOjrw== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/RFi_C-W_izR4UrWxc5atGg== )? Quick LinksComedy ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vY29tZWR5Lw==/4Dd6u79bGHkRu3XEPIgKTg== )Drama ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZHJhbWEv/kgC5yB42BQGliMcdcZOGvQ== )Western ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vd2VzdGVybi8=/pDZK4w7qep2T7hNzKI90OQ== )Musical ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbXVzaWNhbC8=/xtkETEr0_6Oy_uG889hKPg== )Adventure ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYWR2ZW50dXJlLw==/CkV330ywVSwUTAZuzZHmWw== )Documentary ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZG9jdW1lbnRhcnkv/aXvKXhw37a9ypi2NOLh9yg== )Animation ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYW5pbWF0aW9uLw==/fQXaHQnFJCDMfEKi3HS1Ow== )TV Shows ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdHYtc2hvd3Mv/uhTKlxnuWQaNZI8WO_HTKA== ) All orders ship within 24 hours and all shipping is FREE. Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/nWzN1_74jL4z7htAAFkfOA==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774590&sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQ1OTA%3D%2FliB5GXGid-aC2BTqoQJlMA%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBL0W9BQAPkF5NNB_GhMesUYgvxi1n7ix48=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0NTkw/LuGmkUqv6tpgsoICEfIjug==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774590&utm_content=CamID6597069774590 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsimon at us.ibm.com Mon Jul 14 19:29:35 2014 From: rsimon at us.ibm.com (Robert Simon) Date: Mon, 14 Jul 2014 20:29:35 -0400 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase: INFO(1)=-100, INFO(2)=-32766 Message-ID: Hello, Does anyone know what this failure means and how to solve it? It only seems to occur when I run PETSC with IBM POE's MPI. If I run with the INTEL mpirun, I do not see the failure. Best regards, Bob Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From hong at aspiritech.org Mon Jul 14 20:24:43 2014 From: hong at aspiritech.org (hong at aspiritech.org) Date: Mon, 14 Jul 2014 20:24:43 -0500 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase: INFO(1)=-100, INFO(2)=-32766 In-Reply-To: References: Message-ID: See mumps user guide, Chapter 7 Error diagnostics (page 38): "The possible error codes returned in INFO(1) (and INFOG(1)) have the following meaning: ?1 An error occurred on processor INFO(2)." So this does not tell anything useful. Suggest building petsc in debugging mode, then you may get more error info. Hong On Mon, Jul 14, 2014 at 7:29 PM, Robert Simon wrote: > Hello, > > Does anyone know what this failure means and how to solve it? > > It only seems to occur when I run PETSC with IBM POE's MPI. If I run with > the INTEL mpirun, I do not see the > failure. > > Best regards, > > Bob Simon > > From jyang29 at uh.edu Tue Jul 15 14:19:01 2014 From: jyang29 at uh.edu (Brian Yang) Date: Tue, 15 Jul 2014 14:19:01 -0500 Subject: [petsc-users] Is there a way to do the Tikhonov regularization using PETSc Message-ID: Hi all, I am dealing with an A matrix that might be ill-conditioned, wondering whether there is a way to do the regularization? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 15 14:23:34 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 15 Jul 2014 13:23:34 -0600 Subject: [petsc-users] Is there a way to do the Tikhonov regularization using PETSc In-Reply-To: References: Message-ID: <878unufmtl.fsf@jedbrown.org> Brian Yang writes: > Hi all, > > I am dealing with an A matrix that might be ill-conditioned, wondering > whether there is a way to do the regularization? Yes. There is not enough information in your question to answer with more details. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From brianyang1106 at gmail.com Tue Jul 15 14:28:02 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Tue, 15 Jul 2014 14:28:02 -0500 Subject: [petsc-users] Is there a way to do the Tikhonov regularization using PETSc In-Reply-To: <878unufmtl.fsf@jedbrown.org> References: <878unufmtl.fsf@jedbrown.org> Message-ID: Thanks for the quick reply. In the main diagonal of Matrix A, it may have some very small values, I want to add factor to stabilize it, like: A + alpha * I On Tue, Jul 15, 2014 at 2:23 PM, Jed Brown wrote: > Brian Yang writes: > > > Hi all, > > > > I am dealing with an A matrix that might be ill-conditioned, wondering > > whether there is a way to do the regularization? > > Yes. > > There is not enough information in your question to answer with more > details. > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 15 14:31:09 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 15 Jul 2014 13:31:09 -0600 Subject: [petsc-users] Is there a way to do the Tikhonov regularization using PETSc In-Reply-To: References: <878unufmtl.fsf@jedbrown.org> Message-ID: <8738e2fmgy.fsf@jedbrown.org> Brian Yang writes: > Thanks for the quick reply. > > In the main diagonal of Matrix A, it may have some very small values, I > want to add factor to stabilize it, like: > > A + alpha * I http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatShift.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From brianyang1106 at gmail.com Tue Jul 15 14:35:17 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Tue, 15 Jul 2014 14:35:17 -0500 Subject: [petsc-users] Is there a way to do the Tikhonov regularization using PETSc In-Reply-To: <8738e2fmgy.fsf@jedbrown.org> References: <878unufmtl.fsf@jedbrown.org> <8738e2fmgy.fsf@jedbrown.org> Message-ID: OK, thanks. On Tue, Jul 15, 2014 at 2:31 PM, Jed Brown wrote: > Brian Yang writes: > > > Thanks for the quick reply. > > > > In the main diagonal of Matrix A, it may have some very small values, I > > want to add factor to stabilize it, like: > > > > A + alpha * I > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatShift.html > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.null at gmail.com Tue Jul 15 14:46:11 2014 From: sean.null at gmail.com (Xin Zhao) Date: Tue, 15 Jul 2014 15:46:11 -0400 Subject: [petsc-users] Too few arguments to PetscInitialize Message-ID: Dear all, I'm using the old version of PETSc 2.3.3-p16. When I compiled some code with the old version, I've got a batch of errors like: error #165: too few arguments in function call PetscInitialize(&argc, &argv); error #165: too few arguments in function call m_PetscMatrix = MatCreateSeqAIJ(m, n, nNonzero); My thought is that the PetscPolymorphicSubroutine may not work. Since the code is no longer under maintenance (the graduate who wrote the code left), I wonder how much the API has been changed from 2.3.3 to the latest 3.5.0? My configuration for PETSc is ./config/configure.py --with-shared=1 --with-cc=icc --with-cxx=icpc --with-fc=ifort --with-blas-lapack-dir=$MKLROOT --with-mpi=0 --with-debugging=0 Thanks. Best, Xin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 15 14:56:28 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 15 Jul 2014 13:56:28 -0600 Subject: [petsc-users] Too few arguments to PetscInitialize In-Reply-To: References: Message-ID: <87y4vue6qb.fsf@jedbrown.org> Xin Zhao writes: > Dear all, > > I'm using the old version of PETSc 2.3.3-p16. When I compiled some code > with the old version, I've got a batch of errors like: > > error #165: too few arguments in function call > > PetscInitialize(&argc, &argv); > > error #165: too few arguments in function call > > m_PetscMatrix = MatCreateSeqAIJ(m, n, nNonzero); > > My thought is that the PetscPolymorphicSubroutine may not work. Or you are compiling with a C version of PETSc (not --with-clanguage=cxx)? PetscPolymorphicSubroutine was removed in petsc-3.3. > Since the code is no longer under maintenance (the graduate who wrote the > code left), I wonder how much the API has been changed from 2.3.3 to the > latest 3.5.0? Most things will work, but any complex code that has sat unmaintained for 7 years will need some updating. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hus003 at ucsd.edu Tue Jul 15 20:47:45 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 01:47:45 +0000 Subject: [petsc-users] error on running snes example 55 Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBF5D@XMAIL-MBX-BH1.AD.UCSD.EDU> In snes examples/tutorials ex55, there is a comment saying that one can use the following options for running this example: ./ex55 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -fieldsplit_1_ksp_type fgmres -fieldsplit_1_pc_type lsc -snes_vi_monitor -ksp_monitor_true_residual -fieldsplit_ksp_monitor -fieldsplit_0_pc_type hypre -da_grid_x 65 -da_grid_y 65 -snes_atol 1.e-11 -ksp_rtol 1.e-8 However it gives me the following error message: 0 SNES VI Function norm 3.976439267245e-05 Active lower constraints 2001/2241 upper constraints 314/397 Percent of total 0.136982 Percent of bounded 0.182643 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PCFieldSplitSetDefaults line 310 src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: [0] PCSetUp_FieldSplit line 457 src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: [0] PCSetUp line 868 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 192 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 347 src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: [0] SNESSolve line 3589 src/snes/interface/snes.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex55 on a linux-gnu named Huis-MacBook-Pro.local by Paul Tue Jul 15 18:42:59 2014 [0]PETSC ERROR: Libraries linked from /usr/local/petsc-3.4.4/linux-gnu/lib [0]PETSC ERROR: Configure run at Fri Jun 13 23:47:57 2014 [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-fc=gfortran --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Do you might know what's going on? Another strange issue is that I typed the following command: ./ex55 -help | grep -A5 fieldsplit The output is: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) ILU Options -pc_factor_in_place: Form factored matrix in the same memory as the matrix (PCFactorSetUseInPlace) -pc_factor_fill <1>: Expected non-zeros in factored matrix (PCFactorSetFill) -pc_factor_shift_type (choose one of) NONE NONZERO POSITIVE_DEFINITE INBLOCKS (PCFactorSetShiftType) While I'm actually expecting to have something such as -pc_fieldsplit_detect_saddle_point coming out. Then isn't it one of the possible options? -------------- next part -------------- An HTML attachment was scrubbed... URL: From milo at zeusdvds.com Wed Jul 16 15:20:19 2014 From: milo at zeusdvds.com (milo at zeusdvds.com) Date: Wed, 16 Jul 2014 13:20:19 -0700 Subject: [petsc-users] New Offerings From Zeus DVDs, July 16, 2014 Message-ID: <53c6de834be63_32af97deac43995171@prd-nresque01.sf.verticalresponse.com.mail> View Online ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vZW1haWxzLzY1OTcwNjk3NzQ5NTY_Y29udGFjdF9pZD02NTk3MDczNzQwNDk3JmZyb21fZW1haWw9ZmFsc2Umc2s9YXFKUUIySmdHakJLYU05RlFBUGtGNUN4S3JYb28wdnBUcVBrUUVKZ1JCWnMlM0QlMkZhSFIwY0hNNkx5OTJjakl1ZG1WeWRHbGpZV3h5WlhOd2IyNXpaUzVqYjIwdlpXMWhhV3h6THpZMU9UY3dOamszTnpRNU5UWV9ZMjl1ZEdGamRGOXBaRDAyTlRrM01EY3pOelF3TkRrM0ptWnliMjFmWlcxaGFXdzlabUZzYzJVJTNEJTJGWFY0T1lOZUZqMlpwcnhXMWowMGNzUSUzRCUzRA==/1ANnmAPtUfmrkUiT83GYoQ==&merge_field_type=%7BVR_HOSTED_LINK%7D ) The dice of Zeus always fall luckily.? New Offerings FromZeus DVDs ALL ORDERS SHIP FREE AND ALL ORDERS SHIP WITHIN 24 HOURS New Releases Women Of All Nations (1931) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vd29tZW4tb2YtYWxsLW5hdGlvbnMtMTkzMS1kdmQv/nZ5v6XAlOFGho8gAALc0Jw== )The Video Dead (1987) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXZpZGVvLWRlYWQtMTk4Ny1kdmQv/SzUTWqkKLUD-Hi6cPgP-yw== )The Vindicator (1986) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXZpbmRpY2F0b3ItMTk4Ni1kdmQv/67316_WZsNNmYT0QttbqQg== )Overland Pacific (1954) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vb3ZlcmxhbmQtcGFjaWZpYy0xOTU0LWR2ZC8=/FKh4C0-l4WUVSYSVkRPHXQ== )Accomplice (1946) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYWNjb21wbGljZS0xOTQ2LWR2ZC8=/9DyCs2wohH3PdeR8B1mczA== )1900 (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vMTkwMC0xOTc2LWR2ZC8=/hHA0xsGa17MoC8xSt7Dwtw== )...und das is die Hauptsache (1931) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdW5kLWRhcy1pc3QtZGllLWhhdXB0c2FjaGUtMTkzMS1kdmQv/YbWkCsviEY5E3HTjpy0eHA== )Rich In Love (1992) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vcmljaC1pbi1sb3ZlLTE5OTItZHZkLw==/sOTdqqy3h0kNiBR0JWCdkQ== )Wife, Doctor and Nurse (1937) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vd2lmZS1kb2N0b3ItYW5kLW51cnNlLTE5MzctZHZkLw==/twQKHJL2yzWG86pu-RRu1A== )The Black Chapel (1959) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWJsYWNrLWNoYXBlbC0xOTU5LWR2ZC8=/B4d3CxMq3pDwit8px08ukQ== )The Woman in the Window (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLXdvbWFuLWluLXRoZS13aW5kb3ctMTk0NC1kdmQv/b_V2VmJxZAv4upH1P5d4ig== ) ALL ORDERS SHIP FREE AND ALL ORDERS SHIP WITHIN 24 HOURS Best SellersBeau Geste (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYmVhdS1nZXN0ZS0xOTY2LWR2ZC8=/VkkZfK6sN7yTYuhZFo6phA== )A Man Could Get Killed (1966) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1tYW4tY291bGQtZ2V0LWtpbGxlZC0xOTY2LWR2ZC8=/qc4-WD2nfqK5O4BF68IJWQ== )Kiss of Fire (1955) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20va2lzcy1vZi1maXJlLTE5NTUtZHZkLw==/nXkk33gpApE5ACggtkDDpA== )A Perilous Journey (1953) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYS1wZXJpbG91cy1qb3VybmV5LTE5NTMtZHZkLw==/PpPs8IZK2FZdr9c8vm_spw== )Law And Order (1976) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbGF3LWFuZC1vcmRlci0xOTc2LWR2ZC8=/RD5w7zNBe7Y3PDTG1No1ng== )The Honkers (1972) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/9Alugl9z7eQLkLjzhggYyQ== )S ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdGhlLWhvbmtlcnMtMTk3Mi1kdmQv/9Alugl9z7eQLkLjzhggYyQ== )outh of Dixie (1944) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/gYyQn8qD2BV0qMeaEgDJSQ== )Y ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc291dGgtb2YtZGl4aWUtMTk0NC1kdmQv/gYyQn8qD2BV0qMeaEgDJSQ== )ou Never Can Tell (1951) ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20veW91LW5ldmVyLWNhbi10ZWxsLTE5NTEtZHZkLw==/EFu_YL_2qsY55JfA0vYpGQ== ) Search For A Movie ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vc2VhcmNoLnBocD9tb2RlPWFkdmFuY2Vk/6OupZc-d8x7G0wbPm1NA1A== )? Quick LinksComedy ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vY29tZWR5Lw==/o3ft7-a-O6KR6nlR9hiDlQ== )Drama ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZHJhbWEv/7UOtJWD3dry6nXQhIPaafw== )Western ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vd2VzdGVybi8=/lekajtKNGK1dHyPiYvCojA== )Musical ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vbXVzaWNhbC8=/Ee-hBJCtlRbqIZNNK5R6CQ== )Adventure ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYWR2ZW50dXJlLw==/4vV6z4rmbQf9BN0X16Iclw== )Documentary ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vZG9jdW1lbnRhcnkv/jWOPAJY1vVP8J1xOBL7O8g== )Animation ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vYW5pbWF0aW9uLw==/uE_ZzeZPJBevfCvChgK1gA== )TV Shows ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy56ZXVzZHZkcy5jb20vdHYtc2hvd3Mv/A8MW6AOYBMR-USge6Y3f9Q== ) All orders ship within 24 hours and all shipping is FREE. Follow Us ( https://cts.vrmailer1.com/click?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cDovL3d3dy50d2l0dGVyLmNvbS96ZXVzZHZkcw==/pK7RKJk8MQ74iPVeLvnFJw==&merge_field_type=%7BVR_TWITTER_URL%7D ) Zeus DVDs 7027 W. Broward Blvd. #309 Plantation, FL 33317 Update your Email Preferences ( https://vr2.verticalresponse.com/contacts/6597073740497/edit?email_id=6597069774956&sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs%3D%2FaHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny9lZGl0P2VtYWlsX2lkPTY1OTcwNjk3NzQ5NTY%3D%2FWAHpyjCqZTlmczmL_9WiWw%3D%3D ) or Unsubscribe ( https://cts.vrmailer1.com/unsub?sk=aqJQB2JgGjBKaM9FQAPkF5CxKrXoo0vpTqPkQEJgRBZs=/aHR0cHM6Ly92cjIudmVydGljYWxyZXNwb25zZS5jb20vY29udGFjdHMvNjU5NzA3Mzc0MDQ5Ny91bnN1YnNjcmliZT9lbWFpbF9pZD02NTk3MDY5Nzc0OTU2/YLi_5ou8_9Pcdp1VYzxOYg==&merge_field_type=%7BUNSUBSCRIBE_LINK%7D ) ( http://www.verticalresponse.com/?utm_campaign=email-footer&utm_medium=referral&utm_source=footer&sn=CamID6597069774956&utm_content=CamID6597069774956 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 15:44:18 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 20:44:18 +0000 Subject: [petsc-users] fftw in parallel Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBF95@XMAIL-MBX-BH1.AD.UCSD.EDU> Hello Likun, Regarding your question sent 5 days ago. I think you can have a overview of all the examples under the directory src/mat/examples/tests/ following the link: http://www.mcs.anl.gov/petsc/petsc-current/src/mat/examples/tests/ Among these examples, there are 14 about fftw, and there are some with mpi FFTW, for instance, ex143, 144, 146, 158, etc. If you take a look at these examples, you will find that they are using another package FFTW3 as well as the interface between FFTW3 and PETSc. In fact, if you download and install FFTW3 with the right configuration, you should be able to do FFT in parallel without PETSc. If you want to do FFT in parallel in PETSc, you may have to figure out more from those examples provided. Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 16:09:25 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 21:09:25 +0000 Subject: [petsc-users] why a certain option cannot be used Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 16 16:16:34 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 16 Jul 2014 16:16:34 -0500 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui wrote: > I want to solve Stokes equation using user defined schur complement > preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, > and ex70.c. In these examples, there are comments about the usage, for > example: > > mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. > > However the option -fieldsplit_1_user_pc is not recognized by the > executable. The output of the above command is: > This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt > residual u = 2.6315e-05 > > residual p = 0.000229872 > > residual [u,p] = 0.000231373 > > discretization error u = 0.00633503 > > discretization error p = 0.121534 > > discretization error [u,p] = 0.121699 > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-fieldsplit_1_user_pc (no value) > > > I'm wondering what's going on. Is that because I need some specific > options during my configuring of the package? By the way, the version of > PETSc I'm using is 3.4.4. > > > Best, > > Hui > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 16:22:51 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 21:22:51 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 16 16:25:23 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 16 Jul 2014 16:25:23 -0500 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui wrote: > Thank you Matt for answering, however even with that I don't think it > works. For example, the command > > ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi > You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi Matt > gives me the following output > > > residual u = 2.6315e-05 > > residual p = 0.000229872 > > residual [u,p] = 0.000231373 > > discretization error u = 0.00633503 > > discretization error p = 0.121534 > > discretization error [u,p] = 0.121699 > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-fieldsplit_1_type value: jacobi > > > Furthermore, if I look into the possible options with keyword fieldsplit > involved by the command: > > > ./ex70 -help | grep -A5 fieldsplit > > > I get this output: > > > eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat > fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs > (PCSetType) > > -pc_use_amat: use Amat (instead of Pmat) to define > preconditioner in nested inner solves (PCSetUseAmat) > > Krylov Method (KSP) options > ------------------------------------------------- > > -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash > stcg gltr richardson > > chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr > pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres > specest dgmres (KSPSetType) > > -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) > > > So I don't see any option that's similar to -fieldsplit_1_type. > > > Hui > > > > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com] > *Sent:* Wednesday, July 16, 2014 2:16 PM > *To:* Sun, Hui > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] why a certain option cannot be used > > On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui wrote: > >> I want to solve Stokes equation using user defined schur complement >> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >> and ex70.c. In these examples, there are comments about the usage, for >> example: >> >> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >> >> However the option -fieldsplit_1_user_pc is not recognized by the >> executable. The output of the above command is: >> > > This ("user_pc") is just a mnemonic. What it means is that you can use > any PC in this slot. What he should have written is > > -fieldsplit_1_pc_type > > Thanks, > > Matt > > >> residual u = 2.6315e-05 >> >> residual p = 0.000229872 >> >> residual [u,p] = 0.000231373 >> >> discretization error u = 0.00633503 >> >> discretization error p = 0.121534 >> >> discretization error [u,p] = 0.121699 >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-fieldsplit_1_user_pc (no value) >> >> >> I'm wondering what's going on. Is that because I need some specific >> options during my configuring of the package? By the way, the version of >> PETSc I'm using is 3.4.4. >> >> >> Best, >> >> Hui >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 16 16:32:51 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 16 Jul 2014 16:32:51 -0500 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley wrote: > On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui wrote: > >> Thank you Matt for answering, however even with that I don't think it >> works. For example, the command >> >> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >> > > You have mistyped. Look carefully at my last message: > > -fieldsplit_1_pc_type jacobi > I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt > Matt > > >> gives me the following output >> >> >> residual u = 2.6315e-05 >> >> residual p = 0.000229872 >> >> residual [u,p] = 0.000231373 >> >> discretization error u = 0.00633503 >> >> discretization error p = 0.121534 >> >> discretization error [u,p] = 0.121699 >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-fieldsplit_1_type value: jacobi >> >> >> Furthermore, if I look into the possible options with keyword >> fieldsplit involved by the command: >> >> >> ./ex70 -help | grep -A5 fieldsplit >> >> >> I get this output: >> >> >> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >> (PCSetType) >> >> -pc_use_amat: use Amat (instead of Pmat) to define >> preconditioner in nested inner solves (PCSetUseAmat) >> >> Krylov Method (KSP) options >> ------------------------------------------------- >> >> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash >> stcg gltr richardson >> >> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr >> pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >> specest dgmres (KSPSetType) >> >> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >> >> >> So I don't see any option that's similar to -fieldsplit_1_type. >> >> >> Hui >> >> >> >> ------------------------------ >> *From:* Matthew Knepley [knepley at gmail.com] >> *Sent:* Wednesday, July 16, 2014 2:16 PM >> *To:* Sun, Hui >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] why a certain option cannot be used >> >> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui wrote: >> >>> I want to solve Stokes equation using user defined schur complement >>> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >>> and ex70.c. In these examples, there are comments about the usage, for >>> example: >>> >>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>> >>> However the option -fieldsplit_1_user_pc is not recognized by the >>> executable. The output of the above command is: >>> >> >> This ("user_pc") is just a mnemonic. What it means is that you can use >> any PC in this slot. What he should have written is >> >> -fieldsplit_1_pc_type >> >> Thanks, >> >> Matt >> >> >>> residual u = 2.6315e-05 >>> >>> residual p = 0.000229872 >>> >>> residual [u,p] = 0.000231373 >>> >>> discretization error u = 0.00633503 >>> >>> discretization error p = 0.121534 >>> >>> discretization error [u,p] = 0.121699 >>> >>> WARNING! There are options you set that were not used! >>> >>> WARNING! could be spelling mistake, etc! >>> >>> Option left: name:-fieldsplit_1_user_pc (no value) >>> >>> >>> I'm wondering what's going on. Is that because I need some specific >>> options during my configuring of the package? By the way, the version of >>> PETSc I'm using is 3.4.4. >>> >>> >>> Best, >>> >>> Hui >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 16:40:15 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 21:40:15 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> , Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 16 16:43:10 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 16 Jul 2014 16:43:10 -0500 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Wed, Jul 16, 2014 at 4:40 PM, Sun, Hui wrote: > Thank you Matt. I've corrected it to > > ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi > > > However the output is still: > > > residual u = 2.6315e-05 > > residual p = 0.000229872 > > residual [u,p] = 0.000231373 > > discretization error u = 0.00633503 > > discretization error p = 0.121534 > > discretization error [u,p] = 0.121699 > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-fieldsplit_1_pc_type value: jacobi > > > If I type > > > ./ex70 -help | grep fieldsplit_1_ > > > the output is that nothing coming out. > That seems to show that you are not using PCFIELDSPLIT at all. Are you? You can check using -ksp_view Matt > > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com] > *Sent:* Wednesday, July 16, 2014 2:32 PM > > *To:* Sun, Hui > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] why a certain option cannot be used > > On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: > >> On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui wrote: >> >>> Thank you Matt for answering, however even with that I don't think it >>> works. For example, the command >>> >>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >>> >> >> You have mistyped. Look carefully at my last message: >> >> -fieldsplit_1_pc_type jacobi >> > > I also want to note that you can see all the available options with > -help. It does produce > a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You > can also see the > prefix of each solver component using -ksp_view. > > Matt > > >> Matt >> >> >>> gives me the following output >>> >>> >>> residual u = 2.6315e-05 >>> >>> residual p = 0.000229872 >>> >>> residual [u,p] = 0.000231373 >>> >>> discretization error u = 0.00633503 >>> >>> discretization error p = 0.121534 >>> >>> discretization error [u,p] = 0.121699 >>> >>> WARNING! There are options you set that were not used! >>> >>> WARNING! could be spelling mistake, etc! >>> >>> Option left: name:-fieldsplit_1_type value: jacobi >>> >>> >>> Furthermore, if I look into the possible options with keyword >>> fieldsplit involved by the command: >>> >>> >>> ./ex70 -help | grep -A5 fieldsplit >>> >>> >>> I get this output: >>> >>> >>> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >>> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >>> (PCSetType) >>> >>> -pc_use_amat: use Amat (instead of Pmat) to define >>> preconditioner in nested inner solves (PCSetUseAmat) >>> >>> Krylov Method (KSP) options >>> ------------------------------------------------- >>> >>> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash >>> stcg gltr richardson >>> >>> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr >>> pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >>> specest dgmres (KSPSetType) >>> >>> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >>> >>> >>> So I don't see any option that's similar to -fieldsplit_1_type. >>> >>> >>> Hui >>> >>> >>> >>> ------------------------------ >>> *From:* Matthew Knepley [knepley at gmail.com] >>> *Sent:* Wednesday, July 16, 2014 2:16 PM >>> *To:* Sun, Hui >>> *Cc:* petsc-users at mcs.anl.gov >>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>> >>> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui wrote: >>> >>>> I want to solve Stokes equation using user defined schur complement >>>> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >>>> and ex70.c. In these examples, there are comments about the usage, for >>>> example: >>>> >>>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>>> >>>> However the option -fieldsplit_1_user_pc is not recognized by the >>>> executable. The output of the above command is: >>>> >>> >>> This ("user_pc") is just a mnemonic. What it means is that you can use >>> any PC in this slot. What he should have written is >>> >>> -fieldsplit_1_pc_type >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> residual u = 2.6315e-05 >>>> >>>> residual p = 0.000229872 >>>> >>>> residual [u,p] = 0.000231373 >>>> >>>> discretization error u = 0.00633503 >>>> >>>> discretization error p = 0.121534 >>>> >>>> discretization error [u,p] = 0.121699 >>>> >>>> WARNING! There are options you set that were not used! >>>> >>>> WARNING! could be spelling mistake, etc! >>>> >>>> Option left: name:-fieldsplit_1_user_pc (no value) >>>> >>>> >>>> I'm wondering what's going on. Is that because I need some specific >>>> options during my configuring of the package? By the way, the version of >>>> PETSc I'm using is 3.4.4. >>>> >>>> >>>> Best, >>>> >>>> Hui >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 16:47:42 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 21:47:42 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> , , <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jul 16 16:57:56 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 16 Jul 2014 23:57:56 +0200 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui wrote: > No I don't think I'm using fieldsplitting although I want to use it. The > output of > > ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view > > is the following: > > KSP Object: 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: none > > linear system matrix = precond matrix: > > Matrix Object: 1 MPI processes > > type: nest > > rows=4608, cols=4608 > > Matrix object: > > type=nest, rows=2, cols=2 > > MatNest structure: > > (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 > > (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 > > (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 > > (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 > > residual u = 2.6315e-05 > > residual p = 0.000229872 > > residual [u,p] = 0.000231373 > > discretization error u = 0.00633503 > > discretization error p = 0.121534 > > discretization error [u,p] = 0.121699 > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-fieldsplit_1_pc_type value: jacobi > > > ------------------------------ > *From:* Sun, Hui > *Sent:* Wednesday, July 16, 2014 2:40 PM > *To:* Matthew Knepley > *Cc:* petsc-users at mcs.anl.gov > > *Subject:* RE: [petsc-users] why a certain option cannot be used > > Thank you Matt. I've corrected it to > > ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi > > > However the output is still: > > > residual u = 2.6315e-05 > > residual p = 0.000229872 > > residual [u,p] = 0.000231373 > > discretization error u = 0.00633503 > > discretization error p = 0.121534 > > discretization error [u,p] = 0.121699 > > WARNING! There are options you set that were not used! > > WARNING! could be spelling mistake, etc! > > Option left: name:-fieldsplit_1_pc_type value: jacobi > > > If I type > > > ./ex70 -help | grep fieldsplit_1_ > > > the output is that nothing coming out. > > > > > > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com > ] > *Sent:* Wednesday, July 16, 2014 2:32 PM > *To:* Sun, Hui > *Cc:* petsc-users at mcs.anl.gov > > *Subject:* Re: [petsc-users] why a certain option cannot be used > > On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: > >> On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > > wrote: >> >>> Thank you Matt for answering, however even with that I don't think it >>> works. For example, the command >>> >>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >>> >> >> You have mistyped. Look carefully at my last message: >> >> -fieldsplit_1_pc_type jacobi >> > > I also want to note that you can see all the available options with > -help. It does produce > a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You > can also see the > prefix of each solver component using -ksp_view. > > Matt > > >> Matt >> >> >>> gives me the following output >>> >>> >>> residual u = 2.6315e-05 >>> >>> residual p = 0.000229872 >>> >>> residual [u,p] = 0.000231373 >>> >>> discretization error u = 0.00633503 >>> >>> discretization error p = 0.121534 >>> >>> discretization error [u,p] = 0.121699 >>> >>> WARNING! There are options you set that were not used! >>> >>> WARNING! could be spelling mistake, etc! >>> >>> Option left: name:-fieldsplit_1_type value: jacobi >>> >>> >>> Furthermore, if I look into the possible options with keyword >>> fieldsplit involved by the command: >>> >>> >>> ./ex70 -help | grep -A5 fieldsplit >>> >>> >>> I get this output: >>> >>> >>> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >>> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >>> (PCSetType) >>> >>> -pc_use_amat: use Amat (instead of Pmat) to define >>> preconditioner in nested inner solves (PCSetUseAmat) >>> >>> Krylov Method (KSP) options >>> ------------------------------------------------- >>> >>> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash >>> stcg gltr richardson >>> >>> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr >>> pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >>> specest dgmres (KSPSetType) >>> >>> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >>> >>> >>> So I don't see any option that's similar to -fieldsplit_1_type. >>> >>> >>> Hui >>> >>> >>> >>> ------------------------------ >>> *From:* Matthew Knepley [knepley at gmail.com >>> ] >>> *Sent:* Wednesday, July 16, 2014 2:16 PM >>> *To:* Sun, Hui >>> *Cc:* petsc-users at mcs.anl.gov >>> >>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>> >>> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui >> > wrote: >>> >>>> I want to solve Stokes equation using user defined schur complement >>>> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >>>> and ex70.c. In these examples, there are comments about the usage, for >>>> example: >>>> >>>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>>> >>>> However the option -fieldsplit_1_user_pc is not recognized by the >>>> executable. The output of the above command is: >>>> >>> >>> This ("user_pc") is just a mnemonic. What it means is that you can use >>> any PC in this slot. What he should have written is >>> >>> -fieldsplit_1_pc_type >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> residual u = 2.6315e-05 >>>> >>>> residual p = 0.000229872 >>>> >>>> residual [u,p] = 0.000231373 >>>> >>>> discretization error u = 0.00633503 >>>> >>>> discretization error p = 0.121534 >>>> >>>> discretization error [u,p] = 0.121699 >>>> >>>> WARNING! There are options you set that were not used! >>>> >>>> WARNING! could be spelling mistake, etc! >>>> >>>> Option left: name:-fieldsplit_1_user_pc (no value) >>>> >>>> >>>> I'm wondering what's going on. Is that because I need some specific >>>> options during my configuring of the package? By the way, the version of >>>> PETSc I'm using is 3.4.4. >>>> >>>> >>>> Best, >>>> >>>> Hui >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 17:07:44 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 22:07:44 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Dave, it does solve the problem. Another question is: since some options like -fieldsplit_0_pc_type does not show up if I type ./ex70 -help how then can I have a complete list of all the options? Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 2:57 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui > wrote: No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jul 16 17:10:30 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Jul 2014 00:10:30 +0200 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: Run with -help On 17 July 2014 00:07, Sun, Hui wrote: > Thank you Dave, it does solve the problem. Another question is: since > some options like > > -fieldsplit_0_pc_type > > does not show up if I type > > ./ex70 -help > > how then can I have a complete list of all the options? > > Best, > Hui > > > ------------------------------ > *From:* Dave May [dave.mayhem23 at gmail.com] > *Sent:* Wednesday, July 16, 2014 2:57 PM > *To:* Sun, Hui > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > > *Subject:* Re: [petsc-users] why a certain option cannot be used > > You need to specify on the command line > -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy > > Where xxx and yyy are the desired preconditioners for the (1,1) block > and (2,2) block. Note that the default PC (ilu(0)) applied to both splits > will fail for this example. > > The comments in the code are misleading and incorrect. Without the > option -pc_type fieldsplit, all fieldsplit options have no effect. > > When debugging solvers, always run with - ksp_view (as Matt recommends) > > > > On Wednesday, 16 July 2014, Sun, Hui wrote: > >> No I don't think I'm using fieldsplitting although I want to use it. >> The output of >> >> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view >> >> is the following: >> >> KSP Object: 1 MPI processes >> >> type: gmres >> >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> >> GMRES: happy breakdown tolerance 1e-30 >> >> maximum iterations=10000, initial guess is zero >> >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> >> left preconditioning >> >> using PRECONDITIONED norm type for convergence test >> >> PC Object: 1 MPI processes >> >> type: none >> >> linear system matrix = precond matrix: >> >> Matrix Object: 1 MPI processes >> >> type: nest >> >> rows=4608, cols=4608 >> >> Matrix object: >> >> type=nest, rows=2, cols=2 >> >> MatNest structure: >> >> (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 >> >> (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 >> >> (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 >> >> (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 >> >> residual u = 2.6315e-05 >> >> residual p = 0.000229872 >> >> residual [u,p] = 0.000231373 >> >> discretization error u = 0.00633503 >> >> discretization error p = 0.121534 >> >> discretization error [u,p] = 0.121699 >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> >> >> ------------------------------ >> *From:* Sun, Hui >> *Sent:* Wednesday, July 16, 2014 2:40 PM >> *To:* Matthew Knepley >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* RE: [petsc-users] why a certain option cannot be used >> >> Thank you Matt. I've corrected it to >> >> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi >> >> >> However the output is still: >> >> >> residual u = 2.6315e-05 >> >> residual p = 0.000229872 >> >> residual [u,p] = 0.000231373 >> >> discretization error u = 0.00633503 >> >> discretization error p = 0.121534 >> >> discretization error [u,p] = 0.121699 >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> >> >> If I type >> >> >> ./ex70 -help | grep fieldsplit_1_ >> >> >> the output is that nothing coming out. >> >> >> >> >> >> ------------------------------ >> *From:* Matthew Knepley [knepley at gmail.com ] >> *Sent:* Wednesday, July 16, 2014 2:32 PM >> *To:* Sun, Hui >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] why a certain option cannot be used >> >> On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > > wrote: >> >>> On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui >> > wrote: >>> >>>> Thank you Matt for answering, however even with that I don't think it >>>> works. For example, the command >>>> >>>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >>>> >>> >>> You have mistyped. Look carefully at my last message: >>> >>> -fieldsplit_1_pc_type jacobi >>> >> >> I also want to note that you can see all the available options with >> -help. It does produce >> a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). >> You can also see the >> prefix of each solver component using -ksp_view. >> >> Matt >> >> >>> Matt >>> >>> >>>> gives me the following output >>>> >>>> >>>> residual u = 2.6315e-05 >>>> >>>> residual p = 0.000229872 >>>> >>>> residual [u,p] = 0.000231373 >>>> >>>> discretization error u = 0.00633503 >>>> >>>> discretization error p = 0.121534 >>>> >>>> discretization error [u,p] = 0.121699 >>>> >>>> WARNING! There are options you set that were not used! >>>> >>>> WARNING! could be spelling mistake, etc! >>>> >>>> Option left: name:-fieldsplit_1_type value: jacobi >>>> >>>> >>>> Furthermore, if I look into the possible options with keyword >>>> fieldsplit involved by the command: >>>> >>>> >>>> ./ex70 -help | grep -A5 fieldsplit >>>> >>>> >>>> I get this output: >>>> >>>> >>>> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >>>> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >>>> (PCSetType) >>>> >>>> -pc_use_amat: use Amat (instead of Pmat) to define >>>> preconditioner in nested inner solves (PCSetUseAmat) >>>> >>>> Krylov Method (KSP) options >>>> ------------------------------------------------- >>>> >>>> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash >>>> stcg gltr richardson >>>> >>>> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr >>>> pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >>>> specest dgmres (KSPSetType) >>>> >>>> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >>>> >>>> >>>> So I don't see any option that's similar to -fieldsplit_1_type. >>>> >>>> >>>> Hui >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* Matthew Knepley [knepley at gmail.com >>>> ] >>>> *Sent:* Wednesday, July 16, 2014 2:16 PM >>>> *To:* Sun, Hui >>>> *Cc:* petsc-users at mcs.anl.gov >>>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>>> >>>> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui >>> > wrote: >>>> >>>>> I want to solve Stokes equation using user defined schur complement >>>>> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >>>>> and ex70.c. In these examples, there are comments about the usage, for >>>>> example: >>>>> >>>>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>>>> >>>>> However the option -fieldsplit_1_user_pc is not recognized by the >>>>> executable. The output of the above command is: >>>>> >>>> >>>> This ("user_pc") is just a mnemonic. What it means is that you can >>>> use any PC in this slot. What he should have written is >>>> >>>> -fieldsplit_1_pc_type >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> residual u = 2.6315e-05 >>>>> >>>>> residual p = 0.000229872 >>>>> >>>>> residual [u,p] = 0.000231373 >>>>> >>>>> discretization error u = 0.00633503 >>>>> >>>>> discretization error p = 0.121534 >>>>> >>>>> discretization error [u,p] = 0.121699 >>>>> >>>>> WARNING! There are options you set that were not used! >>>>> >>>>> WARNING! could be spelling mistake, etc! >>>>> >>>>> Option left: name:-fieldsplit_1_user_pc (no value) >>>>> >>>>> >>>>> I'm wondering what's going on. Is that because I need some specific >>>>> options during my configuring of the package? By the way, the version of >>>>> PETSc I'm using is 3.4.4. >>>>> >>>>> >>>>> Best, >>>>> >>>>> Hui >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jul 16 17:11:40 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Jul 2014 00:11:40 +0200 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: To see all the fieldsplit options, run with ./ex70 -pc_type fieldsplit -help -help will only display the options relevant to the current configuration of your KSP and PC. On 17 July 2014 00:10, Dave May wrote: > Run with > -help > > > On 17 July 2014 00:07, Sun, Hui wrote: > >> Thank you Dave, it does solve the problem. Another question is: since >> some options like >> >> -fieldsplit_0_pc_type >> >> does not show up if I type >> >> ./ex70 -help >> >> how then can I have a complete list of all the options? >> >> Best, >> Hui >> >> >> ------------------------------ >> *From:* Dave May [dave.mayhem23 at gmail.com] >> *Sent:* Wednesday, July 16, 2014 2:57 PM >> *To:* Sun, Hui >> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov >> >> *Subject:* Re: [petsc-users] why a certain option cannot be used >> >> You need to specify on the command line >> -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy >> >> Where xxx and yyy are the desired preconditioners for the (1,1) block >> and (2,2) block. Note that the default PC (ilu(0)) applied to both splits >> will fail for this example. >> >> The comments in the code are misleading and incorrect. Without the >> option -pc_type fieldsplit, all fieldsplit options have no effect. >> >> When debugging solvers, always run with - ksp_view (as Matt recommends) >> >> >> >> On Wednesday, 16 July 2014, Sun, Hui wrote: >> >>> No I don't think I'm using fieldsplitting although I want to use it. >>> The output of >>> >>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view >>> >>> is the following: >>> >>> KSP Object: 1 MPI processes >>> >>> type: gmres >>> >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >>> Orthogonalization with no iterative refinement >>> >>> GMRES: happy breakdown tolerance 1e-30 >>> >>> maximum iterations=10000, initial guess is zero >>> >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> >>> left preconditioning >>> >>> using PRECONDITIONED norm type for convergence test >>> >>> PC Object: 1 MPI processes >>> >>> type: none >>> >>> linear system matrix = precond matrix: >>> >>> Matrix Object: 1 MPI processes >>> >>> type: nest >>> >>> rows=4608, cols=4608 >>> >>> Matrix object: >>> >>> type=nest, rows=2, cols=2 >>> >>> MatNest structure: >>> >>> (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 >>> >>> (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 >>> >>> (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 >>> >>> (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 >>> >>> residual u = 2.6315e-05 >>> >>> residual p = 0.000229872 >>> >>> residual [u,p] = 0.000231373 >>> >>> discretization error u = 0.00633503 >>> >>> discretization error p = 0.121534 >>> >>> discretization error [u,p] = 0.121699 >>> >>> WARNING! There are options you set that were not used! >>> >>> WARNING! could be spelling mistake, etc! >>> >>> Option left: name:-fieldsplit_1_pc_type value: jacobi >>> >>> >>> ------------------------------ >>> *From:* Sun, Hui >>> *Sent:* Wednesday, July 16, 2014 2:40 PM >>> *To:* Matthew Knepley >>> *Cc:* petsc-users at mcs.anl.gov >>> *Subject:* RE: [petsc-users] why a certain option cannot be used >>> >>> Thank you Matt. I've corrected it to >>> >>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi >>> >>> >>> However the output is still: >>> >>> >>> residual u = 2.6315e-05 >>> >>> residual p = 0.000229872 >>> >>> residual [u,p] = 0.000231373 >>> >>> discretization error u = 0.00633503 >>> >>> discretization error p = 0.121534 >>> >>> discretization error [u,p] = 0.121699 >>> >>> WARNING! There are options you set that were not used! >>> >>> WARNING! could be spelling mistake, etc! >>> >>> Option left: name:-fieldsplit_1_pc_type value: jacobi >>> >>> >>> If I type >>> >>> >>> ./ex70 -help | grep fieldsplit_1_ >>> >>> >>> the output is that nothing coming out. >>> >>> >>> >>> >>> >>> ------------------------------ >>> *From:* Matthew Knepley [knepley at gmail.com >>> ] >>> *Sent:* Wednesday, July 16, 2014 2:32 PM >>> *To:* Sun, Hui >>> *Cc:* petsc-users at mcs.anl.gov >>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>> >>> On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley >> > wrote: >>> >>>> On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui >>> > wrote: >>>> >>>>> Thank you Matt for answering, however even with that I don't think >>>>> it works. For example, the command >>>>> >>>>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >>>>> >>>> >>>> You have mistyped. Look carefully at my last message: >>>> >>>> -fieldsplit_1_pc_type jacobi >>>> >>> >>> I also want to note that you can see all the available options with >>> -help. It does produce >>> a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). >>> You can also see the >>> prefix of each solver component using -ksp_view. >>> >>> Matt >>> >>> >>>> Matt >>>> >>>> >>>>> gives me the following output >>>>> >>>>> >>>>> residual u = 2.6315e-05 >>>>> >>>>> residual p = 0.000229872 >>>>> >>>>> residual [u,p] = 0.000231373 >>>>> >>>>> discretization error u = 0.00633503 >>>>> >>>>> discretization error p = 0.121534 >>>>> >>>>> discretization error [u,p] = 0.121699 >>>>> >>>>> WARNING! There are options you set that were not used! >>>>> >>>>> WARNING! could be spelling mistake, etc! >>>>> >>>>> Option left: name:-fieldsplit_1_type value: jacobi >>>>> >>>>> >>>>> Furthermore, if I look into the possible options with keyword >>>>> fieldsplit involved by the command: >>>>> >>>>> >>>>> ./ex70 -help | grep -A5 fieldsplit >>>>> >>>>> >>>>> I get this output: >>>>> >>>>> >>>>> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >>>>> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >>>>> (PCSetType) >>>>> >>>>> -pc_use_amat: use Amat (instead of Pmat) to define >>>>> preconditioner in nested inner solves (PCSetUseAmat) >>>>> >>>>> Krylov Method (KSP) options >>>>> ------------------------------------------------- >>>>> >>>>> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne >>>>> nash stcg gltr richardson >>>>> >>>>> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr >>>>> pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >>>>> specest dgmres (KSPSetType) >>>>> >>>>> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >>>>> >>>>> >>>>> So I don't see any option that's similar to -fieldsplit_1_type. >>>>> >>>>> >>>>> Hui >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* Matthew Knepley [knepley at gmail.com >>>>> ] >>>>> *Sent:* Wednesday, July 16, 2014 2:16 PM >>>>> *To:* Sun, Hui >>>>> *Cc:* petsc-users at mcs.anl.gov >>>>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>>>> >>>>> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui >>>> > wrote: >>>>> >>>>>> I want to solve Stokes equation using user defined schur complement >>>>>> preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, >>>>>> and ex70.c. In these examples, there are comments about the usage, for >>>>>> example: >>>>>> >>>>>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>>>>> >>>>>> However the option -fieldsplit_1_user_pc is not recognized by the >>>>>> executable. The output of the above command is: >>>>>> >>>>> >>>>> This ("user_pc") is just a mnemonic. What it means is that you can >>>>> use any PC in this slot. What he should have written is >>>>> >>>>> -fieldsplit_1_pc_type >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> residual u = 2.6315e-05 >>>>>> >>>>>> residual p = 0.000229872 >>>>>> >>>>>> residual [u,p] = 0.000231373 >>>>>> >>>>>> discretization error u = 0.00633503 >>>>>> >>>>>> discretization error p = 0.121534 >>>>>> >>>>>> discretization error [u,p] = 0.121699 >>>>>> >>>>>> WARNING! There are options you set that were not used! >>>>>> >>>>>> WARNING! could be spelling mistake, etc! >>>>>> >>>>>> Option left: name:-fieldsplit_1_user_pc (no value) >>>>>> >>>>>> >>>>>> I'm wondering what's going on. Is that because I need some specific >>>>>> options during my configuring of the package? By the way, the version of >>>>>> PETSc I'm using is 3.4.4. >>>>>> >>>>>> >>>>>> Best, >>>>>> >>>>>> Hui >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 17:24:20 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 22:24:20 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> , Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Dave. It does give me the other options. However I want to be able to use the user defined fieldsplit_1_pc_type. In example ex70.c, there is a user defined Schur complement preconditioner. I want to use that. Then how should I set the values for the options? I tried to see which values could be set by typing ./ex70 -pc_type fieldsplit -help 2>/dev/null | grep -A5 fieldsplit_1_pc And I get the following output: -fieldsplit_1_pc_type : Preconditioner (one of) none jacobi pbjacobi bjacobi sor lu shell mg eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -fieldsplit_1_pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Block Jacobi options -fieldsplit_1_pc_bjacobi_blocks <-1>: Total number of blocks (PCBJacobiSetTotalBlocks) Krylov Method (KSP) options ------------------------------------------------- -fieldsplit_1_ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -fieldsplit_1_ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) -fieldsplit_1_ksp_rtol <1e-05>: Relative decrease in residual norm (KSPSetTolerances) I cannot find anything here about the user defined pc. Do you have any idea what might be the choice? ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:11 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used To see all the fieldsplit options, run with ./ex70 -pc_type fieldsplit -help -help will only display the options relevant to the current configuration of your KSP and PC. On 17 July 2014 00:10, Dave May > wrote: Run with -help On 17 July 2014 00:07, Sun, Hui > wrote: Thank you Dave, it does solve the problem. Another question is: since some options like -fieldsplit_0_pc_type does not show up if I type ./ex70 -help how then can I have a complete list of all the options? Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 2:57 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui > wrote: No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Jul 16 17:40:20 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Jul 2014 00:40:20 +0200 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: You need to specify the fieldsplit preconditioning type. The choices are -pc_fieldsplit_type (choose one of) ADDITIVE MULTIPLICATIVE SYMMETRIC_MULTIPLICATIVE SPECIAL SCHUR Maybe what you want to run is the following ./ex70 -ksp_view -pc_type fieldsplit -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi -pc_fieldsplit_type SCHUR -pc_fieldsplit_schur_precondition user The option " -fieldsplit_1_user_pc" seems meaningless to me. I don't see anything in ex70.c which would cause this option to have any effect. The lines ierr = PetscOptionsHasName(NULL, "-user_pc", &s.userPC);CHKERRQ(ierr); ierr = PetscOptionsHasName(NULL, "-user_ksp", &s.userKSP);CHKERRQ(ierr); imply that the option -user_pc and -user_ksp are meaningful and modify how the schur complement preconditioner will be configured. The first option triggers this setup to occur; if (s->userPC) { ierr = PCFieldSplitSchurPrecondition(pc, PC_FIELDSPLIT_SCHUR_PRE_USER, s->myS);CHKERRQ(ierr); } the second option will force this setup to occur if (s->userKSP) { ierr = PCSetUp(pc);CHKERRQ(ierr); ierr = PCFieldSplitGetSubKSP(pc, &n, &subksp);CHKERRQ(ierr); ierr = KSPSetOperators(subksp[1], s->myS, s->myS, SAME_PRECONDITIONER);CHKERRQ(ierr); ierr = PetscFree(subksp);CHKERRQ(ierr); } On 17 July 2014 00:24, Sun, Hui wrote: > Thank you Dave. It does give me the other options. However I want to be > able to use the user defined fieldsplit_1_pc_type. In example ex70.c, there > is a user defined Schur complement preconditioner. I want to use that. Then > how should I set the values for the options? > > I tried to see which values could be set by typing > > ./ex70 -pc_type fieldsplit -help 2>/dev/null | grep -A5 fieldsplit_1_pc > > And I get the following output: > > -fieldsplit_1_pc_type : Preconditioner (one of) none jacobi > pbjacobi bjacobi sor lu shell mg > > eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat > fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs > (PCSetType) > > -fieldsplit_1_pc_use_amat: use Amat (instead of Pmat) to define > preconditioner in nested inner solves (PCSetUseAmat) > > Block Jacobi options > > -fieldsplit_1_pc_bjacobi_blocks <-1>: Total number of blocks > (PCBJacobiSetTotalBlocks) > > Krylov Method (KSP) options > ------------------------------------------------- > > -fieldsplit_1_ksp_type : Krylov method (one of) cg groppcg > pipecg cgne nash stcg gltr richardson > > chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr > pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres > specest dgmres (KSPSetType) > > -fieldsplit_1_ksp_max_it <10000>: Maximum number of iterations > (KSPSetTolerances) > -fieldsplit_1_ksp_rtol <1e-05>: Relative decrease in residual norm > (KSPSetTolerances) > > I cannot find anything here about the user defined pc. Do you have any > idea what might be the choice? > > > > ------------------------------ > *From:* Dave May [dave.mayhem23 at gmail.com] > *Sent:* Wednesday, July 16, 2014 3:11 PM > > *To:* Sun, Hui > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] why a certain option cannot be used > > To see all the fieldsplit options, run with > ./ex70 -pc_type fieldsplit -help > > -help will only display the options relevant to the current configuration > of your KSP and PC. > > > On 17 July 2014 00:10, Dave May wrote: > >> Run with >> -help >> >> >> On 17 July 2014 00:07, Sun, Hui wrote: >> >>> Thank you Dave, it does solve the problem. Another question is: since >>> some options like >>> >>> -fieldsplit_0_pc_type >>> >>> does not show up if I type >>> >>> ./ex70 -help >>> >>> how then can I have a complete list of all the options? >>> >>> Best, >>> Hui >>> >>> >>> ------------------------------ >>> *From:* Dave May [dave.mayhem23 at gmail.com] >>> *Sent:* Wednesday, July 16, 2014 2:57 PM >>> *To:* Sun, Hui >>> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov >>> >>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>> >>> You need to specify on the command line >>> -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy >>> >>> Where xxx and yyy are the desired preconditioners for the (1,1) block >>> and (2,2) block. Note that the default PC (ilu(0)) applied to both splits >>> will fail for this example. >>> >>> The comments in the code are misleading and incorrect. Without the >>> option -pc_type fieldsplit, all fieldsplit options have no effect. >>> >>> When debugging solvers, always run with - ksp_view (as Matt >>> recommends) >>> >>> >>> >>> On Wednesday, 16 July 2014, Sun, Hui wrote: >>> >>>> No I don't think I'm using fieldsplitting although I want to use it. >>>> The output of >>>> >>>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view >>>> >>>> is the following: >>>> >>>> KSP Object: 1 MPI processes >>>> >>>> type: gmres >>>> >>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >>>> Orthogonalization with no iterative refinement >>>> >>>> GMRES: happy breakdown tolerance 1e-30 >>>> >>>> maximum iterations=10000, initial guess is zero >>>> >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> >>>> left preconditioning >>>> >>>> using PRECONDITIONED norm type for convergence test >>>> >>>> PC Object: 1 MPI processes >>>> >>>> type: none >>>> >>>> linear system matrix = precond matrix: >>>> >>>> Matrix Object: 1 MPI processes >>>> >>>> type: nest >>>> >>>> rows=4608, cols=4608 >>>> >>>> Matrix object: >>>> >>>> type=nest, rows=2, cols=2 >>>> >>>> MatNest structure: >>>> >>>> (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 >>>> >>>> (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 >>>> >>>> (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 >>>> >>>> (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 >>>> >>>> residual u = 2.6315e-05 >>>> >>>> residual p = 0.000229872 >>>> >>>> residual [u,p] = 0.000231373 >>>> >>>> discretization error u = 0.00633503 >>>> >>>> discretization error p = 0.121534 >>>> >>>> discretization error [u,p] = 0.121699 >>>> >>>> WARNING! There are options you set that were not used! >>>> >>>> WARNING! could be spelling mistake, etc! >>>> >>>> Option left: name:-fieldsplit_1_pc_type value: jacobi >>>> >>>> >>>> ------------------------------ >>>> *From:* Sun, Hui >>>> *Sent:* Wednesday, July 16, 2014 2:40 PM >>>> *To:* Matthew Knepley >>>> *Cc:* petsc-users at mcs.anl.gov >>>> *Subject:* RE: [petsc-users] why a certain option cannot be used >>>> >>>> Thank you Matt. I've corrected it to >>>> >>>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi >>>> >>>> >>>> However the output is still: >>>> >>>> >>>> residual u = 2.6315e-05 >>>> >>>> residual p = 0.000229872 >>>> >>>> residual [u,p] = 0.000231373 >>>> >>>> discretization error u = 0.00633503 >>>> >>>> discretization error p = 0.121534 >>>> >>>> discretization error [u,p] = 0.121699 >>>> >>>> WARNING! There are options you set that were not used! >>>> >>>> WARNING! could be spelling mistake, etc! >>>> >>>> Option left: name:-fieldsplit_1_pc_type value: jacobi >>>> >>>> >>>> If I type >>>> >>>> >>>> ./ex70 -help | grep fieldsplit_1_ >>>> >>>> >>>> the output is that nothing coming out. >>>> >>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* Matthew Knepley [knepley at gmail.com >>>> ] >>>> *Sent:* Wednesday, July 16, 2014 2:32 PM >>>> *To:* Sun, Hui >>>> *Cc:* petsc-users at mcs.anl.gov >>>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>>> >>>> On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley >>> > wrote: >>>> >>>>> On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui >>>> > wrote: >>>>> >>>>>> Thank you Matt for answering, however even with that I don't think >>>>>> it works. For example, the command >>>>>> >>>>>> ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi >>>>>> >>>>> >>>>> You have mistyped. Look carefully at my last message: >>>>> >>>>> -fieldsplit_1_pc_type jacobi >>>>> >>>> >>>> I also want to note that you can see all the available options with >>>> -help. It does produce >>>> a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). >>>> You can also see the >>>> prefix of each solver component using -ksp_view. >>>> >>>> Matt >>>> >>>> >>>>> Matt >>>>> >>>>> >>>>>> gives me the following output >>>>>> >>>>>> >>>>>> residual u = 2.6315e-05 >>>>>> >>>>>> residual p = 0.000229872 >>>>>> >>>>>> residual [u,p] = 0.000231373 >>>>>> >>>>>> discretization error u = 0.00633503 >>>>>> >>>>>> discretization error p = 0.121534 >>>>>> >>>>>> discretization error [u,p] = 0.121699 >>>>>> >>>>>> WARNING! There are options you set that were not used! >>>>>> >>>>>> WARNING! could be spelling mistake, etc! >>>>>> >>>>>> Option left: name:-fieldsplit_1_type value: jacobi >>>>>> >>>>>> >>>>>> Furthermore, if I look into the possible options with keyword >>>>>> fieldsplit involved by the command: >>>>>> >>>>>> >>>>>> ./ex70 -help | grep -A5 fieldsplit >>>>>> >>>>>> >>>>>> I get this output: >>>>>> >>>>>> >>>>>> eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat >>>>>> fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs >>>>>> (PCSetType) >>>>>> >>>>>> -pc_use_amat: use Amat (instead of Pmat) to define >>>>>> preconditioner in nested inner solves (PCSetUseAmat) >>>>>> >>>>>> Krylov Method (KSP) options >>>>>> ------------------------------------------------- >>>>>> >>>>>> -ksp_type : Krylov method (one of) cg groppcg pipecg cgne >>>>>> nash stcg gltr richardson >>>>>> >>>>>> chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr >>>>>> cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres >>>>>> specest dgmres (KSPSetType) >>>>>> >>>>>> -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) >>>>>> >>>>>> >>>>>> So I don't see any option that's similar to -fieldsplit_1_type. >>>>>> >>>>>> >>>>>> Hui >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* Matthew Knepley [knepley at gmail.com >>>>>> ] >>>>>> *Sent:* Wednesday, July 16, 2014 2:16 PM >>>>>> *To:* Sun, Hui >>>>>> *Cc:* petsc-users at mcs.anl.gov >>>>>> *Subject:* Re: [petsc-users] why a certain option cannot be used >>>>>> >>>>>> On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui >>>>> > wrote: >>>>>> >>>>>>> I want to solve Stokes equation using user defined schur >>>>>>> complement preconditioner. So I'm reading and testing >>>>>>> snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are >>>>>>> comments about the usage, for example: >>>>>>> >>>>>>> mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. >>>>>>> >>>>>>> However the option -fieldsplit_1_user_pc is not recognized by the >>>>>>> executable. The output of the above command is: >>>>>>> >>>>>> >>>>>> This ("user_pc") is just a mnemonic. What it means is that you can >>>>>> use any PC in this slot. What he should have written is >>>>>> >>>>>> -fieldsplit_1_pc_type >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> residual u = 2.6315e-05 >>>>>>> >>>>>>> residual p = 0.000229872 >>>>>>> >>>>>>> residual [u,p] = 0.000231373 >>>>>>> >>>>>>> discretization error u = 0.00633503 >>>>>>> >>>>>>> discretization error p = 0.121534 >>>>>>> >>>>>>> discretization error [u,p] = 0.121699 >>>>>>> >>>>>>> WARNING! There are options you set that were not used! >>>>>>> >>>>>>> WARNING! could be spelling mistake, etc! >>>>>>> >>>>>>> Option left: name:-fieldsplit_1_user_pc (no value) >>>>>>> >>>>>>> >>>>>>> I'm wondering what's going on. Is that because I need some >>>>>>> specific options during my configuring of the package? By the way, the >>>>>>> version of PETSc I'm using is 3.4.4. >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Hui >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Wed Jul 16 18:28:17 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 16 Jul 2014 23:28:17 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC055@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Dave! This information is really helpful to me. Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:40 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify the fieldsplit preconditioning type. The choices are -pc_fieldsplit_type (choose one of) ADDITIVE MULTIPLICATIVE SYMMETRIC_MULTIPLICATIVE SPECIAL SCHUR Maybe what you want to run is the following ./ex70 -ksp_view -pc_type fieldsplit -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi -pc_fieldsplit_type SCHUR -pc_fieldsplit_schur_precondition user The option " -fieldsplit_1_user_pc" seems meaningless to me. I don't see anything in ex70.c which would cause this option to have any effect. The lines ierr = PetscOptionsHasName(NULL, "-user_pc", &s.userPC);CHKERRQ(ierr); ierr = PetscOptionsHasName(NULL, "-user_ksp", &s.userKSP);CHKERRQ(ierr); imply that the option -user_pc and -user_ksp are meaningful and modify how the schur complement preconditioner will be configured. The first option triggers this setup to occur; if (s->userPC) { ierr = PCFieldSplitSchurPrecondition(pc, PC_FIELDSPLIT_SCHUR_PRE_USER, s->myS);CHKERRQ(ierr); } the second option will force this setup to occur if (s->userKSP) { ierr = PCSetUp(pc);CHKERRQ(ierr); ierr = PCFieldSplitGetSubKSP(pc, &n, &subksp);CHKERRQ(ierr); ierr = KSPSetOperators(subksp[1], s->myS, s->myS, SAME_PRECONDITIONER);CHKERRQ(ierr); ierr = PetscFree(subksp);CHKERRQ(ierr); } On 17 July 2014 00:24, Sun, Hui > wrote: Thank you Dave. It does give me the other options. However I want to be able to use the user defined fieldsplit_1_pc_type. In example ex70.c, there is a user defined Schur complement preconditioner. I want to use that. Then how should I set the values for the options? I tried to see which values could be set by typing ./ex70 -pc_type fieldsplit -help 2>/dev/null | grep -A5 fieldsplit_1_pc And I get the following output: -fieldsplit_1_pc_type : Preconditioner (one of) none jacobi pbjacobi bjacobi sor lu shell mg eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -fieldsplit_1_pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Block Jacobi options -fieldsplit_1_pc_bjacobi_blocks <-1>: Total number of blocks (PCBJacobiSetTotalBlocks) Krylov Method (KSP) options ------------------------------------------------- -fieldsplit_1_ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -fieldsplit_1_ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) -fieldsplit_1_ksp_rtol <1e-05>: Relative decrease in residual norm (KSPSetTolerances) I cannot find anything here about the user defined pc. Do you have any idea what might be the choice? ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:11 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used To see all the fieldsplit options, run with ./ex70 -pc_type fieldsplit -help -help will only display the options relevant to the current configuration of your KSP and PC. On 17 July 2014 00:10, Dave May > wrote: Run with -help On 17 July 2014 00:07, Sun, Hui > wrote: Thank you Dave, it does solve the problem. Another question is: since some options like -fieldsplit_0_pc_type does not show up if I type ./ex70 -help how then can I have a complete list of all the options? Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 2:57 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui > wrote: No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Thu Jul 17 00:11:05 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Thu, 17 Jul 2014 05:11:05 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> Dave, and other PETSc users, sorry for bothering you all again. However continuing this topic, I have another question. Within this ex70.c, there is a user defined schur complement StokesSetupApproxSchur, defined as /* Schur complement approximation: myS = A11 - A10 diag(A00)^(-1) A01 */ The preconditioner here can be written out explicitly as a matrix, and is not changed between iterations. Each time it is applied to the residue is simply matrix times vector. However, if I want to define a preconditioner which is allowed to vary between iterations, for example, if I need to solve a linear system each time I apply the preconditioner to the residue, then is there a way I can do this? Any hint will be greatly appreciated. Thank you very much! Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:40 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify the fieldsplit preconditioning type. The choices are -pc_fieldsplit_type (choose one of) ADDITIVE MULTIPLICATIVE SYMMETRIC_MULTIPLICATIVE SPECIAL SCHUR Maybe what you want to run is the following ./ex70 -ksp_view -pc_type fieldsplit -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi -pc_fieldsplit_type SCHUR -pc_fieldsplit_schur_precondition user The option " -fieldsplit_1_user_pc" seems meaningless to me. I don't see anything in ex70.c which would cause this option to have any effect. The lines ierr = PetscOptionsHasName(NULL, "-user_pc", &s.userPC);CHKERRQ(ierr); ierr = PetscOptionsHasName(NULL, "-user_ksp", &s.userKSP);CHKERRQ(ierr); imply that the option -user_pc and -user_ksp are meaningful and modify how the schur complement preconditioner will be configured. The first option triggers this setup to occur; if (s->userPC) { ierr = PCFieldSplitSchurPrecondition(pc, PC_FIELDSPLIT_SCHUR_PRE_USER, s->myS);CHKERRQ(ierr); } the second option will force this setup to occur if (s->userKSP) { ierr = PCSetUp(pc);CHKERRQ(ierr); ierr = PCFieldSplitGetSubKSP(pc, &n, &subksp);CHKERRQ(ierr); ierr = KSPSetOperators(subksp[1], s->myS, s->myS, SAME_PRECONDITIONER);CHKERRQ(ierr); ierr = PetscFree(subksp);CHKERRQ(ierr); } On 17 July 2014 00:24, Sun, Hui > wrote: Thank you Dave. It does give me the other options. However I want to be able to use the user defined fieldsplit_1_pc_type. In example ex70.c, there is a user defined Schur complement preconditioner. I want to use that. Then how should I set the values for the options? I tried to see which values could be set by typing ./ex70 -pc_type fieldsplit -help 2>/dev/null | grep -A5 fieldsplit_1_pc And I get the following output: -fieldsplit_1_pc_type : Preconditioner (one of) none jacobi pbjacobi bjacobi sor lu shell mg eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -fieldsplit_1_pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Block Jacobi options -fieldsplit_1_pc_bjacobi_blocks <-1>: Total number of blocks (PCBJacobiSetTotalBlocks) Krylov Method (KSP) options ------------------------------------------------- -fieldsplit_1_ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -fieldsplit_1_ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) -fieldsplit_1_ksp_rtol <1e-05>: Relative decrease in residual norm (KSPSetTolerances) I cannot find anything here about the user defined pc. Do you have any idea what might be the choice? ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:11 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used To see all the fieldsplit options, run with ./ex70 -pc_type fieldsplit -help -help will only display the options relevant to the current configuration of your KSP and PC. On 17 July 2014 00:10, Dave May > wrote: Run with -help On 17 July 2014 00:07, Sun, Hui > wrote: Thank you Dave, it does solve the problem. Another question is: since some options like -fieldsplit_0_pc_type does not show up if I type ./ex70 -help how then can I have a complete list of all the options? Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 2:57 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui > wrote: No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jul 17 00:16:53 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 16 Jul 2014 23:16:53 -0600 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <877g3cwomy.fsf@jedbrown.org> "Sun, Hui" writes: > Dave, and other PETSc users, sorry for bothering you all again. However continuing this topic, I have another question. Within this ex70.c, there is a user defined schur complement StokesSetupApproxSchur, defined as > > > /* Schur complement approximation: myS = A11 - A10 diag(A00)^(-1) A01 */ > > The preconditioner here can be written out explicitly as a matrix, and > is not changed between iterations. Each time it is applied to the > residue is simply matrix times vector. However, if I want to define a > preconditioner which is allowed to vary between iterations, for > example, if I need to solve a linear system each time I apply the > preconditioner to the residue, then is there a way I can do this? Any > hint will be greatly appreciated. Thank you very much! You either write your own PCShell or compose a PC from existing components (perhaps via options). In any case, the application of this PC will involve calls to KSPSolve. You can configure those to be iterative if that makes sense for your use case. Note that if you run an inexact inner solve, the outer solves should use a flexible method (e.g., FGMRES or GCR). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hus003 at ucsd.edu Thu Jul 17 00:18:00 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Thu, 17 Jul 2014 05:18:00 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU>, , <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC081@XMAIL-MBX-BH1.AD.UCSD.EDU> Sorry, don't worry about my previous post. I just found out that ex55.c might have some hint to this, let me dig out more before I ask any questions. ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 10:11 PM To: Dave May Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Dave, and other PETSc users, sorry for bothering you all again. However continuing this topic, I have another question. Within this ex70.c, there is a user defined schur complement StokesSetupApproxSchur, defined as /* Schur complement approximation: myS = A11 - A10 diag(A00)^(-1) A01 */ The preconditioner here can be written out explicitly as a matrix, and is not changed between iterations. Each time it is applied to the residue is simply matrix times vector. However, if I want to define a preconditioner which is allowed to vary between iterations, for example, if I need to solve a linear system each time I apply the preconditioner to the residue, then is there a way I can do this? Any hint will be greatly appreciated. Thank you very much! Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:40 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify the fieldsplit preconditioning type. The choices are -pc_fieldsplit_type (choose one of) ADDITIVE MULTIPLICATIVE SYMMETRIC_MULTIPLICATIVE SPECIAL SCHUR Maybe what you want to run is the following ./ex70 -ksp_view -pc_type fieldsplit -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi -pc_fieldsplit_type SCHUR -pc_fieldsplit_schur_precondition user The option " -fieldsplit_1_user_pc" seems meaningless to me. I don't see anything in ex70.c which would cause this option to have any effect. The lines ierr = PetscOptionsHasName(NULL, "-user_pc", &s.userPC);CHKERRQ(ierr); ierr = PetscOptionsHasName(NULL, "-user_ksp", &s.userKSP);CHKERRQ(ierr); imply that the option -user_pc and -user_ksp are meaningful and modify how the schur complement preconditioner will be configured. The first option triggers this setup to occur; if (s->userPC) { ierr = PCFieldSplitSchurPrecondition(pc, PC_FIELDSPLIT_SCHUR_PRE_USER, s->myS);CHKERRQ(ierr); } the second option will force this setup to occur if (s->userKSP) { ierr = PCSetUp(pc);CHKERRQ(ierr); ierr = PCFieldSplitGetSubKSP(pc, &n, &subksp);CHKERRQ(ierr); ierr = KSPSetOperators(subksp[1], s->myS, s->myS, SAME_PRECONDITIONER);CHKERRQ(ierr); ierr = PetscFree(subksp);CHKERRQ(ierr); } On 17 July 2014 00:24, Sun, Hui > wrote: Thank you Dave. It does give me the other options. However I want to be able to use the user defined fieldsplit_1_pc_type. In example ex70.c, there is a user defined Schur complement preconditioner. I want to use that. Then how should I set the values for the options? I tried to see which values could be set by typing ./ex70 -pc_type fieldsplit -help 2>/dev/null | grep -A5 fieldsplit_1_pc And I get the following output: -fieldsplit_1_pc_type : Preconditioner (one of) none jacobi pbjacobi bjacobi sor lu shell mg eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -fieldsplit_1_pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Block Jacobi options -fieldsplit_1_pc_bjacobi_blocks <-1>: Total number of blocks (PCBJacobiSetTotalBlocks) Krylov Method (KSP) options ------------------------------------------------- -fieldsplit_1_ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -fieldsplit_1_ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) -fieldsplit_1_ksp_rtol <1e-05>: Relative decrease in residual norm (KSPSetTolerances) I cannot find anything here about the user defined pc. Do you have any idea what might be the choice? ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 3:11 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used To see all the fieldsplit options, run with ./ex70 -pc_type fieldsplit -help -help will only display the options relevant to the current configuration of your KSP and PC. On 17 July 2014 00:10, Dave May > wrote: Run with -help On 17 July 2014 00:07, Sun, Hui > wrote: Thank you Dave, it does solve the problem. Another question is: since some options like -fieldsplit_0_pc_type does not show up if I type ./ex70 -help how then can I have a complete list of all the options? Best, Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Wednesday, July 16, 2014 2:57 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used You need to specify on the command line -pc_type fieldsplit -fieldsplit_0_pc_type xxx -fieldsplit_1_pc_type yyy Where xxx and yyy are the desired preconditioners for the (1,1) block and (2,2) block. Note that the default PC (ilu(0)) applied to both splits will fail for this example. The comments in the code are misleading and incorrect. Without the option -pc_type fieldsplit, all fieldsplit options have no effect. When debugging solvers, always run with - ksp_view (as Matt recommends) On Wednesday, 16 July 2014, Sun, Hui > wrote: No I don't think I'm using fieldsplitting although I want to use it. The output of ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi -ksp_view is the following: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: nest rows=4608, cols=4608 Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="a00_", type=mpiaij, rows=3072, cols=3072 (0,1) : prefix="a01_", type=mpiaij, rows=3072, cols=1536 (1,0) : prefix="a10_", type=mpiaij, rows=1536, cols=3072 (1,1) : prefix="a11_", type=mpiaij, rows=1536, cols=1536 residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi ________________________________ From: Sun, Hui Sent: Wednesday, July 16, 2014 2:40 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used Thank you Matt. I've corrected it to ./ex70 -nx 32 -ny 48 -fieldsplit_1_pc_type jacobi However the output is still: residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_pc_type value: jacobi If I type ./ex70 -help | grep fieldsplit_1_ the output is that nothing coming out. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:32 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:25 PM, Matthew Knepley > wrote: On Wed, Jul 16, 2014 at 4:22 PM, Sun, Hui > wrote: Thank you Matt for answering, however even with that I don't think it works. For example, the command ./ex70 -nx 32 -ny 48 -fieldsplit_1_type jacobi You have mistyped. Look carefully at my last message: -fieldsplit_1_pc_type jacobi I also want to note that you can see all the available options with -help. It does produce a lot of output, but you can segregate it by prefix ("fieldsplit_1_"). You can also see the prefix of each solver component using -ksp_view. Matt Matt gives me the following output residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_type value: jacobi Furthermore, if I look into the possible options with keyword fieldsplit involved by the command: ./ex70 -help | grep -A5 fieldsplit I get this output: eisenstat ilu icc cholesky asm gasm ksp composite redundant nn mat fieldsplit galerkin exotic hmpi asa cp lsc redistribute svd gamg tfs (PCSetType) -pc_use_amat: use Amat (instead of Pmat) to define preconditioner in nested inner solves (PCSetUseAmat) Krylov Method (KSP) options ------------------------------------------------- -ksp_type : Krylov method (one of) cg groppcg pipecg cgne nash stcg gltr richardson chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_max_it <10000>: Maximum number of iterations (KSPSetTolerances) So I don't see any option that's similar to -fieldsplit_1_type. Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Wednesday, July 16, 2014 2:16 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used On Wed, Jul 16, 2014 at 4:09 PM, Sun, Hui > wrote: I want to solve Stokes equation using user defined schur complement preconditioner. So I'm reading and testing snes/examples/tutorial/ex55.c, and ex70.c. In these examples, there are comments about the usage, for example: mpiexec -n 2 ./stokes -nx 32 -ny 48 -fieldsplit_1_user_pc. However the option -fieldsplit_1_user_pc is not recognized by the executable. The output of the above command is: This ("user_pc") is just a mnemonic. What it means is that you can use any PC in this slot. What he should have written is -fieldsplit_1_pc_type Thanks, Matt residual u = 2.6315e-05 residual p = 0.000229872 residual [u,p] = 0.000231373 discretization error u = 0.00633503 discretization error p = 0.121534 discretization error [u,p] = 0.121699 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_1_user_pc (no value) I'm wondering what's going on. Is that because I need some specific options during my configuring of the package? By the way, the version of PETSc I'm using is 3.4.4. Best, Hui -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Thu Jul 17 00:20:01 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Thu, 17 Jul 2014 05:20:01 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <877g3cwomy.fsf@jedbrown.org> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU>, <877g3cwomy.fsf@jedbrown.org> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC093@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Jed. I will look into that. ________________________________________ From: Jed Brown [jed at jedbrown.org] Sent: Wednesday, July 16, 2014 10:16 PM To: Sun, Hui; Dave May Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used "Sun, Hui" writes: > Dave, and other PETSc users, sorry for bothering you all again. However continuing this topic, I have another question. Within this ex70.c, there is a user defined schur complement StokesSetupApproxSchur, defined as > > > /* Schur complement approximation: myS = A11 - A10 diag(A00)^(-1) A01 */ > > The preconditioner here can be written out explicitly as a matrix, and > is not changed between iterations. Each time it is applied to the > residue is simply matrix times vector. However, if I want to define a > preconditioner which is allowed to vary between iterations, for > example, if I need to solve a linear system each time I apply the > preconditioner to the residue, then is there a way I can do this? Any > hint will be greatly appreciated. Thank you very much! You either write your own PCShell or compose a PC from existing components (perhaps via options). In any case, the application of this PC will involve calls to KSPSolve. You can configure those to be iterative if that makes sense for your use case. Note that if you run an inexact inner solve, the outer solves should use a flexible method (e.g., FGMRES or GCR). From C.Klaij at marin.nl Thu Jul 17 01:41:12 2014 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 17 Jul 2014 06:41:12 +0000 Subject: [petsc-users] why a certain option cannot be used Message-ID: <7bdfbdcfb61540aeafdb3447e16e06e4@MAR190n2.marin.local> Just for future reference, below are the correct comments that should be at the top of ex70. (Jed, Matt, I sent these to the maintainer list a long time ago ([petsc-maint #155997]), but you never put them in, it would avoid a lot of confusion) /* Poiseuille flow problem. */ /* */ /* Viscous, laminar flow in a 2D channel with parabolic velocity */ /* profile and linear pressure drop, exact solution of the 2D Stokes */ /* equations. */ /* */ /* Discretized with the cell-centered finite-volume method on a */ /* Cartesian grid with co-located variables. Variables ordered as */ /* [u1...uN v1...vN p1...pN]^T. Matrix [A00 A01; A10, A11] solved with */ /* PCFIELDSPLIT. Lower factorization is used to mimick the Semi-Implicit */ /* Method for Pressure Linked Equations (SIMPLE) used as preconditioner */ /* instead of solver. */ /* */ /* Disclaimer: does not contain the pressure-weighed interpolation */ /* method needed to suppress spurious pressure modes in real-life */ /* problems. */ /* */ /* usage: */ /* */ /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -fieldsplit_1_pc_type none /* */ /* Runs with PCFIELDSPLIT on 32x48 grid, no PC for the Schur */ /* complement because A11 is zero. FGMRES is needed because */ /* PCFIELDSPLIT is a variable preconditioner. */ /* */ /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -user_pc /* */ /* Same as above but with user defined PC for the true Schur */ /* complement. PC based on the SIMPLE-type approximation (inverse of */ /* A00 approximated by inverse of its diagonal). */ /* */ /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -user_ksp /* */ /* Replace the true Schur complement with a user defined Schur */ /* complement based on the SIMPLE-type approximation. Same matrix is */ /* used as PC. */ /* */ /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi -fieldsplit_1_pc_type jacobi -fieldsplit_1_inner_ksp_type preonly -fieldsplit_1_inner_pc_type jacobi -fieldsplit_1_upper_ksp_type preonly -fieldsplit_1_upper_pc_type jacobi /* */ /* Out-of-the-box SIMPLE-type preconditioning. The major advantage */ /* is that the user neither needs to provide the approximation of */ /* the Schur complement, nor the corresponding preconditioner. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From bichinhoverde at spwinternet.com.br Thu Jul 17 06:35:54 2014 From: bichinhoverde at spwinternet.com.br (Italo Tasso) Date: Thu, 17 Jul 2014 08:35:54 -0300 Subject: [petsc-users] How to efficiently assemble this matrix? Message-ID: I have two matrices: matrix A is 4n x 4n and matrix C is n x n. I want to set the values of A using the values of C, like this: A[4i+k][4j+k] = C[i][j] 0 < i < n 0 < j < n 0 < k < 3 How can I do this efficiently? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 17 08:53:28 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 17 Jul 2014 08:53:28 -0500 Subject: [petsc-users] How to efficiently assemble this matrix? In-Reply-To: References: Message-ID: On Thu, Jul 17, 2014 at 6:35 AM, Italo Tasso < bichinhoverde at spwinternet.com.br> wrote: > I have two matrices: matrix A is 4n x 4n and matrix C is n x n. I want to > set the values of A using the values of C, like this: > > A[4i+k][4j+k] = C[i][j] > 0 < i < n > 0 < j < n > 0 < k < 3 > > How can I do this efficiently? > The best way to do it is to assemble directly into the big matrix. You can do this by explicitly translating indices, or use something like this http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jul 17 08:54:37 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 17 Jul 2014 07:54:37 -0600 Subject: [petsc-users] How to efficiently assemble this matrix? In-Reply-To: References: Message-ID: <87sim0um3m.fsf@jedbrown.org> Italo Tasso writes: > I have two matrices: matrix A is 4n x 4n and matrix C is n x n. I want to > set the values of A using the values of C, like this: > > A[4i+k][4j+k] = C[i][j] > 0 < i < n > 0 < j < n > 0 < k < 3 MatCreateMAIJ(C,4,&A) will create this linear operator in O(1) time and space (just referencing C). Is that what you want? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bichinhoverde at spwinternet.com.br Thu Jul 17 13:50:02 2014 From: bichinhoverde at spwinternet.com.br (Italo Tasso) Date: Thu, 17 Jul 2014 15:50:02 -0300 Subject: [petsc-users] How to efficiently assemble this matrix? In-Reply-To: References: Message-ID: Thanks Matt and Jed. After further analysis, it turns out that the operation I must perform is even more complicated than what I initially described. So direct assembly seems to be the only option. I ended up using MatGetRow and MatSetValues. On Thu, Jul 17, 2014 at 10:53 AM, Matthew Knepley wrote: > On Thu, Jul 17, 2014 at 6:35 AM, Italo Tasso < > bichinhoverde at spwinternet.com.br> wrote: > >> I have two matrices: matrix A is 4n x 4n and matrix C is n x n. I want to >> set the values of A using the values of C, like this: >> >> A[4i+k][4j+k] = C[i][j] >> 0 < i < n >> 0 < j < n >> 0 < k < 3 >> >> How can I do this efficiently? >> > > The best way to do it is to assemble directly into the big matrix. You can > do this by explicitly > translating indices, or use something like this > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Jul 17 13:58:45 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 17 Jul 2014 12:58:45 -0600 Subject: [petsc-users] How to efficiently assemble this matrix? In-Reply-To: References: Message-ID: <878unru80q.fsf@jedbrown.org> Italo Tasso writes: > Thanks Matt and Jed. After further analysis, it turns out that the > operation I must perform is even more complicated than what I initially > described. So direct assembly seems to be the only option. I ended up using > MatGetRow and MatSetValues. Depending in what it is, a new "MatTAIJ" format that we're working on might serve you well. This currently deals with matrices of the form I \otimes S + J \otimes T where J is a distributed sparse matrix and S and T are (smallish) non-distributed matrices. The matrix you originally described is J \otimes I_4 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jbakosi at lanl.gov Thu Jul 17 17:28:40 2014 From: jbakosi at lanl.gov (Jozsef Bakosi) Date: Thu, 17 Jul 2014 16:28:40 -0600 Subject: [petsc-users] Symmetric non-positive definite system solved with CG+ML - why does it not fail? Message-ID: <20140717222839.GC6840@karman.lanl.gov> Hi folks, I have a matrix as a result of a finite element discretization of the Poisson operator and associated right hand side. As it turns out, the matrix is symmetric but not positive definite since it has at least two negative small eigenvalues. I have been solving this system without problem using the conjugate gradients (CG) algorithm with ML as a preconditioner, but I'm wondering why it works. Shouldn't CG fail for a non-positive-definite matrix? Does PETSc do some additional magic if it detects that the dot-product in the CG algorithm is negative? Does it solve the system using the normal equations, A'A, instead? Please let me know if I should send the matrix + rhs. Thanks, Jozsef From jed at jedbrown.org Thu Jul 17 17:41:33 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 17 Jul 2014 16:41:33 -0600 Subject: [petsc-users] Symmetric non-positive definite system solved with CG+ML - why does it not fail? In-Reply-To: <20140717222839.GC6840@karman.lanl.gov> References: <20140717222839.GC6840@karman.lanl.gov> Message-ID: <87lhrrsj4y.fsf@jedbrown.org> Jozsef Bakosi writes: > Hi folks, > > I have a matrix as a result of a finite element discretization of the Poisson > operator and associated right hand side. As it turns out, the matrix is > symmetric but not positive definite since it has at least two negative small > eigenvalues. I have been solving this system without problem using the conjugate > gradients (CG) algorithm with ML as a preconditioner, but I'm wondering why it > works. Is the preconditioned matrix P^{-1/2} A P^{-T/2} positive definite? > Shouldn't CG fail for a non-positive-definite matrix? Does PETSc do some > additional magic if it detects that the dot-product in the CG algorithm is > negative? Does it solve the system using the normal equations, A'A, instead? CG will report divergence in case a direction of negative curvature is found. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jbakosi at lanl.gov Thu Jul 17 17:50:07 2014 From: jbakosi at lanl.gov (Jozsef Bakosi) Date: Thu, 17 Jul 2014 16:50:07 -0600 Subject: [petsc-users] Symmetric non-positive definite system solved with CG+ML - why does it not fail? In-Reply-To: <87lhrrsj4y.fsf@jedbrown.org> References: <20140717222839.GC6840@karman.lanl.gov> <87lhrrsj4y.fsf@jedbrown.org> Message-ID: <20140717225007.GE6840@karman.lanl.gov> On 07.17.2014 16:41, Jed Brown wrote: > Jozsef Bakosi writes: > > > Hi folks, > > > > I have a matrix as a result of a finite element discretization of the Poisson > > operator and associated right hand side. As it turns out, the matrix is > > symmetric but not positive definite since it has at least two negative small > > eigenvalues. I have been solving this system without problem using the conjugate > > gradients (CG) algorithm with ML as a preconditioner, but I'm wondering why it > > works. > > Is the preconditioned matrix > > P^{-1/2} A P^{-T/2} > > positive definite? How would you extract P? From jed at jedbrown.org Thu Jul 17 18:06:18 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 17 Jul 2014 17:06:18 -0600 Subject: [petsc-users] Symmetric non-positive definite system solved with CG+ML - why does it not fail? In-Reply-To: <20140717225007.GE6840@karman.lanl.gov> References: <20140717222839.GC6840@karman.lanl.gov> <87lhrrsj4y.fsf@jedbrown.org> <20140717225007.GE6840@karman.lanl.gov> Message-ID: <87ha2fshzp.fsf@jedbrown.org> Jozsef Bakosi writes: > On 07.17.2014 16:41, Jed Brown wrote: >> Jozsef Bakosi writes: >> >> > Hi folks, >> > >> > I have a matrix as a result of a finite element discretization of the Poisson >> > operator and associated right hand side. As it turns out, the matrix is >> > symmetric but not positive definite since it has at least two negative small >> > eigenvalues. I have been solving this system without problem using the conjugate >> > gradients (CG) algorithm with ML as a preconditioner, but I'm wondering why it >> > works. >> >> Is the preconditioned matrix >> >> P^{-1/2} A P^{-T/2} >> >> positive definite? > > How would you extract P? You don't, you apply P^{-1}. Have you checked whether your solution is accurate? If you want to study the operator further, you can consider the generalized eigenvalue problem A x = \lambda P x Note that eigensolvers can work on this system while only being able to apply A and P^{-1}. You can find any negative eigenvalues of that problem, get the eigenvectors, and see how it compares to the unpreconditioned case. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From wumeng07maths at qq.com Fri Jul 18 04:19:44 2014 From: wumeng07maths at qq.com (=?ISO-8859-1?B?T28gICAgICA=?=) Date: Fri, 18 Jul 2014 17:19:44 +0800 Subject: [petsc-users] MatViewAndMatlabLoad Message-ID: Dear All, I met a problem about using "load()" function in Matlab to read a m file which describes a PETSc Mat object by : //Save coeff_matrix as a matlab file PetscViewer viewer; PetscViewerASCIIOpen(PETSC_COMM_WORLD,"CoeffMatrixFileName.m",&viewer); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); MatView(coeff_matrix, viewer); PetscViewerDestroy(&viewer); In Matlab, I used:M=load("CoeffMatrixFileName.m") and I got an error.The error is:???Error using ===> loadNumber of columns on line 2 of ASCII file CoeffMatrixFileName.m must be the same as previous lines. Do you know how to deal with that error?Or Do you think there is some a mistake in my output of PETSc Mat Obj with the PetscViewer? Thank you~:) Meng -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Fri Jul 18 05:55:53 2014 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Fri, 18 Jul 2014 12:55:53 +0200 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7bdfbdcfb61540aeafdb3447e16e06e4@MAR190n2.marin.local> References: <7bdfbdcfb61540aeafdb3447e16e06e4@MAR190n2.marin.local> Message-ID: <53C8FD39.2020709@iue.tuwien.ac.at> Hi Christiaan, thanks, I updated the documentation here: https://bitbucket.org/petsc/petsc/commits/35a2f1ba208ddbb74d657014264e2a9c06170dbe Best regards, Karli On 07/17/2014 08:41 AM, Klaij, Christiaan wrote: > Just for future reference, below are the correct comments that > should be at the top of ex70. > > (Jed, Matt, I sent these to the maintainer list a long time > ago ([petsc-maint #155997]), but you never put them in, it would > avoid a lot of confusion) > > /* Poiseuille flow problem. */ > /* */ > /* Viscous, laminar flow in a 2D channel with parabolic velocity */ > /* profile and linear pressure drop, exact solution of the 2D Stokes */ > /* equations. */ > /* */ > /* Discretized with the cell-centered finite-volume method on a */ > /* Cartesian grid with co-located variables. Variables ordered as */ > /* [u1...uN v1...vN p1...pN]^T. Matrix [A00 A01; A10, A11] solved with */ > /* PCFIELDSPLIT. Lower factorization is used to mimick the Semi-Implicit */ > /* Method for Pressure Linked Equations (SIMPLE) used as preconditioner */ > /* instead of solver. */ > /* */ > /* Disclaimer: does not contain the pressure-weighed interpolation */ > /* method needed to suppress spurious pressure modes in real-life */ > /* problems. */ > /* */ > /* usage: */ > /* */ > /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -fieldsplit_1_pc_type none > /* */ > /* Runs with PCFIELDSPLIT on 32x48 grid, no PC for the Schur */ > /* complement because A11 is zero. FGMRES is needed because */ > /* PCFIELDSPLIT is a variable preconditioner. */ > /* */ > /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -user_pc > /* */ > /* Same as above but with user defined PC for the true Schur */ > /* complement. PC based on the SIMPLE-type approximation (inverse of */ > /* A00 approximated by inverse of its diagonal). */ > /* */ > /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -user_ksp > /* */ > /* Replace the true Schur complement with a user defined Schur */ > /* complement based on the SIMPLE-type approximation. Same matrix is */ > /* used as PC. */ > /* */ > /* mpiexec -n 2 ./ex70 -nx 32 -ny 48 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type lower -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi -fieldsplit_1_pc_type jacobi -fieldsplit_1_inner_ksp_type preonly -fieldsplit_1_inner_pc_type jacobi -fieldsplit_1_upper_ksp_type preonly -fieldsplit_1_upper_pc_type jacobi > /* */ > /* Out-of-the-box SIMPLE-type preconditioning. The major advantage */ > /* is that the user neither needs to provide the approximation of */ > /* the Schur complement, nor the corresponding preconditioner. > > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > From natacha.bereux at gmail.com Fri Jul 18 07:27:05 2014 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Fri, 18 Jul 2014 14:27:05 +0200 Subject: [petsc-users] MatSetNearNullSpace and BoomerAMG ? Message-ID: Dear all, I am testing three multigrid preconditionners: BoomerAMG, ML and GAMG on a 3D elasticity problem (a cube with prescribed displacement on upper and lower faces). I set the null space by calling MatNullSpaceCreateRigidBody and then MatSetNearNullSpace). I see a difference (about half iterations) for ML and GAMG but nothing happens with BoomerAMG (same number of iterations with and without setting the NearNullspace). Does it make sense ? Is this setting only relevant for Smoothed Aggregation AMG ? and of no use for BoomeAMG? Or do I miss something in the setting ? Thanks a lot, Best regards Natacha -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 18 08:04:45 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 18 Jul 2014 07:04:45 -0600 Subject: [petsc-users] MatViewAndMatlabLoad In-Reply-To: References: Message-ID: <87vbqurf6a.fsf@jedbrown.org> Oo writes: > Dear All, > > > I met a problem about using "load()" function in Matlab to read a m file which describes a PETSc Mat object by : > //Save coeff_matrix as a matlab file PetscViewer viewer; PetscViewerASCIIOpen(PETSC_COMM_WORLD,"CoeffMatrixFileName.m",&viewer); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); MatView(coeff_matrix, viewer); PetscViewerDestroy(&viewer); > In Matlab, I used:M=load("CoeffMatrixFileName.m") and I got an error.The error is:???Error using ===> loadNumber of columns on line 2 of ASCII file CoeffMatrixFileName.m must be the same as previous lines. What does the file look like? Note that we recommend using a binary viewer and A = PetscBinaryRead('binaryoutput') to read in MATLAB. It is more accurate and much faster. > Do you know how to deal with that error?Or Do you think there is some a mistake in my output of PETSc Mat Obj with the PetscViewer? > Thank you~:) > Meng -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Fri Jul 18 08:07:43 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 18 Jul 2014 07:07:43 -0600 Subject: [petsc-users] MatSetNearNullSpace and BoomerAMG ? In-Reply-To: References: Message-ID: <87silyrf1c.fsf@jedbrown.org> Natacha BEREUX writes: > Dear all, > > I am testing three multigrid preconditionners: BoomerAMG, ML and GAMG on a > 3D elasticity problem (a cube with prescribed displacement on upper and > lower faces). > I set the null space by calling MatNullSpaceCreateRigidBody and then > MatSetNearNullSpace). > > I see a difference (about half iterations) for ML and GAMG but nothing > happens with BoomerAMG (same number of iterations with and without setting > the NearNullspace). That's expected because BoomerAMG does not use any such information (though the algorithm might be extended in the future so that those vectors become seed vectors for Bootstrap AMG, for example; I don't know if the BoomerAMG developers have any interest in implementing this). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From natacha.bereux at gmail.com Fri Jul 18 08:25:23 2014 From: natacha.bereux at gmail.com (Natacha BEREUX) Date: Fri, 18 Jul 2014 15:25:23 +0200 Subject: [petsc-users] MatSetNearNullSpace and BoomerAMG ? In-Reply-To: <87silyrf1c.fsf@jedbrown.org> References: <87silyrf1c.fsf@jedbrown.org> Message-ID: OK, thanks a lot Jed for your quick answer. I have a second question on multigrid : I define a matrix block size (equal to 3). Is this mandatory for any type of multigrid preconditioner (BoomerAMG, ML and GAMG) ? Or is it only required with ML and GAMG ? Best regards Natacha On Fri, Jul 18, 2014 at 3:07 PM, Jed Brown wrote: > Natacha BEREUX writes: > > > Dear all, > > > > I am testing three multigrid preconditionners: BoomerAMG, ML and GAMG on > a > > 3D elasticity problem (a cube with prescribed displacement on upper and > > lower faces). > > I set the null space by calling MatNullSpaceCreateRigidBody and then > > MatSetNearNullSpace). > > > > I see a difference (about half iterations) for ML and GAMG but nothing > > happens with BoomerAMG (same number of iterations with and without > setting > > the NearNullspace). > > That's expected because BoomerAMG does not use any such information > (though the algorithm might be extended in the future so that those > vectors become seed vectors for Bootstrap AMG, for example; I don't know > if the BoomerAMG developers have any interest in implementing this). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 18 08:37:06 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 18 Jul 2014 07:37:06 -0600 Subject: [petsc-users] MatSetNearNullSpace and BoomerAMG ? In-Reply-To: References: <87silyrf1c.fsf@jedbrown.org> Message-ID: <87pph2rdod.fsf@jedbrown.org> Natacha BEREUX writes: > OK, thanks a lot Jed for your quick answer. > > I have a second question on multigrid : > I define a matrix block size (equal to 3). Is this mandatory for any type > of multigrid preconditioner (BoomerAMG, ML and GAMG) ? > Or is it only required with ML and GAMG ? It is always preferable. BoomerAMG distinguishes those components in its coarsening, it just doesn't use the near null space explicitly. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hus003 at ucsd.edu Fri Jul 18 11:56:34 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 18 Jul 2014 16:56:34 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <877g3cwomy.fsf@jedbrown.org> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU>, <877g3cwomy.fsf@jedbrown.org> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC0B5@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Jed. After reading ex15, I got some idea of how to define the PCShell with KSP solver. However, I want to verify with you if the following procedure is normally what one should do. 1. Define a ShellPC struct, which includes a MAT object, say Pmat, and a KSP object, say innerKsp. 2. PCSetType(pc, PCSHELL) 3. Create a context for the user defined PC. It is here that we call KSPSetOperators(innerKsp, Pmat, Pmat) 4. Set the user defined routine for applying the preconditioner. It is in this user-define routine that we call KSPSolver(innerKsp, r, y) I'm wondering the way I set up the inner KSP for the PC is standard? Thank you! ________________________________________ From: Jed Brown [jed at jedbrown.org] Sent: Wednesday, July 16, 2014 10:16 PM To: Sun, Hui; Dave May Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] why a certain option cannot be used "Sun, Hui" writes: > Dave, and other PETSc users, sorry for bothering you all again. However continuing this topic, I have another question. Within this ex70.c, there is a user defined schur complement StokesSetupApproxSchur, defined as > > > /* Schur complement approximation: myS = A11 - A10 diag(A00)^(-1) A01 */ > > The preconditioner here can be written out explicitly as a matrix, and > is not changed between iterations. Each time it is applied to the > residue is simply matrix times vector. However, if I want to define a > preconditioner which is allowed to vary between iterations, for > example, if I need to solve a linear system each time I apply the > preconditioner to the residue, then is there a way I can do this? Any > hint will be greatly appreciated. Thank you very much! You either write your own PCShell or compose a PC from existing components (perhaps via options). In any case, the application of this PC will involve calls to KSPSolve. You can configure those to be iterative if that makes sense for your use case. Note that if you run an inexact inner solve, the outer solves should use a flexible method (e.g., FGMRES or GCR). From jed at jedbrown.org Fri Jul 18 12:23:07 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 18 Jul 2014 11:23:07 -0600 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC0B5@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> <877g3cwomy.fsf@jedbrown.o rg> <750 1CC2B7BBCC44A92ECEEC316170ECB6CC0B5@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <877g3ar37n.fsf@jedbrown.org> "Sun, Hui" writes: > Thank you Jed. After reading ex15, I got some idea of how to define the PCShell with KSP solver. However, I want to verify with you if the following procedure is normally what one should do. > 1. Define a ShellPC struct, which includes a MAT object, say Pmat, and a KSP object, say innerKsp. The context should indeed contain innerKSP. The ShellPC may not need a Mat (you can just pass it down to innerKSP). > 2. PCSetType(pc, PCSHELL) > 3. Create a context for the user defined PC. It is here that we call KSPSetOperators(innerKsp, Pmat, Pmat) You can do it eagerly or via PCShellSetSetUp(), if you want the shell PC to be more reusable/independent. > 4. Set the user defined routine for applying the preconditioner. It is in this user-define routine that we call KSPSolver(innerKsp, r, y) > > I'm wondering the way I set up the inner KSP for the PC is standard? Thank you! Pretty much. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From chetan.jhurani at gmail.com Fri Jul 18 13:19:27 2014 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Fri, 18 Jul 2014 11:19:27 -0700 Subject: [petsc-users] Symmetric non-positive definite system In-Reply-To: References: Message-ID: <53c96519.4467460a.3be3.1075@mx.google.com> > From: Jed Brown > > Jozsef Bakosi writes: > > > Hi folks, > > > > I have a matrix as a result of a finite element discretization of the Poisson > > operator and associated right hand side. As it turns out, the matrix is > > symmetric but not positive definite since it has at least two negative small > > eigenvalues. I have been solving this system without problem using the conjugate > > gradients (CG) algorithm with ML as a preconditioner, but I'm wondering why it > > works. > > Is the preconditioned matrix > > P^{-1/2} A P^{-T/2} > > positive definite? This does not answer the original question regarding CG/ML, but will reduce some debugging work. P^{-1/2} A P^{-T/2} cannot be positive definite since A is not positive definite. This is due to Sylvester's inertia theorem. http://books.google.com/books?id=P3bPAgAAQBAJ&pg=PA202 Applied Numerical Linear Algebra By James W. Demmel, p 202 Chetan > > Shouldn't CG fail for a non-positive-definite matrix? Does PETSc do some > > additional magic if it detects that the dot-product in the CG algorithm is > > negative? Does it solve the system using the normal equations, A'A, instead? > > CG will report divergence in case a direction of negative curvature is > found. From shchen at www.phys.lsu.edu Fri Jul 18 13:18:22 2014 From: shchen at www.phys.lsu.edu (Shaohao Chen) Date: Fri, 18 Jul 2014 13:18:22 -0500 Subject: [petsc-users] sign off Message-ID: <20140718181716.M2345@physics.lsu.edu> Dear manager, Could you please sign me off this email list. Thank you! -- Shaohao Chen Department of Physics & Astronomy, Louisiana State University, Baton Rouge, LA From knepley at gmail.com Fri Jul 18 13:22:30 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 18 Jul 2014 13:22:30 -0500 Subject: [petsc-users] Symmetric non-positive definite system In-Reply-To: <53c96519.4467460a.3be3.1075@mx.google.com> References: <53c96519.4467460a.3be3.1075@mx.google.com> Message-ID: On Fri, Jul 18, 2014 at 1:19 PM, Chetan Jhurani wrote: > > From: Jed Brown > > > > Jozsef Bakosi writes: > > > > > Hi folks, > > > > > > I have a matrix as a result of a finite element discretization of the > Poisson > > > operator and associated right hand side. As it turns out, the matrix is > > > symmetric but not positive definite since it has at least two negative > small > > > eigenvalues. I have been solving this system without problem using the > conjugate > > > gradients (CG) algorithm with ML as a preconditioner, but I'm > wondering why it > > > works. > > > > Is the preconditioned matrix > > > > P^{-1/2} A P^{-T/2} > > > > positive definite? > > This does not answer the original question regarding CG/ML, but > It is possible for CG to convergence with an indefinite matrix, but it is also possible for it to fail. Matt > will reduce some debugging work. P^{-1/2} A P^{-T/2} cannot be > positive definite since A is not positive definite. This is due > to Sylvester's inertia theorem. > > http://books.google.com/books?id=P3bPAgAAQBAJ&pg=PA202 > Applied Numerical Linear Algebra By James W. Demmel, p 202 > > Chetan > > > > Shouldn't CG fail for a non-positive-definite matrix? Does PETSc do > some > > > additional magic if it detects that the dot-product in the CG > algorithm is > > > negative? Does it solve the system using the normal equations, A'A, > instead? > > > > CG will report divergence in case a direction of negative curvature is > > found. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpau at lbl.gov Fri Jul 18 17:50:28 2014 From: gpau at lbl.gov (George Pau) Date: Fri, 18 Jul 2014 15:50:28 -0700 Subject: [petsc-users] Unable to download elemental Message-ID: I tried the following: ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-elemental --with-cxx-dialect=C++11 but get a configure error that says: Unable to download elemental I tried downloaded the package manually and specifying the path -download-elemental=/home/gpau/Downloads/Elemental-0.84-p5.tgz but still get the same error. Attached is the log file for the configure. Thanks, George -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1541205 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Jul 18 18:06:42 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 18 Jul 2014 18:06:42 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: References: Message-ID: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> Sorry, there is an error in our process: Looking for ELEMENTAL in directory starting with elemental Could not locate an existing copy of elemental: ['Elemental-0.84-p5?] It is looking for a directory that starts with a e while the directory has a capital e. To fix change the name of the directory $PETSC_ARCH/externalpackages/Elemental-0.84-p5 to $PETSC_ARCH/externalpackages/elemental-0.84-p5 Then rerun with the same options as below (PETSc will use the directory you?ve renamed automatically). Satish, Please check why it is using the elemental instead of the Elemental. Why doesn?t it hit our tests? Barry On Jul 18, 2014, at 5:50 PM, George Pau wrote: > I tried the following: > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-elemental --with-cxx-dialect=C++11 > > but get a configure error that says: > Unable to download elemental > > I tried downloaded the package manually and specifying the path -download-elemental=/home/gpau/Downloads/Elemental-0.84-p5.tgz > > but still get the same error. Attached is the log file for the configure. > > Thanks, > George > > -- > George Pau > Earth Sciences Division > Lawrence Berkeley National Laboratory > One Cyclotron, MS 74-120 > Berkeley, CA 94720 > > (510) 486-7196 > gpau at lbl.gov > http://esd.lbl.gov/about/staff/georgepau/ > From dmeiser at txcorp.com Fri Jul 18 18:33:43 2014 From: dmeiser at txcorp.com (Dominic Meiser) Date: Fri, 18 Jul 2014 17:33:43 -0600 Subject: [petsc-users] Viewing DM Vector stored on Multiple GPUs In-Reply-To: References: Message-ID: <53C9AED7.7020103@txcorp.com> Hi Ashwin, I've been working on this issue the last couple of days and wanted to give you an update. First off, thanks for the test case. This has helped a lot in tracking down the issues. I've created a couple of unit tests based on your test case. There were several independent bugs that prevented the VecView to work with -dm_vec_type cusp. I've fixed some of them in this branch: https://bitbucket.org/dmeiser/petsc/branch/fix-cusp-dmda Viewing a vector created from a DM should now be possible. For instance the test case you provided should work with this branch. There are still some issues remaining. DMGlobalToLocalBegin/End and DMLocalToGlobalBegin/End are not working. I haven't been able to track these down yet. I'll return to this next week. Cheers, Dominic On 06/24/2014 08:15 PM, Ashwin Srinath wrote: > Hello, petsc-users > > I'm having trouble /viewing/ an mpicusp vector. Here's the simplest > case that reproduces the problem: > > int main(int argc, char** argv) { > > PetscInitialize(&argc, &argv, NULL, NULL); > > DM da; > Vec V; > > DMDACreate2d( PETSC_COMM_WORLD, > DM_BOUNDARY_NONE, DM_BOUNDARY_NONE, > DMDA_STENCIL_BOX, > 5, 5, > PETSC_DECIDE, PETSC_DECIDE, > 1, > 1, > NULL, NULL, > &da); > > DMCreateGlobalVector(da, &V); > > VecSet(V, 1); > VecView(V, PETSC_VIEWER_STDOUT_WORLD); > > PetscFinalize(); > return 0; > } > > I get the error: > [1]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Trying to copy from a null pointer > > I executed with the following command: > mpiexec -n 2 ./main -dm_vec_type cusp -vec_type cusp > Both GPUs are attached to two different processes. > > This program works fine for vecmpi vectors, i.e., -dm_vec_type mpi and > -vec_type mpi. Also, I don't get an error unless I try to /view/ the > vector. Can someone please point out what I'm doing wrong? > > Thanks for your time, > Ashwin Srinath > > > > > -- Dominic Meiser Tech-X Corporation 5621 Arapahoe Avenue Boulder, CO 80303 USA Telephone: 303-996-2036 Fax: 303-448-7756 www.txcorp.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashwinsrnth at gmail.com Fri Jul 18 18:29:45 2014 From: ashwinsrnth at gmail.com (Ashwin Srinath) Date: Fri, 18 Jul 2014 19:29:45 -0400 Subject: [petsc-users] Viewing DM Vector stored on Multiple GPUs In-Reply-To: <53C9AED7.7020103@txcorp.com> References: <53C9AED7.7020103@txcorp.com> Message-ID: Thanks so much, Dominic. I'll let you know soon how this branch works for my code. Ashwin On Fri, Jul 18, 2014 at 7:33 PM, Dominic Meiser wrote: > Hi Ashwin, > > I've been working on this issue the last couple of days and wanted to give > you an update. First off, thanks for the test case. This has helped a lot > in tracking down the issues. I've created a couple of unit tests based on > your test case. > > There were several independent bugs that prevented the VecView to work > with -dm_vec_type cusp. I've fixed some of them in this branch: > > https://bitbucket.org/dmeiser/petsc/branch/fix-cusp-dmda > > Viewing a vector created from a DM should now be possible. For instance > the test case you provided should work with this branch. > > There are still some issues remaining. DMGlobalToLocalBegin/End and > DMLocalToGlobalBegin/End are not working. I haven't been able to track > these down yet. I'll return to this next week. > > Cheers, > Dominic > > > > On 06/24/2014 08:15 PM, Ashwin Srinath wrote: > > Hello, petsc-users > > I'm having trouble *viewing* an mpicusp vector. Here's the simplest case > that reproduces the problem: > > int main(int argc, char** argv) { > > PetscInitialize(&argc, &argv, NULL, NULL); > > DM da; > Vec V; > > DMDACreate2d( PETSC_COMM_WORLD, > DM_BOUNDARY_NONE, DM_BOUNDARY_NONE, > DMDA_STENCIL_BOX, > 5, 5, > PETSC_DECIDE, PETSC_DECIDE, > 1, > 1, > NULL, NULL, > &da); > > DMCreateGlobalVector(da, &V); > > VecSet(V, 1); > VecView(V, PETSC_VIEWER_STDOUT_WORLD); > > PetscFinalize(); > return 0; > } > > I get the error: > [1]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Trying to copy from a null pointer > > I executed with the following command: > mpiexec -n 2 ./main -dm_vec_type cusp -vec_type cusp > Both GPUs are attached to two different processes. > > This program works fine for vecmpi vectors, i.e., -dm_vec_type mpi and > -vec_type mpi. Also, I don't get an error unless I try to *view* the > vector. Can someone please point out what I'm doing wrong? > > Thanks for your time, > Ashwin Srinath > > > > > > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756www.txcorp.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpau at lbl.gov Fri Jul 18 18:38:04 2014 From: gpau at lbl.gov (George Pau) Date: Fri, 18 Jul 2014 16:38:04 -0700 Subject: [petsc-users] Unable to download elemental In-Reply-To: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> Message-ID: Hi Barry, Thanks, that works on my Ubuntu machine. On Mac however, I have problem with the --with-cxx-dialect=C++11: Could not determine compiler flag for with-cxx-dialect=C++11, use CXXFLAGS So, I configure using the following: ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpiexec=mpiexec --CXXFLAGS=-std=c++11 --with-cxx-dialect=C++11 --download-elemental but, I still get Could not determine compiler flag for with-cxx-dialect=C++11, use CXXFLAGS I tried with just ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpiexec=mpiexec --CXXFLAGS=-std=c++11 and it is fine. I cannot specify --download-elemental without --with-cxx-dialect=C++11. Attached is the configure.log again. Thanks, George On Fri, Jul 18, 2014 at 4:06 PM, Barry Smith wrote: > > Sorry, there is an error in our process: > > Looking for ELEMENTAL in directory starting with > elemental > Could not locate an existing copy of elemental: > ['Elemental-0.84-p5?] > > It is looking for a directory that starts with a e while the directory has > a capital e. > > To fix change the name of the directory > $PETSC_ARCH/externalpackages/Elemental-0.84-p5 to > $PETSC_ARCH/externalpackages/elemental-0.84-p5 > > Then rerun with the same options as below (PETSc will use the directory > you?ve renamed automatically). > > Satish, > > Please check why it is using the elemental instead of the Elemental. > Why doesn?t it hit our tests? > > Barry > > On Jul 18, 2014, at 5:50 PM, George Pau wrote: > > > I tried the following: > > > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --download-elemental --with-cxx-dialect=C++11 > > > > but get a configure error that says: > > Unable to download elemental > > > > I tried downloaded the package manually and specifying the path > -download-elemental=/home/gpau/Downloads/Elemental-0.84-p5.tgz > > > > but still get the same error. Attached is the log file for the configure. > > > > Thanks, > > George > > > > -- > > George Pau > > Earth Sciences Division > > Lawrence Berkeley National Laboratory > > One Cyclotron, MS 74-120 > > Berkeley, CA 94720 > > > > (510) 486-7196 > > gpau at lbl.gov > > http://esd.lbl.gov/about/staff/georgepau/ > > > > -- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720 (510) 486-7196 gpau at lbl.gov http://esd.lbl.gov/about/staff/georgepau/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 211139 bytes Desc: not available URL: From jed at jedbrown.org Fri Jul 18 19:02:49 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 18 Jul 2014 18:02:49 -0600 Subject: [petsc-users] Unable to download elemental In-Reply-To: References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> Message-ID: <87a986p652.fsf@jedbrown.org> George Pau writes: > Hi Barry, > > Thanks, that works on my Ubuntu machine. > > On Mac however, I have problem with the --with-cxx-dialect=C++11: > > Could not determine compiler flag for with-cxx-dialect=C++11, use CXXFLAGS > > So, I configure using the following: > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-mpiexec=mpiexec --CXXFLAGS=-std=c++11 --with-cxx-dialect=C++11 > --download-elemental > > but, I still get > > Could not determine compiler flag for with-cxx-dialect=C++11, use CXXFLAGS > > I tried with just > > ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-mpiexec=mpiexec --CXXFLAGS=-std=c++11 > > and it is fine. I cannot specify --download-elemental without > --with-cxx-dialect=C++11. Attached is the configure.log again. In your first email Executing: mpicxx --version stdout: g++ (Ubuntu 4.8.2-19ubuntu1) 4.8.2 Copyright (C) 2013 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. getCompilerVersion: mpicxx g++ (Ubuntu 4.8.2-19ubuntu1) 4.8.2 Executing: mpicxx -show stdout: g++ -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi_cxx -lmpi -ldl -lhwloc but in your most recent, Executing: mpicxx --version stdout: Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) Target: x86_64-apple-darwin11.4.2 Thread model: posix getCompilerVersion: mpicxx Apple clang version 3.1 (tags/Apple/clang-318.0.61) (based on LLVM 3.1svn) Ignoring default options which were overridden using --CXXFLAGS -std=c++11 Executing: mpicxx -show stdout: /usr/bin/clang++ -I/opt/local/include/openmpi-mp -Wl,-flat_namespace -L/opt/local/lib/openmpi-mp -lmpi_cxx -lmpi The first supports -std=c++11, but the second does not. Either use GCC or upgrade Clang. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hong at aspiritech.org Fri Jul 18 20:10:50 2014 From: hong at aspiritech.org (hong at aspiritech.org) Date: Fri, 18 Jul 2014 20:10:50 -0500 Subject: [petsc-users] Symmetric non-positive definite system In-Reply-To: References: <53c96519.4467460a.3be3.1075@mx.google.com> Message-ID: You may try MINRES (-ksp_type minres) See http://web.stanford.edu/group/SOL/reports/SOL-2011-2R.pdf Hong On Fri, Jul 18, 2014 at 1:22 PM, Matthew Knepley wrote: > On Fri, Jul 18, 2014 at 1:19 PM, Chetan Jhurani > wrote: >> >> > From: Jed Brown >> > >> > Jozsef Bakosi writes: >> > >> > > Hi folks, >> > > >> > > I have a matrix as a result of a finite element discretization of the >> > > Poisson >> > > operator and associated right hand side. As it turns out, the matrix >> > > is >> > > symmetric but not positive definite since it has at least two negative >> > > small >> > > eigenvalues. I have been solving this system without problem using the >> > > conjugate >> > > gradients (CG) algorithm with ML as a preconditioner, but I'm >> > > wondering why it >> > > works. >> > >> > Is the preconditioned matrix >> > >> > P^{-1/2} A P^{-T/2} >> > >> > positive definite? >> >> This does not answer the original question regarding CG/ML, but > > > It is possible for CG to convergence with an indefinite matrix, but it is > also > possible for it to fail. > > Matt > >> >> will reduce some debugging work. P^{-1/2} A P^{-T/2} cannot be >> positive definite since A is not positive definite. This is due >> to Sylvester's inertia theorem. >> >> http://books.google.com/books?id=P3bPAgAAQBAJ&pg=PA202 >> Applied Numerical Linear Algebra By James W. Demmel, p 202 >> >> Chetan >> >> > > Shouldn't CG fail for a non-positive-definite matrix? Does PETSc do >> > > some >> > > additional magic if it detects that the dot-product in the CG >> > > algorithm is >> > > negative? Does it solve the system using the normal equations, A'A, >> > > instead? >> > >> > CG will report divergence in case a direction of negative curvature is >> > found. >> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From jyawney123 at gmail.com Fri Jul 18 20:16:54 2014 From: jyawney123 at gmail.com (John Yawney) Date: Fri, 18 Jul 2014 21:16:54 -0400 Subject: [petsc-users] Question about PETSc installs and MPI Message-ID: Hello, I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: ./configure --download-f-blas-lapack --with-scalar-type=complex or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 Could this be causing a problem with the parallelization under PETSc? Thanks for the help and sorry for the long question. Best regards, John -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: main.cpp Type: text/x-c++src Size: 7391 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Jul 18 20:57:55 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 18 Jul 2014 20:57:55 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: Message-ID: <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Barry On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > Hello, > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > ./configure --download-f-blas-lapack --with-scalar-type=complex > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > Could this be causing a problem with the parallelization under PETSc? > > Thanks for the help and sorry for the long question. > > Best regards, > John > From knepley at gmail.com Fri Jul 18 20:58:41 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 18 Jul 2014 20:58:41 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: Message-ID: On Fri, Jul 18, 2014 at 8:16 PM, John Yawney wrote: > Hello, > > I had a question about PETSc installations. On my local computer I > configured PETSc (v 3.4.2) using the options: > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack > --download-mpich --download-hypre > > I wrote a test program that defines a vector using DMDAs, computes a dot > product, exchanges halo elements, and computes a low-order FD derivative of > the vector. Under my installation of PETSc everything works fine. For > some reason, when my colleagues run the program, they get segmentation > fault errors. If they change the y and z boundary types to GHOSTED as well, > they get the program to run until the end (seg faults at the end though) > but they get a local value of the dot product. I've attached the main.cpp > file for this script. > > When they installed their versions of PETSc they didn't use the > --download-mpich option but instead used either: > ./configure --download-f-blas-lapack --with-scalar-type=complex > or with the > option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > Could this be causing a problem with the parallelization under PETSc? > I have run your code on my machine up to P=8 and used valgrind. No problems turned up other than the fact that you need a "return 0" is missing from main(). If there are still problems, please have them send a stack trace. Thanks, Matt > Thanks for the help and sorry for the long question. > > Best regards, > John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jul 18 21:21:30 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 18 Jul 2014 21:21:30 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: Message-ID: On Fri, 18 Jul 2014, John Yawney wrote: > Hello, > > I had a question about PETSc installations. On my local computer I > configured PETSc (v 3.4.2) using the options: > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack > --download-mpich --download-hypre BTW: the above doesn't make sense. Its suprising that this install is working. You should either use the prebuild mpi you have: ./configure --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpif90 --download-f-blas-lapack --download-hypre or build mpich with the native compilers: ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-f-blas-lapack --download-hypre Satish From balay at mcs.anl.gov Fri Jul 18 22:10:36 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 18 Jul 2014 22:10:36 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> Message-ID: On Fri, 18 Jul 2014, Barry Smith wrote: > > Satish, > > Please check why it is using the elemental instead of the Elemental. Why doesn?t it hit our tests? Hm - I should convert one of the tests to using --with-git=0 for this case.. The fix for this issue would be: git mv elemental.py Elemental.py [or add to elemental.py: self.downloadfilename = 'Elemental'] Should we do a case insensitive search here? And then use lowercase for all package files [Superlu_DIST.py -> superlu_dist.py etc..] Satish -------- diff --git a/config/BuildSystem/config/package.py b/config/BuildSystem/config/package.py index 281450f..e9f1f5d 100644 --- a/config/BuildSystem/config/package.py +++ b/config/BuildSystem/config/package.py @@ -402,7 +402,7 @@ class Package(config.base.Configure): def matchExcludeDir(self,dir): '''Check is the dir matches something in the excluded directory list''' for exdir in self.excludedDirs: - if dir.startswith(exdir): + if dir.lower().startswith(exdir.lower()): return 1 return 0 @@ -415,7 +415,7 @@ class Package(config.base.Configure): Dir = None self.framework.logPrint('Looking for '+self.PACKAGE+' in directory starting with '+str(self.downloadfilename)) for d in os.listdir(packages): - if d.startswith(self.downloadfilename) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): + if d.lower().startswith(self.downloadfilename.lower()) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): self.framework.logPrint('Found a copy of '+self.PACKAGE+' in '+str(d)) Dir = d break From rupp at iue.tuwien.ac.at Sat Jul 19 03:17:16 2014 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Sat, 19 Jul 2014 10:17:16 +0200 Subject: [petsc-users] sign off In-Reply-To: <20140718181716.M2345@physics.lsu.edu> References: <20140718181716.M2345@physics.lsu.edu> Message-ID: <53CA298C.9090908@iue.tuwien.ac.at> Hi, > > Could you please sign me off this email list. Thank you! please do so here: http://lists.mcs.anl.gov/mailman/listinfo/petsc-users Best regards, Karli From fpoulin at uwaterloo.ca Sat Jul 19 06:10:10 2014 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Sat, 19 Jul 2014 11:10:10 +0000 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> Message-ID: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> Hello Barry, I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. Thank, Francis ------------------ Francis Poulin Associate Professor Associate Chair, Undergraduate Studies Department of Applied Mathematics University of Waterloo email: fpoulin at uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, July 18, 2014 9:57 PM To: John Yawney Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi Subject: Re: [petsc-users] Question about PETSc installs and MPI I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Barry On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > Hello, > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > ./configure --download-f-blas-lapack --with-scalar-type=complex > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > Could this be causing a problem with the parallelization under PETSc? > > Thanks for the help and sorry for the long question. > > Best regards, > John > -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2426901 bytes Desc: configure.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 86758 bytes Desc: make.log URL: From balay at mcs.anl.gov Sat Jul 19 10:26:55 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 19 Jul 2014 10:26:55 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> Message-ID: On Sat, 19 Jul 2014, Francis Poulin wrote: > Hello Barry, > > I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: > > ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre > > This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. PETSc does not use scalapack. Its useful only if you are using mumps.. --download-metis --download-parmetis --download-scalapack --download-mumps > > It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. > > fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 > -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory Perhaps you do not have csh installed on this machine. You can use mpiexec directly ./ex1 /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 Satish > > I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. > > Thank, > Francis > > > ------------------ > Francis Poulin > Associate Professor > Associate Chair, Undergraduate Studies > Department of Applied Mathematics > University of Waterloo > > email: fpoulin at uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, July 18, 2014 9:57 PM > To: John Yawney > Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. > > Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. > > The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Barry > > On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > > > Hello, > > > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > > ./configure --download-f-blas-lapack --with-scalar-type=complex > > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > > > Could this be causing a problem with the parallelization under PETSc? > > > > Thanks for the help and sorry for the long question. > > > > Best regards, > > John > > > > From bsmith at mcs.anl.gov Sat Jul 19 10:30:58 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 10:30:58 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> Message-ID: Satish, I am fine with you fixing it either way. Thanks Barry On Jul 18, 2014, at 10:10 PM, Satish Balay wrote: > On Fri, 18 Jul 2014, Barry Smith wrote: > >> >> Satish, >> >> Please check why it is using the elemental instead of the Elemental. Why doesn?t it hit our tests? > > Hm - I should convert one of the tests to using --with-git=0 for this case.. > > > The fix for this issue would be: > > git mv elemental.py Elemental.py > [or add to elemental.py: self.downloadfilename = 'Elemental'] > > Should we do a case insensitive search here? > And then use lowercase for all package files [Superlu_DIST.py -> superlu_dist.py etc..] > > > Satish > -------- > diff --git a/config/BuildSystem/config/package.py b/config/BuildSystem/config/package.py > index 281450f..e9f1f5d 100644 > --- a/config/BuildSystem/config/package.py > +++ b/config/BuildSystem/config/package.py > @@ -402,7 +402,7 @@ class Package(config.base.Configure): > def matchExcludeDir(self,dir): > '''Check is the dir matches something in the excluded directory list''' > for exdir in self.excludedDirs: > - if dir.startswith(exdir): > + if dir.lower().startswith(exdir.lower()): > return 1 > return 0 > > @@ -415,7 +415,7 @@ class Package(config.base.Configure): > Dir = None > self.framework.logPrint('Looking for '+self.PACKAGE+' in directory starting with '+str(self.downloadfilename)) > for d in os.listdir(packages): > - if d.startswith(self.downloadfilename) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): > + if d.lower().startswith(self.downloadfilename.lower()) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): > self.framework.logPrint('Found a copy of '+self.PACKAGE+' in '+str(d)) > Dir = d > break From jyawney123 at gmail.com Sat Jul 19 10:35:42 2014 From: jyawney123 at gmail.com (John Yawney) Date: Sat, 19 Jul 2014 11:35:42 -0400 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: Message-ID: Hi Barry, Matt, and Satish, Thank you for the quick responses. It's true I forgot the return 0 at the end. I remember I was looking at it before and meant to add the line in but I sent the original version of the code. I had a working version of the mpi compilers on my computer but I understand what you meant Satish. I'll definitely use those configure options for other installs. Thanks again, John On Jul 18, 2014 9:58 PM, "Matthew Knepley" wrote: > On Fri, Jul 18, 2014 at 8:16 PM, John Yawney wrote: > >> Hello, >> >> I had a question about PETSc installations. On my local computer I >> configured PETSc (v 3.4.2) using the options: >> >> ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack >> --download-mpich --download-hypre >> >> I wrote a test program that defines a vector using DMDAs, computes a dot >> product, exchanges halo elements, and computes a low-order FD derivative of >> the vector. Under my installation of PETSc everything works fine. For >> some reason, when my colleagues run the program, they get segmentation >> fault errors. If they change the y and z boundary types to GHOSTED as well, >> they get the program to run until the end (seg faults at the end though) >> but they get a local value of the dot product. I've attached the main.cpp >> file for this script. >> >> When they installed their versions of PETSc they didn't use the >> --download-mpich option but instead used either: >> ./configure --download-f-blas-lapack --with-scalar-type=complex >> or with the >> option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 >> >> Could this be causing a problem with the parallelization under PETSc? >> > > I have run your code on my machine up to P=8 and used valgrind. No > problems turned up other > than the fact that you need a "return 0" is missing from main(). > > If there are still problems, please have them send a stack trace. > > Thanks, > > Matt > > >> Thanks for the help and sorry for the long question. >> >> Best regards, >> John >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Jul 19 10:52:49 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 19 Jul 2014 10:52:49 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> Message-ID: And it appears the original code was tested with 'real' numbers and petsc-3.4. It was not really tested with complex numbers. I had to make the following changes for it to run with complex & petsc-3.5 Satish --------- balay at es^/scratch/balay/test $ ../petsc-3.5.0/arch-linux2-c-debug/bin/mpiexec -n 8 valgrind --tool=memcheck -q ./main Dot Product Check... Dot Product (should be 2*256) = 512.000000 Done. Derivative Check... Min (should be 0; if halos not defined properly, would be -1.0/dDeltaX) = 0.000000 Max (should be 0; if halos not defined properly, would be +1.0/dDeltaX) = 0.000000 Done. balay at es^/scratch/balay/test $ diff -Nru main.cpp.orig main.cpp --- main.cpp.orig 2014-07-19 10:48:21.072486076 -0500 +++ main.cpp 2014-07-19 10:47:26.977985006 -0500 @@ -31,9 +31,9 @@ // DMDA Environment ////////////////////////////////////////////////////////////////////////// // Boundary Conditions - DMDABoundaryType bx = DMDA_BOUNDARY_GHOSTED; - DMDABoundaryType by = DMDA_BOUNDARY_NONE; // no boundary points since nGx = 1 - DMDABoundaryType bz = DMDA_BOUNDARY_NONE; // no boundary points since nGz = 1 + DMBoundaryType bx = DM_BOUNDARY_GHOSTED; + DMBoundaryType by = DM_BOUNDARY_NONE; // no boundary points since nGx = 1 + DMBoundaryType bz = DM_BOUNDARY_NONE; // no boundary points since nGz = 1 ////////////////////////////////////////////////////////////////////////// // Stencil type @@ -96,7 +96,7 @@ DMDAGetCorners(m_3da, &si, 0, 0, &ei, 0, 0); // Access the elements of the local arrays as C++ multi-dim. array structures - double ***pvecA, ***pvecB; + PetscScalar ***pvecA, ***pvecB; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecB, &pvecB); @@ -132,7 +132,7 @@ VecAssemblyEnd(m_gTempB); // Take dot product - double dDotProduct; + PetscScalar dDotProduct; VecDot(m_gTempA, m_gTempB, &dDotProduct); // Output dot product to check @@ -163,7 +163,7 @@ // Alternative (local vector): VecDuplicate(m_vecA, &m_vecDx); // Access the data in the vectors by using pointers - double ***pDx; + PetscScalar ***pDx; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecDx, &pDx); balay at es^/scratch/balay/test $ On Sat, 19 Jul 2014, Satish Balay wrote: > On Sat, 19 Jul 2014, Francis Poulin wrote: > > > Hello Barry, > > > > I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: > > > > ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre > > > > This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. > > PETSc does not use scalapack. Its useful only if you are using mumps.. > > --download-metis --download-parmetis --download-scalapack --download-mumps > > > > > It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. > > > > fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 > > -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory > > Perhaps you do not have csh installed on this machine. You can use mpiexec directly > > ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 > > Satish > > > > > I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. > > > > Thank, > > Francis > > > > > > ------------------ > > Francis Poulin > > Associate Professor > > Associate Chair, Undergraduate Studies > > Department of Applied Mathematics > > University of Waterloo > > > > email: fpoulin at uwaterloo.ca > > Web: https://uwaterloo.ca/poulin-research-group/ > > Telephone: +1 519 888 4567 x32637 > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, July 18, 2014 9:57 PM > > To: John Yawney > > Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi > > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > > > I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. > > > > Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. > > > > The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > > > > > Hello, > > > > > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > > > > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > > > > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > > > > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > > > ./configure --download-f-blas-lapack --with-scalar-type=complex > > > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > > > > > Could this be causing a problem with the parallelization under PETSc? > > > > > > Thanks for the help and sorry for the long question. > > > > > > Best regards, > > > John > > > > > > > > From jed at jedbrown.org Sat Jul 19 11:00:20 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 19 Jul 2014 10:00:20 -0600 Subject: [petsc-users] Unable to download elemental In-Reply-To: References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> Message-ID: <87mwc5nxt7.fsf@jedbrown.org> Please always use "reply-all" so that your messages go to the list and other people can see that the issue is resolved. George Pau writes: > Hi Jed, > > Thanks. It works now. I didn't realize the default openmpi in macport > uses native apple c compiler. If I didn't know better, I'd conclude that Apple's development environment is always a rickety mess. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From wumeng07maths at qq.com Sat Jul 19 11:20:03 2014 From: wumeng07maths at qq.com (=?ISO-8859-1?B?T28gICAgICA=?=) Date: Sun, 20 Jul 2014 00:20:03 +0800 Subject: [petsc-users] MatViewAndMatlabLoad In-Reply-To: <87vbqurf6a.fsf@jedbrown.org> References: <87vbqurf6a.fsf@jedbrown.org> Message-ID: Hi Jed, The matrix file's structure likes ============================== %Size = n n %Nonzeros = m zzz = zeros(m, 3); zzz = [ *** *** *** *** ... ... ]; Mat_0x84000000_0 = spconvert(zzz); ================================ How can I read this file in Matlab? Thanks, Meng ------------------ Original ------------------ From: "Jed Brown";; Date: Jul 18, 2014 To: "Oo "; "petsc-users"; Subject: Re: [petsc-users] MatViewAndMatlabLoad Oo writes: > Dear All, > > > I met a problem about using "load()" function in Matlab to read a m file which describes a PETSc Mat object by : > //Save coeff_matrix as a matlab file PetscViewer viewer; PetscViewerASCIIOpen(PETSC_COMM_WORLD,"CoeffMatrixFileName.m",&viewer); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); MatView(coeff_matrix, viewer); PetscViewerDestroy(&viewer); > In Matlab, I used:M=load("CoeffMatrixFileName.m") and I got an error.The error is:???Error using ===> loadNumber of columns on line 2 of ASCII file CoeffMatrixFileName.m must be the same as previous lines. What does the file look like? Note that we recommend using a binary viewer and A = PetscBinaryRead('binaryoutput') to read in MATLAB. It is more accurate and much faster. > Do you know how to deal with that error?Or Do you think there is some a mistake in my output of PETSc Mat Obj with the PetscViewer? > Thank you~:) > Meng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Jul 19 11:33:23 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 19 Jul 2014 10:33:23 -0600 Subject: [petsc-users] MatViewAndMatlabLoad In-Reply-To: References: <87vbqurf6a.fsf@jedbrown.org> Message-ID: <87fvhxnwa4.fsf@jedbrown.org> Oo writes: > Hi Jed, > The matrix file's structure likes > ============================== > %Size = n n > %Nonzeros = m > zzz = zeros(m, 3); > zzz = [ > *** > *** > *** > *** > ... ... > ]; > > > Mat_0x84000000_0 = spconvert(zzz); > ================================ > > > How can I read this file in Matlab? You just execute that file, but you should definitely use the binary format unless this is just for debugging (and even then, you should generally use the binary format; just last week a user thought they had a bug, but in fact, small errors from rounding in the ASCII format cascaded to make what looked like a significant discrepancy). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From fpoulin at uwaterloo.ca Sat Jul 19 13:09:33 2014 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Sat, 19 Jul 2014 18:09:33 +0000 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> , Message-ID: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> Hello Satish, Thanks for the help on both counts. 1) I installed csh and petscmpiexec works, but I think I will stick to mpirun 2) Your modifications of the code make things work beautifully! Again, sorry to bug you but the help is greatly appreciated. Cheers, Francis ------------------ Francis Poulin Associate Professor Associate Chair, Undergraduate Studies Department of Applied Mathematics University of Waterloo email: fpoulin at uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Saturday, July 19, 2014 11:52 AM To: petsc-users Cc: Francis Poulin Subject: Re: [petsc-users] Question about PETSc installs and MPI And it appears the original code was tested with 'real' numbers and petsc-3.4. It was not really tested with complex numbers. I had to make the following changes for it to run with complex & petsc-3.5 Satish --------- balay at es^/scratch/balay/test $ ../petsc-3.5.0/arch-linux2-c-debug/bin/mpiexec -n 8 valgrind --tool=memcheck -q ./main Dot Product Check... Dot Product (should be 2*256) = 512.000000 Done. Derivative Check... Min (should be 0; if halos not defined properly, would be -1.0/dDeltaX) = 0.000000 Max (should be 0; if halos not defined properly, would be +1.0/dDeltaX) = 0.000000 Done. balay at es^/scratch/balay/test $ diff -Nru main.cpp.orig main.cpp --- main.cpp.orig 2014-07-19 10:48:21.072486076 -0500 +++ main.cpp 2014-07-19 10:47:26.977985006 -0500 @@ -31,9 +31,9 @@ // DMDA Environment ////////////////////////////////////////////////////////////////////////// // Boundary Conditions - DMDABoundaryType bx = DMDA_BOUNDARY_GHOSTED; - DMDABoundaryType by = DMDA_BOUNDARY_NONE; // no boundary points since nGx = 1 - DMDABoundaryType bz = DMDA_BOUNDARY_NONE; // no boundary points since nGz = 1 + DMBoundaryType bx = DM_BOUNDARY_GHOSTED; + DMBoundaryType by = DM_BOUNDARY_NONE; // no boundary points since nGx = 1 + DMBoundaryType bz = DM_BOUNDARY_NONE; // no boundary points since nGz = 1 ////////////////////////////////////////////////////////////////////////// // Stencil type @@ -96,7 +96,7 @@ DMDAGetCorners(m_3da, &si, 0, 0, &ei, 0, 0); // Access the elements of the local arrays as C++ multi-dim. array structures - double ***pvecA, ***pvecB; + PetscScalar ***pvecA, ***pvecB; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecB, &pvecB); @@ -132,7 +132,7 @@ VecAssemblyEnd(m_gTempB); // Take dot product - double dDotProduct; + PetscScalar dDotProduct; VecDot(m_gTempA, m_gTempB, &dDotProduct); // Output dot product to check @@ -163,7 +163,7 @@ // Alternative (local vector): VecDuplicate(m_vecA, &m_vecDx); // Access the data in the vectors by using pointers - double ***pDx; + PetscScalar ***pDx; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecDx, &pDx); balay at es^/scratch/balay/test $ On Sat, 19 Jul 2014, Satish Balay wrote: > On Sat, 19 Jul 2014, Francis Poulin wrote: > > > Hello Barry, > > > > I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: > > > > ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre > > > > This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. > > PETSc does not use scalapack. Its useful only if you are using mumps.. > > --download-metis --download-parmetis --download-scalapack --download-mumps > > > > > It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. > > > > fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 > > -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory > > Perhaps you do not have csh installed on this machine. You can use mpiexec directly > > ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 > > Satish > > > > > I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. > > > > Thank, > > Francis > > > > > > ------------------ > > Francis Poulin > > Associate Professor > > Associate Chair, Undergraduate Studies > > Department of Applied Mathematics > > University of Waterloo > > > > email: fpoulin at uwaterloo.ca > > Web: https://uwaterloo.ca/poulin-research-group/ > > Telephone: +1 519 888 4567 x32637 > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, July 18, 2014 9:57 PM > > To: John Yawney > > Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi > > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > > > I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. > > > > Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. > > > > The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > > > > > Hello, > > > > > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > > > > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > > > > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > > > > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > > > ./configure --download-f-blas-lapack --with-scalar-type=complex > > > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > > > > > Could this be causing a problem with the parallelization under PETSc? > > > > > > Thanks for the help and sorry for the long question. > > > > > > Best regards, > > > John > > > > > > > > From bsmith at mcs.anl.gov Sat Jul 19 15:47:21 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 15:47:21 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: <87mwc5nxt7.fsf@jedbrown.org> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> Message-ID: <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> On Jul 19, 2014, at 11:00 AM, Jed Brown wrote: > Please always use "reply-all" so that your messages go to the list and > other people can see that the issue is resolved. > > George Pau writes: > >> Hi Jed, >> >> Thanks. It works now. I didn't realize the default openmpi in macport >> uses native apple c compiler. > > If I didn't know better, I'd conclude that Apple's development > environment is always a rickety mess. Actually it is very simple and clean (though a little sparse), the problems come from when people use macports, home-brew and all those other ?let?s make package systems like Linux has, for Apple? but lets make them a little less than perfect. Barry From fpoulin at uwaterloo.ca Sat Jul 19 17:16:32 2014 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Sat, 19 Jul 2014 22:16:32 +0000 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> , , <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> Message-ID: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> Hello again, My installation on my desktop seems to work well. I am having problems with my installation of petsc on my mac though. It seems to go through the installation alright and even shows me the nice graph of the performance. I used, ./configure --with-scalar-type=complex --with-cc=clang --with-cxx=clang++ --with-fc=gfortran --with-c2html=0 --download-mpich However, when I try compiling the main.cpp I get an error, which I copied below. I know this should work and wonder what my petsc installation is not finding. Any suggestions? Cheers, Francis TEST_MPI fpoulin$ make all -s clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/lib' clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib' In file included from main.cpp:17: In file included from /Users/fpoulin/software/petsc/include/petsc.h:5: In file included from /Users/fpoulin/software/petsc/include/petscbag.h:4: In file included from /Users/fpoulin/software/petsc/include/petscsys.h:366: /Users/fpoulin/software/petsc/include/petscmath.h:562:10: error: use of undeclared identifier 'cpow'; did you mean 'pow'? return PetscPowScalar(base,cpower); ^ /Users/fpoulin/software/petsc/include/petscmath.h:252:31: note: expanded from macro 'PetscPowScalar' #define PetscPowScalar(a,b) PetscPowComplex(a,b) ^ /Users/fpoulin/software/petsc/include/petscmath.h:176:38: note: expanded from macro 'PetscPowComplex' #define PetscPowComplex(a,b) cpow(a,b) ^ /usr/include/math.h:443:15: note: 'pow' declared here extern double pow(double, double); ^ 1 error generated. make: *** [build/main.o] Error 1 ------------------ Francis Poulin Associate Professor Associate Chair, Undergraduate Studies Department of Applied Mathematics University of Waterloo email: fpoulin at uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________________ From: Francis Poulin Sent: Saturday, July 19, 2014 2:09 PM To: petsc-users Subject: RE: [petsc-users] Question about PETSc installs and MPI Hello Satish, Thanks for the help on both counts. 1) I installed csh and petscmpiexec works, but I think I will stick to mpirun 2) Your modifications of the code make things work beautifully! Again, sorry to bug you but the help is greatly appreciated. Cheers, Francis ------------------ Francis Poulin Associate Professor Associate Chair, Undergraduate Studies Department of Applied Mathematics University of Waterloo email: fpoulin at uwaterloo.ca Web: https://uwaterloo.ca/poulin-research-group/ Telephone: +1 519 888 4567 x32637 ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Saturday, July 19, 2014 11:52 AM To: petsc-users Cc: Francis Poulin Subject: Re: [petsc-users] Question about PETSc installs and MPI And it appears the original code was tested with 'real' numbers and petsc-3.4. It was not really tested with complex numbers. I had to make the following changes for it to run with complex & petsc-3.5 Satish --------- balay at es^/scratch/balay/test $ ../petsc-3.5.0/arch-linux2-c-debug/bin/mpiexec -n 8 valgrind --tool=memcheck -q ./main Dot Product Check... Dot Product (should be 2*256) = 512.000000 Done. Derivative Check... Min (should be 0; if halos not defined properly, would be -1.0/dDeltaX) = 0.000000 Max (should be 0; if halos not defined properly, would be +1.0/dDeltaX) = 0.000000 Done. balay at es^/scratch/balay/test $ diff -Nru main.cpp.orig main.cpp --- main.cpp.orig 2014-07-19 10:48:21.072486076 -0500 +++ main.cpp 2014-07-19 10:47:26.977985006 -0500 @@ -31,9 +31,9 @@ // DMDA Environment ////////////////////////////////////////////////////////////////////////// // Boundary Conditions - DMDABoundaryType bx = DMDA_BOUNDARY_GHOSTED; - DMDABoundaryType by = DMDA_BOUNDARY_NONE; // no boundary points since nGx = 1 - DMDABoundaryType bz = DMDA_BOUNDARY_NONE; // no boundary points since nGz = 1 + DMBoundaryType bx = DM_BOUNDARY_GHOSTED; + DMBoundaryType by = DM_BOUNDARY_NONE; // no boundary points since nGx = 1 + DMBoundaryType bz = DM_BOUNDARY_NONE; // no boundary points since nGz = 1 ////////////////////////////////////////////////////////////////////////// // Stencil type @@ -96,7 +96,7 @@ DMDAGetCorners(m_3da, &si, 0, 0, &ei, 0, 0); // Access the elements of the local arrays as C++ multi-dim. array structures - double ***pvecA, ***pvecB; + PetscScalar ***pvecA, ***pvecB; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecB, &pvecB); @@ -132,7 +132,7 @@ VecAssemblyEnd(m_gTempB); // Take dot product - double dDotProduct; + PetscScalar dDotProduct; VecDot(m_gTempA, m_gTempB, &dDotProduct); // Output dot product to check @@ -163,7 +163,7 @@ // Alternative (local vector): VecDuplicate(m_vecA, &m_vecDx); // Access the data in the vectors by using pointers - double ***pDx; + PetscScalar ***pDx; DMDAVecGetArray(m_3da, m_vecA, &pvecA); DMDAVecGetArray(m_3da, m_vecDx, &pDx); balay at es^/scratch/balay/test $ On Sat, 19 Jul 2014, Satish Balay wrote: > On Sat, 19 Jul 2014, Francis Poulin wrote: > > > Hello Barry, > > > > I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: > > > > ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre > > > > This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. > > PETSc does not use scalapack. Its useful only if you are using mumps.. > > --download-metis --download-parmetis --download-scalapack --download-mumps > > > > > It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. > > > > fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 > > -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory > > Perhaps you do not have csh installed on this machine. You can use mpiexec directly > > ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 > /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 > > Satish > > > > > I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. > > > > Thank, > > Francis > > > > > > ------------------ > > Francis Poulin > > Associate Professor > > Associate Chair, Undergraduate Studies > > Department of Applied Mathematics > > University of Waterloo > > > > email: fpoulin at uwaterloo.ca > > Web: https://uwaterloo.ca/poulin-research-group/ > > Telephone: +1 519 888 4567 x32637 > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, July 18, 2014 9:57 PM > > To: John Yawney > > Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi > > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > > > I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. > > > > Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. > > > > The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jul 18, 2014, at 8:16 PM, John Yawney wrote: > > > > > Hello, > > > > > > I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: > > > > > > ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre > > > > > > I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. > > > > > > When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: > > > ./configure --download-f-blas-lapack --with-scalar-type=complex > > > or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 > > > > > > Could this be causing a problem with the parallelization under PETSc? > > > > > > Thanks for the help and sorry for the long question. > > > > > > Best regards, > > > John > > > > > > > > From bsmith at mcs.anl.gov Sat Jul 19 17:29:45 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 17:29:45 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> , , <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> Message-ID: <2DCBFDCD-BA1E-464A-9BD3-40DA74606B9B@mcs.anl.gov> This is happening because PETSc was configured to compile with the C compiler and use C complex numbers, meanwhile your main program is c++ so doesn?t appreciate the C version of complex numbers (yes, C and C++ are not as friendly as they should be). I?ll try a build and see if I can fix this. You can run the same configure but use the additional ./configure option ?with-clanguage=c++ (two dashes before the with) and it should work because then everything will get built with the C++ compiler and using C++ complex. Barry On Jul 19, 2014, at 5:16 PM, Francis Poulin wrote: > Hello again, > > My installation on my desktop seems to work well. > > I am having problems with my installation of petsc on my mac though. It seems to go through the installation alright and even shows me the nice graph of the performance. I used, > > ./configure --with-scalar-type=complex --with-cc=clang --with-cxx=clang++ --with-fc=gfortran --with-c2html=0 --download-mpich > > However, when I try compiling the main.cpp I get an error, which I copied below. I know this should work and wonder what my petsc installation is not finding. Any suggestions? > > Cheers, Francis > > > TEST_MPI fpoulin$ make all -s > clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/lib' > clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib' > In file included from main.cpp:17: > In file included from /Users/fpoulin/software/petsc/include/petsc.h:5: > In file included from /Users/fpoulin/software/petsc/include/petscbag.h:4: > In file included from /Users/fpoulin/software/petsc/include/petscsys.h:366: > /Users/fpoulin/software/petsc/include/petscmath.h:562:10: error: use of undeclared identifier 'cpow'; did you mean 'pow'? > return PetscPowScalar(base,cpower); > ^ > /Users/fpoulin/software/petsc/include/petscmath.h:252:31: note: expanded from macro 'PetscPowScalar' > #define PetscPowScalar(a,b) PetscPowComplex(a,b) > ^ > /Users/fpoulin/software/petsc/include/petscmath.h:176:38: note: expanded from macro 'PetscPowComplex' > #define PetscPowComplex(a,b) cpow(a,b) > ^ > /usr/include/math.h:443:15: note: 'pow' declared here > extern double pow(double, double); > ^ > 1 error generated. > make: *** [build/main.o] Error 1 > > > > ------------------ > Francis Poulin > Associate Professor > Associate Chair, Undergraduate Studies > Department of Applied Mathematics > University of Waterloo > > email: fpoulin at uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: Francis Poulin > Sent: Saturday, July 19, 2014 2:09 PM > To: petsc-users > Subject: RE: [petsc-users] Question about PETSc installs and MPI > > Hello Satish, > > Thanks for the help on both counts. > > 1) I installed csh and petscmpiexec works, but I think I will stick to mpirun > > 2) Your modifications of the code make things work beautifully! > > Again, sorry to bug you but the help is greatly appreciated. > > Cheers, Francis > > ------------------ > Francis Poulin > Associate Professor > Associate Chair, Undergraduate Studies > Department of Applied Mathematics > University of Waterloo > > email: fpoulin at uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Saturday, July 19, 2014 11:52 AM > To: petsc-users > Cc: Francis Poulin > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > And it appears the original code was tested with 'real' numbers and petsc-3.4. It was > not really tested with complex numbers. I had to make the following changes for > it to run with complex & petsc-3.5 > > Satish > > --------- > > balay at es^/scratch/balay/test $ ../petsc-3.5.0/arch-linux2-c-debug/bin/mpiexec -n 8 valgrind --tool=memcheck -q ./main > Dot Product Check... > > Dot Product (should be 2*256) = 512.000000 > > Done. > > Derivative Check... > > Min (should be 0; if halos not defined properly, would be -1.0/dDeltaX) = 0.000000 > Max (should be 0; if halos not defined properly, would be +1.0/dDeltaX) = 0.000000 > > Done. > balay at es^/scratch/balay/test $ diff -Nru main.cpp.orig main.cpp > --- main.cpp.orig 2014-07-19 10:48:21.072486076 -0500 > +++ main.cpp 2014-07-19 10:47:26.977985006 -0500 > @@ -31,9 +31,9 @@ > // DMDA Environment > ////////////////////////////////////////////////////////////////////////// > // Boundary Conditions > - DMDABoundaryType bx = DMDA_BOUNDARY_GHOSTED; > - DMDABoundaryType by = DMDA_BOUNDARY_NONE; // no boundary points since nGx = 1 > - DMDABoundaryType bz = DMDA_BOUNDARY_NONE; // no boundary points since nGz = 1 > + DMBoundaryType bx = DM_BOUNDARY_GHOSTED; > + DMBoundaryType by = DM_BOUNDARY_NONE; // no boundary points since nGx = 1 > + DMBoundaryType bz = DM_BOUNDARY_NONE; // no boundary points since nGz = 1 > > ////////////////////////////////////////////////////////////////////////// > // Stencil type > @@ -96,7 +96,7 @@ > DMDAGetCorners(m_3da, &si, 0, 0, &ei, 0, 0); > > // Access the elements of the local arrays as C++ multi-dim. array structures > - double ***pvecA, ***pvecB; > + PetscScalar ***pvecA, ***pvecB; > DMDAVecGetArray(m_3da, m_vecA, &pvecA); > DMDAVecGetArray(m_3da, m_vecB, &pvecB); > > @@ -132,7 +132,7 @@ > VecAssemblyEnd(m_gTempB); > > // Take dot product > - double dDotProduct; > + PetscScalar dDotProduct; > VecDot(m_gTempA, m_gTempB, &dDotProduct); > > // Output dot product to check > @@ -163,7 +163,7 @@ > // Alternative (local vector): VecDuplicate(m_vecA, &m_vecDx); > > // Access the data in the vectors by using pointers > - double ***pDx; > + PetscScalar ***pDx; > DMDAVecGetArray(m_3da, m_vecA, &pvecA); > DMDAVecGetArray(m_3da, m_vecDx, &pDx); > > balay at es^/scratch/balay/test $ > > > > On Sat, 19 Jul 2014, Satish Balay wrote: > >> On Sat, 19 Jul 2014, Francis Poulin wrote: >> >>> Hello Barry, >>> >>> I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: >>> >>> ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre >>> >>> This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. >> >> PETSc does not use scalapack. Its useful only if you are using mumps.. >> >> --download-metis --download-parmetis --download-scalapack --download-mumps >> >>> >>> It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. >>> >>> fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 >>> -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory >> >> Perhaps you do not have csh installed on this machine. You can use mpiexec directly >> >> ./ex1 >> /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 >> /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 >> >> Satish >> >>> >>> I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. >>> >>> Thank, >>> Francis >>> >>> >>> ------------------ >>> Francis Poulin >>> Associate Professor >>> Associate Chair, Undergraduate Studies >>> Department of Applied Mathematics >>> University of Waterloo >>> >>> email: fpoulin at uwaterloo.ca >>> Web: https://uwaterloo.ca/poulin-research-group/ >>> Telephone: +1 519 888 4567 x32637 >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Friday, July 18, 2014 9:57 PM >>> To: John Yawney >>> Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi >>> Subject: Re: [petsc-users] Question about PETSc installs and MPI >>> >>> I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. >>> >>> Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. >>> >>> The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> Barry >>> >>> On Jul 18, 2014, at 8:16 PM, John Yawney wrote: >>> >>>> Hello, >>>> >>>> I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: >>>> >>>> ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre >>>> >>>> I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. >>>> >>>> When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: >>>> ./configure --download-f-blas-lapack --with-scalar-type=complex >>>> or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 >>>> >>>> Could this be causing a problem with the parallelization under PETSc? >>>> >>>> Thanks for the help and sorry for the long question. >>>> >>>> Best regards, >>>> John >>>> >>> >>> >> From bsmith at mcs.anl.gov Sat Jul 19 17:49:22 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 17:49:22 -0500 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> References: , <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> , , <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> Message-ID: Jed, How about the attached patch for getting PETSc built for complex in either C or C++ to work in the other language as well? Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: complexpatch Type: application/octet-stream Size: 2484 bytes Desc: not available URL: -------------- next part -------------- On Jul 19, 2014, at 5:16 PM, Francis Poulin wrote: > Hello again, > > My installation on my desktop seems to work well. > > I am having problems with my installation of petsc on my mac though. It seems to go through the installation alright and even shows me the nice graph of the performance. I used, > > ./configure --with-scalar-type=complex --with-cc=clang --with-cxx=clang++ --with-fc=gfortran --with-c2html=0 --download-mpich > > However, when I try compiling the main.cpp I get an error, which I copied below. I know this should work and wonder what my petsc installation is not finding. Any suggestions? > > Cheers, Francis > > > TEST_MPI fpoulin$ make all -s > clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/lib' > clang: warning: argument unused during compilation: '-L/Users/fpoulin/software/petsc/darwin10.6.0-c-debug/lib' > In file included from main.cpp:17: > In file included from /Users/fpoulin/software/petsc/include/petsc.h:5: > In file included from /Users/fpoulin/software/petsc/include/petscbag.h:4: > In file included from /Users/fpoulin/software/petsc/include/petscsys.h:366: > /Users/fpoulin/software/petsc/include/petscmath.h:562:10: error: use of undeclared identifier 'cpow'; did you mean 'pow'? > return PetscPowScalar(base,cpower); > ^ > /Users/fpoulin/software/petsc/include/petscmath.h:252:31: note: expanded from macro 'PetscPowScalar' > #define PetscPowScalar(a,b) PetscPowComplex(a,b) > ^ > /Users/fpoulin/software/petsc/include/petscmath.h:176:38: note: expanded from macro 'PetscPowComplex' > #define PetscPowComplex(a,b) cpow(a,b) > ^ > /usr/include/math.h:443:15: note: 'pow' declared here > extern double pow(double, double); > ^ > 1 error generated. > make: *** [build/main.o] Error 1 > > > > ------------------ > Francis Poulin > Associate Professor > Associate Chair, Undergraduate Studies > Department of Applied Mathematics > University of Waterloo > > email: fpoulin at uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: Francis Poulin > Sent: Saturday, July 19, 2014 2:09 PM > To: petsc-users > Subject: RE: [petsc-users] Question about PETSc installs and MPI > > Hello Satish, > > Thanks for the help on both counts. > > 1) I installed csh and petscmpiexec works, but I think I will stick to mpirun > > 2) Your modifications of the code make things work beautifully! > > Again, sorry to bug you but the help is greatly appreciated. > > Cheers, Francis > > ------------------ > Francis Poulin > Associate Professor > Associate Chair, Undergraduate Studies > Department of Applied Mathematics > University of Waterloo > > email: fpoulin at uwaterloo.ca > Web: https://uwaterloo.ca/poulin-research-group/ > Telephone: +1 519 888 4567 x32637 > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Saturday, July 19, 2014 11:52 AM > To: petsc-users > Cc: Francis Poulin > Subject: Re: [petsc-users] Question about PETSc installs and MPI > > And it appears the original code was tested with 'real' numbers and petsc-3.4. It was > not really tested with complex numbers. I had to make the following changes for > it to run with complex & petsc-3.5 > > Satish > > --------- > > balay at es^/scratch/balay/test $ ../petsc-3.5.0/arch-linux2-c-debug/bin/mpiexec -n 8 valgrind --tool=memcheck -q ./main > Dot Product Check... > > Dot Product (should be 2*256) = 512.000000 > > Done. > > Derivative Check... > > Min (should be 0; if halos not defined properly, would be -1.0/dDeltaX) = 0.000000 > Max (should be 0; if halos not defined properly, would be +1.0/dDeltaX) = 0.000000 > > Done. > balay at es^/scratch/balay/test $ diff -Nru main.cpp.orig main.cpp > --- main.cpp.orig 2014-07-19 10:48:21.072486076 -0500 > +++ main.cpp 2014-07-19 10:47:26.977985006 -0500 > @@ -31,9 +31,9 @@ > // DMDA Environment > ////////////////////////////////////////////////////////////////////////// > // Boundary Conditions > - DMDABoundaryType bx = DMDA_BOUNDARY_GHOSTED; > - DMDABoundaryType by = DMDA_BOUNDARY_NONE; // no boundary points since nGx = 1 > - DMDABoundaryType bz = DMDA_BOUNDARY_NONE; // no boundary points since nGz = 1 > + DMBoundaryType bx = DM_BOUNDARY_GHOSTED; > + DMBoundaryType by = DM_BOUNDARY_NONE; // no boundary points since nGx = 1 > + DMBoundaryType bz = DM_BOUNDARY_NONE; // no boundary points since nGz = 1 > > ////////////////////////////////////////////////////////////////////////// > // Stencil type > @@ -96,7 +96,7 @@ > DMDAGetCorners(m_3da, &si, 0, 0, &ei, 0, 0); > > // Access the elements of the local arrays as C++ multi-dim. array structures > - double ***pvecA, ***pvecB; > + PetscScalar ***pvecA, ***pvecB; > DMDAVecGetArray(m_3da, m_vecA, &pvecA); > DMDAVecGetArray(m_3da, m_vecB, &pvecB); > > @@ -132,7 +132,7 @@ > VecAssemblyEnd(m_gTempB); > > // Take dot product > - double dDotProduct; > + PetscScalar dDotProduct; > VecDot(m_gTempA, m_gTempB, &dDotProduct); > > // Output dot product to check > @@ -163,7 +163,7 @@ > // Alternative (local vector): VecDuplicate(m_vecA, &m_vecDx); > > // Access the data in the vectors by using pointers > - double ***pDx; > + PetscScalar ***pDx; > DMDAVecGetArray(m_3da, m_vecA, &pvecA); > DMDAVecGetArray(m_3da, m_vecDx, &pDx); > > balay at es^/scratch/balay/test $ > > > > On Sat, 19 Jul 2014, Satish Balay wrote: > >> On Sat, 19 Jul 2014, Francis Poulin wrote: >> >>> Hello Barry, >>> >>> I was one of the two people that had difficulties with getting the correct results with John's code. Previously, I didn't have valgrind installed so I installed it using apt-get. Then I configured it using the following: >>> >>> ./configure --with-scalar-type=complex --with-cc=gcc -?with-cxx=c++ --with-fc=gfortran --with-c2html=0 --download-mpich --download-scalapack --download-hypre >>> >>> This is on ubuntu and is different from what I tried before in that now I am downloading mpich, scalapack and hypre. I decided to download scalapack since that seems like it could be useful. I was told that HYPRE doesn't work with complex variables. Too bad, but not a big deal. >> >> PETSc does not use scalapack. Its useful only if you are using mumps.. >> >> --download-metis --download-parmetis --download-scalapack --download-mumps >> >>> >>> It completes the configure, make all and make test and even gives me the figures of the parallel efficiency (or not quite efficiency maybe). I didn't catch any errors, but there are possible errors in the log. When I went to try making an example I found that I can't use petscmpiexec to run anything in serial or parallel. >>> >>> fpoulin at vortex:~/software/petsc/src/ts/examples/tutorials$ /home/fpoulin/software/petsc/bin/petscmpiexec -n 1 ./ex1 >>> -bash: /home/fpoulin/software/petsc/bin/petscmpiexec: /bin/csh: bad interpreter: No such file or directory >> >> Perhaps you do not have csh installed on this machine. You can use mpiexec directly >> >> ./ex1 >> /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 1 ./ex1 >> /home/fpoulin/software/petsc/arch-linux2-c-debug/bin/mpiexec -n 2 ./ex1 >> >> Satish >> >>> >>> I am sorry to bother you with this. I am also having issues with my installation on my mac but I thought if i can figure this one out then maybe I will have a better idea what's wrong with the other. >>> >>> Thank, >>> Francis >>> >>> >>> ------------------ >>> Francis Poulin >>> Associate Professor >>> Associate Chair, Undergraduate Studies >>> Department of Applied Mathematics >>> University of Waterloo >>> >>> email: fpoulin at uwaterloo.ca >>> Web: https://uwaterloo.ca/poulin-research-group/ >>> Telephone: +1 519 888 4567 x32637 >>> >>> >>> ________________________________________ >>> From: Barry Smith [bsmith at mcs.anl.gov] >>> Sent: Friday, July 18, 2014 9:57 PM >>> To: John Yawney >>> Cc: petsc-users at mcs.anl.gov; Francis Poulin; Kim Usi >>> Subject: Re: [petsc-users] Question about PETSc installs and MPI >>> >>> I ran the program on linux with 1,2, 4 processes under valgrind for both types of boundary conditions and it ran fine. >>> >>> Suggest your colleagues do a test configure of PETSc using ?download-mpich and see if they still get the problem or if it runs ok. >>> >>> The can also run with valgrind and see what it reports. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> Barry >>> >>> On Jul 18, 2014, at 8:16 PM, John Yawney wrote: >>> >>>> Hello, >>>> >>>> I had a question about PETSc installations. On my local computer I configured PETSc (v 3.4.2) using the options: >>>> >>>> ./configure --with-cc=mpicc --with-cxx=mpic++ --download-f-blas-lapack --download-mpich --download-hypre >>>> >>>> I wrote a test program that defines a vector using DMDAs, computes a dot product, exchanges halo elements, and computes a low-order FD derivative of the vector. Under my installation of PETSc everything works fine. For some reason, when my colleagues run the program, they get segmentation fault errors. If they change the y and z boundary types to GHOSTED as well, they get the program to run until the end (seg faults at the end though) but they get a local value of the dot product. I've attached the main.cpp file for this script. >>>> >>>> When they installed their versions of PETSc they didn't use the --download-mpich option but instead used either: >>>> ./configure --download-f-blas-lapack --with-scalar-type=complex >>>> or with the option: --with-mpi-dir=/home/kim/anaconda/pkgs/mpich2-1.3-py27_0 >>>> >>>> Could this be causing a problem with the parallelization under PETSc? >>>> >>>> Thanks for the help and sorry for the long question. >>>> >>>> Best regards, >>>> John >>>> >>> >>> >> From jed at jedbrown.org Sat Jul 19 19:16:30 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 19 Jul 2014 18:16:30 -0600 Subject: [petsc-users] Unable to download elemental In-Reply-To: <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> Message-ID: <877g38opep.fsf@jedbrown.org> Barry Smith writes: > Actually it is very simple and clean (though a little sparse), the > problems come from when people use macports, home-brew and all those > other ?let?s make package systems like Linux has, for Apple? but > lets make them a little less than perfect. Either everyone that has attempted to automate software installation, upgrades, and dependencies for Mac systems is an idiot or Apple has made an ecosystem that is hostile to the automation of such things. Doing these things manually is a monumental waste of time. Sometime in the 90s, I took it for granted that compilers, debuggers, Octave, and the like Just Worked, would always be up-to-date with normalized dependencies so that any combination could be linked and so that binary security patches were trivial. Valgrind came soon after and I never looked back. To think that we're even talking about this in 2014 is depressing. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Jul 19 19:35:11 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 19:35:11 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: <877g38opep.fsf@jedbrown.org> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> <877g38opep.fsf@jedbrown.org> Message-ID: <9B91A46D-04E7-4B21-A352-E1F633F69904@mcs.anl.gov> Jed, You miss the point that Apple thinks all those Linux things you want to use on Apple are just not relevant and don?t need to be used. In the Apple world you use Xcode and the Frameworks that Apple supplies and everything is hunky-dory; no need to mess with other stuff. Apple simply doesn?t believe in an ecosystem provided by ?a bunch of people?. Now you can disagree that the Frameworks that Apple provides are all you will ever need but if you buy into it then it is a sane and relatively simple, though far from perfect, system. Barry On Jul 19, 2014, at 7:16 PM, Jed Brown wrote: > Barry Smith writes: >> Actually it is very simple and clean (though a little sparse), the >> problems come from when people use macports, home-brew and all those >> other ?let?s make package systems like Linux has, for Apple? but >> lets make them a little less than perfect. > > Either everyone that has attempted to automate software installation, > upgrades, and dependencies for Mac systems is an idiot or Apple has made > an ecosystem that is hostile to the automation of such things. Doing > these things manually is a monumental waste of time. Sometime in the > 90s, I took it for granted that compilers, debuggers, Octave, and the > like Just Worked, would always be up-to-date with normalized > dependencies so that any combination could be linked and so that binary > security patches were trivial. Valgrind came soon after and I never > looked back. To think that we're even talking about this in 2014 is > depressing. From jed at jedbrown.org Sat Jul 19 19:58:02 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 19 Jul 2014 18:58:02 -0600 Subject: [petsc-users] Unable to download elemental In-Reply-To: <9B91A46D-04E7-4B21-A352-E1F633F69904@mcs.anl.gov> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> <877g38opep.fsf@jedbrown.org> <9B91A46D-04E7-4B21-A352-E1F633F69904@mcs.anl.gov> Message-ID: <874myconhh.fsf@jedbrown.org> Barry Smith writes: >Frameworks And this is what got us into the whole mess. People always think they have built the perfect island and that everyone will be content to live on the island forever. As long as you want to do what the Ghost of Steve Jobs thinks will maximize shareholder value, Apple products will work wonderfully. But don't try to do something creative or a ship loaded with dogs, rats, pigs, and cockroaches will run aground and destroy all the wonderful things, leaving you on a screwed-up island in the middle of an ocean. I wouldn't even mind except that the cost of Apple's perverse incentives creating a hostile environment for this enterprising user base is largely borne by the developers of well-behaving portable software that end up supporting those poor saps. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Jul 19 20:16:49 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 19 Jul 2014 20:16:49 -0500 Subject: [petsc-users] Unable to download elemental In-Reply-To: <874myconhh.fsf@jedbrown.org> References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> <877g38opep.fsf@jedbrown.org> <9B91A46D-04E7-4B21-A352-E1F633F69904@mcs.anl.gov> <874myconhh.fsf@jedbrown.org> Message-ID: The cost and effort for us to support Apple is much much less then the cost and effort for us to support Microsoft Windows! In fact one could rationally argue that the cost of supporting the multitude of badly configured Linux system is larger that that of support Apple machines. Barry On Jul 19, 2014, at 7:58 PM, Jed Brown wrote: > Barry Smith writes: >> Frameworks > > And this is what got us into the whole mess. People always think they > have built the perfect island and that everyone will be content to live > on the island forever. > > As long as you want to do what the Ghost of Steve Jobs thinks will > maximize shareholder value, Apple products will work wonderfully. But > don't try to do something creative or a ship loaded with dogs, rats, > pigs, and cockroaches will run aground and destroy all the wonderful > things, leaving you on a screwed-up island in the middle of an ocean. > > I wouldn't even mind except that the cost of Apple's perverse incentives > creating a hostile environment for this enterprising user base is > largely borne by the developers of well-behaving portable software that > end up supporting those poor saps. From jed at jedbrown.org Sat Jul 19 20:37:54 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 19 Jul 2014 19:37:54 -0600 Subject: [petsc-users] Unable to download elemental In-Reply-To: References: <5BAD42FC-1007-42EE-ACAC-15CCD38DA87A@mcs.anl.gov> <87a986p652.fsf@jedbrown.org> <87mwc5nxt7.fsf@jedbrown.org> <4889E525-C8B3-4236-ACFC-B5568CAEFA89@mcs.anl.gov> <877g38opep.fsf@jedbrown.org> <9B91A46D-04E7-4B21-A352-E1F633F69904@mcs.anl.gov> <874myconhh.fsf@jedbrown.org> Message-ID: <871ttgoln1.fsf@jedbrown.org> Barry Smith writes: > The cost and effort for us to support Apple is much much less then > the cost and effort for us to support Microsoft Windows! Indeed, and there are airplanes even worse for air-to-air combat than the F-35. > In fact one could rationally argue that the cost of supporting the > multitude of badly configured Linux system is larger that that of > support Apple machines. Those $50M machines have such catastrophic package management they can hardly be called a Linux Distribution. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From dominik at itis.ethz.ch Sun Jul 20 09:00:57 2014 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sun, 20 Jul 2014 16:00:57 +0200 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: <8761j7fc9j.fsf@jedbrown.org> References: <8761j7fc9j.fsf@jedbrown.org> Message-ID: __FUNCT__="ScalarSolver::SetupStiffnessMatrix" does not compile. I assume you meant: #undef __FUNCT__ #define __FUNCT__ "ScalarSolver::SetupCoefficientMatrix" but now I do not get exact lines leading to a crash. Also skipping __FUNCT__ completely does not help. It used to work before, I can not figure out what has changed. Using MSVC 2010. Please advise... [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption error [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] ScalarSolver::SetupCoefficientMatrix line 1216 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp [0]PETSC ERROR: [0] ScalarSolver::Initialize line 181 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp [0]PETSC ERROR: [0] ScalarSolverXMLDriver::Initialize line 74 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolverXMLDriver.cpp [0]PETSC ERROR: [0] ScalarSolverXMLDriver::ScalarSolverXMLDriver line 53 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolverXMLDriver.cpp [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 2014-07-09 5:21 GMT+02:00 Jed Brown : > Satish Balay writes: > > > One can skip __FUNCT__ alltogether in user sourcefiles [assuming you > > have a c99 compiler] - as __func__ would automatically be used > > by the petsc macros [in includes] > > You can literally skip it everywhere, C99 or not, and PETSc will not > issue these warnings. If __func__ is not available, traces will say > "User defined function". > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jul 20 11:23:46 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 20 Jul 2014 11:23:46 -0500 Subject: [petsc-users] __FUNCT__ does not agree with __FUNCTION__ In-Reply-To: References: <8761j7fc9j.fsf@jedbrown.org> Message-ID: <145A9925-8063-4B88-8E84-2B20A0F8EB18@mcs.anl.gov> You need to run in the debugger to find out exactly where and why it is crashing. Barry On Jul 20, 2014, at 9:00 AM, Dominik Szczerba wrote: > __FUNCT__="ScalarSolver::SetupStiffnessMatrix" > > does not compile. I assume you meant: > > #undef __FUNCT__ > #define __FUNCT__ "ScalarSolver::SetupCoefficientMatrix" > > but now I do not get exact lines leading to a crash. Also skipping __FUNCT__ completely does not help. It used to work before, I can not figure out what has changed. Using MSVC 2010. > Please advise... > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption error > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] ScalarSolver::SetupCoefficientMatrix line 1216 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp > [0]PETSC ERROR: [0] ScalarSolver::Initialize line 181 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolver.cpp > [0]PETSC ERROR: [0] ScalarSolverXMLDriver::Initialize line 74 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolverXMLDriver.cpp > [0]PETSC ERROR: [0] ScalarSolverXMLDriver::ScalarSolverXMLDriver line 53 "unknowndirectory/"..\..\..\Source\ScalarSolver\ScalarSolver.x\ScalarSolverXMLDriver.cpp > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > > 2014-07-09 5:21 GMT+02:00 Jed Brown : > Satish Balay writes: > > > One can skip __FUNCT__ alltogether in user sourcefiles [assuming you > > have a c99 compiler] - as __func__ would automatically be used > > by the petsc macros [in includes] > > You can literally skip it everywhere, C99 or not, and PETSc will not > issue these warnings. If __func__ is not available, traces will say > "User defined function". > From jed at jedbrown.org Sun Jul 20 22:48:02 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 20 Jul 2014 21:48:02 -0600 Subject: [petsc-users] Question about PETSc installs and MPI In-Reply-To: References: <49B49687-8610-4D98-B88C-29E96B420CC1@mcs.anl.gov> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11644@connmbx02> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D11C2C@connmbx02> <4C887A7BEDB3D64E9F30EBF58F8D4BDE13D122A6@connmbx02> Message-ID: <87k377l6dp.fsf@jedbrown.org> Barry Smith writes: > How about the attached patch for getting PETSc built for complex > in either C or C++ to work in the other language as well? I like it and now I notice that PETSC_CLANGUAGE_CXX now appears in only one place (CHKERRCXX), which I hope we can also change to __cplusplus. Somehow I was hung up on std::complex being a non-POD type, but this commit is fairly orthogonal. There is no std::complex<__float128>, but as far as I'm concerned, that's what people get if they try to use C++. We could use __complex128 even in C++ mode, but I could see this breaking in more confusing ways. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From stephen.wornom at inria.fr Mon Jul 21 04:44:01 2014 From: stephen.wornom at inria.fr (Stephen Wornom) Date: Mon, 21 Jul 2014 11:44:01 +0200 Subject: [petsc-users] MPI-IO Message-ID: <53CCE0E1.4040504@inria.fr> I have an unstructured mesh code used to compute vortex shedding problems saving the solutions every 500-1000 time steps. The mesh size is 3 to 20 MNodes. The minimum number of cores that I us is 128 for the 3MNode mesh. I would like to know if PETSC could be used to use to save the solutions using MPI-IO? Hope my question is clear, Stephen From lawrence.mitchell at imperial.ac.uk Mon Jul 21 05:18:10 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 21 Jul 2014 11:18:10 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel Message-ID: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> Hello all, I'm implementing a multigrid solver using PCMG, starting with a simple Poisson equation (with strong boundary conditions) to ensure I'm doing things right. Everything works fine in serial, but when running on two processes, with the default chebyshev smoother, convergence goes to pot, in particular, the preconditioner becomes indefinite. I've determined that the operators I'm building on each level are indeed symmetric, furthermore, if I switch the smoother to jacobi (or sor) preconditioned richardson iterations then I get good convergence in both serial and parallel runs. The eigenvalue estimates for the chebyshev smoother look plausible in both cases. Any suggestions as to where to look next to figure out what's wrong in my code? I can try and untangle it such that it demonstrates the problem standalone, but that may take a little time. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dave.mayhem23 at gmail.com Mon Jul 21 05:50:08 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 21 Jul 2014 12:50:08 +0200 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> Message-ID: Hi Lawrence, Hmm this sounds odd. The convergence obtained with chebyshev should be essentially identical in serial and parallel when using a jacobi preconditioner 1) How did you configure the coarse grid solver in the serial and parallel test? Are they consistent? 2) Does using one level with PCMG and a chebyshev smoother give the same answers in serial and parallel? If you precondition with jacobi, the residuals in serial and parallel should be very similar. If this test didn't pass, try a 1 level method again again without the preconditioner (i.e. just apply chebyshev). If the residuals are really different between these runs, there is likely something wrong/inconsistent with the definition of the operator on each level, or the way the boundary conditions are imposed. 3) Is the code valgrind clean? Cheers, Dave On 21 July 2014 12:18, Lawrence Mitchell wrote: > Hello all, > > I'm implementing a multigrid solver using PCMG, starting with a simple > Poisson equation (with strong boundary conditions) to ensure I'm doing > things right. Everything works fine in serial, but when running on two > processes, with the default chebyshev smoother, convergence goes to pot, in > particular, the preconditioner becomes indefinite. > > I've determined that the operators I'm building on each level are indeed > symmetric, furthermore, if I switch the smoother to jacobi (or sor) > preconditioned richardson iterations then I get good convergence in both > serial and parallel runs. The eigenvalue estimates for the chebyshev > smoother look plausible in both cases. Any suggestions as to where to look > next to figure out what's wrong in my code? I can try and untangle it such > that it demonstrates the problem standalone, but that may take a little > time. > > Cheers, > > Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 21 06:01:16 2014 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Jul 2014 06:01:16 -0500 Subject: [petsc-users] MPI-IO In-Reply-To: <53CCE0E1.4040504@inria.fr> References: <53CCE0E1.4040504@inria.fr> Message-ID: On Jul 21, 2014 5:44 AM, "Stephen Wornom" wrote: > > I have an unstructured mesh code used to compute vortex shedding problems saving the solutions every 500-1000 time steps. The mesh size is 3 to 20 MNodes. The minimum number of cores that I us is 128 for the 3MNode mesh. > I would like to know if PETSC could be used to use to save the solutions using MPI-IO? The normal VecView() for the binary viewer will use MPI/IO. Thanks Matt > Hope my question is clear, > Stephen -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Mon Jul 21 06:41:35 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 21 Jul 2014 12:41:35 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> Message-ID: On 21 Jul 2014, at 11:50, Dave May wrote: > Hi Lawrence, > > Hmm this sounds odd. The convergence obtained with chebyshev should be essentially identical in serial and parallel when using a jacobi preconditioner So I was maybe a bit unclear in my previous mail: If I run with: -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi -mg_levels_ksp_max_it 2 then I get identical convergence in serial and parallel if, however, I run with -pc_type mg -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it 2 (the default according to -ksp_view) then I get very differing convergence in serial and parallel as described. > 1) How did you configure the coarse grid solver in the serial and parallel test? Are they consistent? I initially just used the default (which is LU in serial and redundant with LU in parallel), when I rerun with: -pc_type mg -ksp_type fgmres -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 Which should be the same in serial and parallel, I again see bad behaviour in parallel. > 2) Does using one level with PCMG and a chebyshev smoother give the same answers in serial and parallel? If you precondition with jacobi, the residuals in serial and parallel should be very similar. If this test didn't pass, try a 1 level method again again without the preconditioner (i.e. just apply chebyshev). If the residuals are really different between these runs, there is likely something wrong/inconsistent with the definition of the operator on each level, or the way the boundary conditions are imposed. For these tests, I use the following options: -pc_type mg -ksp_type fgmres -pc_mg_levels 2 -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 I then tried the following options for the smoother: -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it 2 Works in serial, doesn't converge well in parallel. -mg_levels_ksp_type chebyshev -mg_levels_pc_type jacobi -mg_levels_ksp_max_it 2 Converges veery slowly in both serial and parallel (but has the same convergence in both cases). -mg_levels_ksp_type chebyshev -mg_levels_pc_type none -mg_levels_ksp_max_it 2 Converges well in both serial and parallel (identical convergence behaviour). > 3) Is the code valgrind clean? It's python-based, so it's a little difficult to say. It appears to be so. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dave.mayhem23 at gmail.com Mon Jul 21 06:52:27 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 21 Jul 2014 13:52:27 +0200 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> Message-ID: > -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi > -mg_levels_ksp_max_it 2 > > then I get identical convergence in serial and parallel > > Good. That's the correct result. > if, however, I run with > > -pc_type mg -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor > -mg_levels_ksp_max_it 2 > (the default according to -ksp_view) > > then I get very differing convergence in serial and parallel as described. > > It's normal that the behaviour is different. The PETSc SOR implementation is not parallel. It only performs SOR on your local subdomain. > > 1) How did you configure the coarse grid solver in the serial and > parallel test? Are they consistent? > > I initially just used the default (which is LU in serial and redundant > with LU in parallel), when I rerun with: > > -pc_type mg -ksp_type fgmres -mg_coarse_ksp_type gmres -mg_coarse_pc_type > jacobi -mg_coarse_ksp_max_it 100 > > Which should be the same in serial and parallel, I again see bad behaviour > in parallel. > > I see that this is a nested Krylov solve. Using fgmres on the outer sometimes is not enough. I've had problems where I needed to use the more stable orthogonalization routine in gmres. Do you also observe different convergence behaviour (serial versus parallel) with these choices 1) -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 1 2) -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_coarse_ksp_gmres_modifiedgramschmidt 3) -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 > > 2) Does using one level with PCMG and a chebyshev smoother give the same > answers in serial and parallel? If you precondition with jacobi, the > residuals in serial and parallel should be very similar. If this test > didn't pass, try a 1 level method again again without the preconditioner > (i.e. just apply chebyshev). If the residuals are really different between > these runs, there is likely something wrong/inconsistent with the > definition of the operator on each level, or the way the boundary > conditions are imposed. > > For these tests, I use the following options: > > -pc_type mg -ksp_type fgmres -pc_mg_levels 2 -mg_coarse_ksp_type gmres > -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 > > I then tried the following options for the smoother: > > -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it > 2 > > Works in serial, doesn't converge well in parallel. > > Sure - see above. > -mg_levels_ksp_type chebyshev -mg_levels_pc_type jacobi > -mg_levels_ksp_max_it 2 > > Converges veery slowly in both serial and parallel (but has the same > convergence in both cases). > > -mg_levels_ksp_type chebyshev -mg_levels_pc_type none > -mg_levels_ksp_max_it 2 > > Converges well in both serial and parallel (identical convergence > behaviour). > > Sure - this wasn't a convergence test. I just wanted to see that the methods which should be identical in serial and parallel are in fact behaving as expected. Seems there are. So I'm included to think the problem is associated with having nested Krylov solves. Cheers, Dave > > > 3) Is the code valgrind clean? > > It's python-based, so it's a little difficult to say. It appears to be so. > > Cheers, > > Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Mon Jul 21 07:11:48 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 21 Jul 2014 13:11:48 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> Message-ID: <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> On 21 Jul 2014, at 12:52, Dave May wrote: > > -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi -mg_levels_ksp_max_it 2 > > then I get identical convergence in serial and parallel > > > Good. That's the correct result. > > if, however, I run with > > -pc_type mg -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it 2 > (the default according to -ksp_view) > > then I get very differing convergence in serial and parallel as described. > > > It's normal that the behaviour is different. The PETSc SOR implementation is not parallel. It only performs SOR on your local subdomain. Sure, however, with only two subdomains, I was not expecting to see such poor behaviour. Below I show output from a run on 1 process and then two (along with ksp_view) for the following options: -pc_type mg -ksp_rtol 1e-8 -ksp_max_it 6 -pc_mg_levels 2 -mg_levels_pc_type sor -ksp_monitor On 1 process: 0 KSP Residual norm 5.865090856053e+02 1 KSP Residual norm 1.293159126247e+01 2 KSP Residual norm 5.181199296299e-01 3 KSP Residual norm 1.268870802643e-02 4 KSP Residual norm 5.116058930806e-04 5 KSP Residual norm 3.735036960550e-05 6 KSP Residual norm 1.755288530515e-06 KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=6, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=2 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5, needed 3.17724 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=144, cols=144 package used to perform factorization: petsc total: nonzeros=2904, allocated nonzeros=2904 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=144, cols=144 total: nonzeros=914, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 1 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.0999972, max = 1.09997 Chebyshev: estimated using: [0 0.1; 0 1.1] KSP Object: (mg_levels_1_est_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=529, cols=529 total: nonzeros=3521, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=529, cols=529 total: nonzeros=3521, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=529, cols=529 total: nonzeros=3521, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines On 2: 0 KSP Residual norm 5.867749653193e+02 1 KSP Residual norm 1.353369658350e+01 2 KSP Residual norm 1.350163644248e+01 3 KSP Residual norm 1.007552895680e+01 4 KSP Residual norm 1.294191582208e+00 5 KSP Residual norm 9.409953768968e-01 6 KSP Residual norm 9.409360529590e-01 KSP Object: 2 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=6, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 2 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=2 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 2 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 2 MPI processes type: redundant Redundant preconditioner: First (color=0) of 2 PCs follows KSP Object: (mg_coarse_redundant_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_redundant_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5, needed 2.72494 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=144, cols=144 package used to perform factorization: petsc total: nonzeros=2120, allocated nonzeros=2120 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=144, cols=144 total: nonzeros=778, allocated nonzeros=778 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpiaij rows=144, cols=144 total: nonzeros=778, allocated nonzeros=914 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 2 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.099992, max = 1.09991 Chebyshev: estimated using: [0 0.1; 0 1.1] KSP Object: (mg_levels_1_est_) 2 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 2 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpiaij rows=529, cols=529 total: nonzeros=3253, allocated nonzeros=3521 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 2 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpiaij rows=529, cols=529 total: nonzeros=3253, allocated nonzeros=3521 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 2 MPI processes type: mpiaij rows=529, cols=529 total: nonzeros=3253, allocated nonzeros=3521 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines So notice that in the parallel case the residual reduction was ~10^3, rather than ~10^8 for the serial case. > I see that this is a nested Krylov solve. Using fgmres on the outer sometimes is not enough. I've had problems where I needed to use the more stable orthogonalization routine in gmres. > > Do you also observe different convergence behaviour (serial versus parallel) with these choices > 1) -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 1 Full options are (in addition to the above): -ksp_type fgmres -pc_mg_levels 2 -ksp_monitor -ksp_max_it 6 -ksp_rtol 1e-8 -pc_type mg 1 process: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.294103921871e+01 2 KSP Residual norm 4.325949294172e+00 3 KSP Residual norm 1.373260455913e+00 4 KSP Residual norm 1.612639229769e-01 5 KSP Residual norm 1.896600662807e-02 6 KSP Residual norm 5.900847991084e-03 2 processes: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.242896923248e+01 2 KSP Residual norm 1.092088559774e+01 3 KSP Residual norm 7.383276000966e+00 4 KSP Residual norm 5.634790202135e+00 5 KSP Residual norm 4.329897745238e+00 6 KSP Residual norm 3.754170628391e+00 > 2) -mg_coarse_ksp_type gmres -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_coarse_ksp_gmres_modifiedgramschmidt 1 process: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.030455192067e+01 2 KSP Residual norm 4.628068378242e-01 3 KSP Residual norm 1.965313019262e-02 4 KSP Residual norm 1.204109484597e-03 5 KSP Residual norm 5.812650812813e-05 6 KSP Residual norm 3.161780444565e-06 2 processes: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.324768309183e+01 2 KSP Residual norm 1.225921121405e+01 3 KSP Residual norm 1.173286143250e+01 4 KSP Residual norm 7.033886488294e+00 5 KSP Residual norm 4.825036058054e+00 6 KSP Residual norm 4.265434976636e+00 > 3) -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 1 process: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.030455192067e+01 2 KSP Residual norm 4.628068378242e-01 3 KSP Residual norm 1.965313019262e-02 4 KSP Residual norm 1.204109484597e-03 5 KSP Residual norm 5.812650812814e-05 6 KSP Residual norm 3.161780444567e-06 2 processes: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.324768309183e+01 2 KSP Residual norm 1.225921121405e+01 3 KSP Residual norm 1.173286143250e+01 4 KSP Residual norm 7.033886488294e+00 5 KSP Residual norm 4.825036058053e+00 6 KSP Residual norm 4.265434976635e+00 > Sure - this wasn't a convergence test. I just wanted to see that the methods which should be identical in serial and parallel are in fact behaving as expected. Seems there are. So I'm included to think the problem is associated with having nested Krylov solves. My observation appears to be that if I use unpreconditioned chebyshev as a smoother, then convergence in serial and parallel is identical and good. As soon as I turn on SOR preconditioning for the smoother, the parallel convergence falls to pieces (and the preconditioner becomes indefinite): e.g. with -pc_type mg -ksp_rtol 1e-8 -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor -ksp_type fgmres -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_levels_pc_type none 1 process: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.530397174638e+01 2 KSP Residual norm 1.027554200472e+00 3 KSP Residual norm 3.809236982955e-02 4 KSP Residual norm 2.445633720099e-03 5 KSP Residual norm 1.192136916270e-04 6 KSP Residual norm 7.067629143105e-06 2 processes: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.530397174638e+01 2 KSP Residual norm 1.027554200472e+00 3 KSP Residual norm 3.809236982955e-02 4 KSP Residual norm 2.445633720099e-03 5 KSP Residual norm 1.192136916270e-04 6 KSP Residual norm 7.067629143079e-06 with sor as a preconditioner: -pc_type mg -ksp_rtol 1e-8 -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor -ksp_type fgmres -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_levels_pc_type sor 1 process: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.030455192067e+01 2 KSP Residual norm 4.628068378242e-01 3 KSP Residual norm 1.965313019262e-02 4 KSP Residual norm 1.204109484597e-03 5 KSP Residual norm 5.812650812814e-05 6 KSP Residual norm 3.161780444567e-06 2 processes: 0 KSP Residual norm 2.802543487620e+02 1 KSP Residual norm 1.324768309183e+01 2 KSP Residual norm 1.225921121405e+01 3 KSP Residual norm 1.173286143250e+01 4 KSP Residual norm 7.033886488294e+00 5 KSP Residual norm 4.825036058053e+00 6 KSP Residual norm 4.265434976635e+00 Maybe it's just that I shouldn't be expecting this to work, but it seems odd to me. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jed at jedbrown.org Mon Jul 21 08:41:57 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 21 Jul 2014 07:41:57 -0600 Subject: [petsc-users] MPI-IO In-Reply-To: References: <53CCE0E1.4040504@inria.fr> Message-ID: <871tteltga.fsf@jedbrown.org> Matthew Knepley writes: > On Jul 21, 2014 5:44 AM, "Stephen Wornom" wrote: >> >> I have an unstructured mesh code used to compute vortex shedding problems > saving the solutions every 500-1000 time steps. The mesh size is 3 to 20 > MNodes. The minimum number of cores that I us is 128 for the 3MNode mesh. >> I would like to know if PETSC could be used to use to save the solutions > using MPI-IO? > > The normal VecView() for the binary viewer will use MPI/IO. You need -viewer_binary_mpiio or PetscViewerBinarySetMPIIO(). PETSc devs, do you suppose MPI-IO support is stable enough that we could make this a default? In any case, PetscViewerBinarySetMPIIO should take a PetscBool. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Jul 21 10:49:29 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Jul 2014 10:49:29 -0500 Subject: [petsc-users] MPI-IO In-Reply-To: <871tteltga.fsf@jedbrown.org> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> Message-ID: <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> On Jul 21, 2014, at 8:41 AM, Jed Brown wrote: > Matthew Knepley writes: > >> On Jul 21, 2014 5:44 AM, "Stephen Wornom" wrote: >>> >>> I have an unstructured mesh code used to compute vortex shedding problems >> saving the solutions every 500-1000 time steps. The mesh size is 3 to 20 >> MNodes. The minimum number of cores that I us is 128 for the 3MNode mesh. >>> I would like to know if PETSC could be used to use to save the solutions >> using MPI-IO? >> >> The normal VecView() for the binary viewer will use MPI/IO. > > You need -viewer_binary_mpiio or PetscViewerBinarySetMPIIO(). We should change this to use a PetscViewerFormat instead of special casing it. > > PETSc devs, do you suppose MPI-IO support is stable enough that we could > make this a default? It is not so much a question of stability, it is more a question of (I believe) MPI IO is just loads slower for what might normal PETSc users use, it only pays off for large numbers of nodes. Barry > In any case, PetscViewerBinarySetMPIIO should take > a PetscBool. From jed at jedbrown.org Mon Jul 21 10:55:52 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 21 Jul 2014 09:55:52 -0600 Subject: [petsc-users] MPI-IO In-Reply-To: <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> Message-ID: <87y4vmk8on.fsf@jedbrown.org> Barry Smith writes: > We should change this to use a PetscViewerFormat instead of special casing it. It's not really a "format", but it would be nice to have a common way to address other viewer implementations. >> PETSc devs, do you suppose MPI-IO support is stable enough that we could >> make this a default? > > It is not so much a question of stability, it is more a question > of (I believe) MPI IO is just loads slower for what might normal > PETSc users use, it only pays off for large numbers of nodes. Do we have current data demonstrating that this is still the case? It's really a failure of the implementations if they can't fall back to the dumbest mechanism when it's faster. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Jul 21 11:04:18 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Jul 2014 11:04:18 -0500 Subject: [petsc-users] MPI-IO In-Reply-To: <87y4vmk8on.fsf@jedbrown.org> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> Message-ID: On Jul 21, 2014, at 10:55 AM, Jed Brown wrote: > Barry Smith writes: >> We should change this to use a PetscViewerFormat instead of special casing it. > > It's not really a "format?, Many of them are not ?format?, :-) typedef enum { PETSC_VIEWER_DEFAULT, PETSC_VIEWER_ASCII_MATLAB, PETSC_VIEWER_ASCII_MATHEMATICA, PETSC_VIEWER_ASCII_IMPL, PETSC_VIEWER_ASCII_INFO, PETSC_VIEWER_ASCII_INFO_DETAIL, PETSC_VIEWER_ASCII_COMMON, PETSC_VIEWER_ASCII_SYMMODU, PETSC_VIEWER_ASCII_INDEX, PETSC_VIEWER_ASCII_DENSE, PETSC_VIEWER_ASCII_MATRIXMARKET, PETSC_VIEWER_ASCII_VTK, PETSC_VIEWER_ASCII_VTK_CELL, PETSC_VIEWER_ASCII_VTK_COORDS, PETSC_VIEWER_ASCII_PCICE, PETSC_VIEWER_ASCII_PYTHON, PETSC_VIEWER_ASCII_FACTOR_INFO, PETSC_VIEWER_ASCII_LATEX, PETSC_VIEWER_DRAW_BASIC, PETSC_VIEWER_DRAW_LG, PETSC_VIEWER_DRAW_CONTOUR, PETSC_VIEWER_DRAW_PORTS, PETSC_VIEWER_VTK_VTS, PETSC_VIEWER_VTK_VTR, PETSC_VIEWER_VTK_VTU, PETSC_VIEWER_BINARY_MATLAB, PETSC_VIEWER_NATIVE, PETSC_VIEWER_HDF5_VIZ, PETSC_VIEWER_NOFORMAT } PetscViewerFormat; > but it would be nice to have a common way to > address other viewer implementations. What do you mean by this? > >>> PETSc devs, do you suppose MPI-IO support is stable enough that we could >>> make this a default? >> >> It is not so much a question of stability, it is more a question >> of (I believe) MPI IO is just loads slower for what might normal >> PETSc users use, it only pays off for large numbers of nodes. > > Do we have current data demonstrating that this is still the case? It's > really a failure of the implementations if they can't fall back to the > dumbest mechanism when it's faster. Oh, actually I just remembered what always drove me crazy with MPI IO is that it saves it in native format meaning it is not portable to all systems (which have different endian). Portability before performance :-) From jed at jedbrown.org Mon Jul 21 11:25:22 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 21 Jul 2014 10:25:22 -0600 Subject: [petsc-users] MPI-IO In-Reply-To: References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> Message-ID: <87siluk7bh.fsf@jedbrown.org> Barry Smith writes: >> but it would be nice to have a common way to >> address other viewer implementations. > > What do you mean by this? That we should be able to tell the VTK viewer to use MPI-IO in the same way we tell the Binary viewer or any other viewer capable of using MPI-IO. >>>> PETSc devs, do you suppose MPI-IO support is stable enough that we could >>>> make this a default? >>> >>> It is not so much a question of stability, it is more a question >>> of (I believe) MPI IO is just loads slower for what might normal >>> PETSc users use, it only pays off for large numbers of nodes. >> >> Do we have current data demonstrating that this is still the case? It's >> really a failure of the implementations if they can't fall back to the >> dumbest mechanism when it's faster. > > Oh, actually I just remembered what always drove me crazy with MPI > IO is that it saves it in native format meaning it is not portable > to all systems (which have different endian). Portability before > performance :-) Uh, MPI_Pack_external and MPI_Unpack_external? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Jul 21 11:40:17 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Jul 2014 11:40:17 -0500 Subject: [petsc-users] MPI-IO In-Reply-To: <87siluk7bh.fsf@jedbrown.org> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> <87siluk7bh.fsf@jedbrown.org> Message-ID: <39D00C64-E4B1-44A0-86B0-649E0322C048@mcs.anl.gov> On Jul 21, 2014, at 11:25 AM, Jed Brown wrote: > Barry Smith writes: > >>> but it would be nice to have a common way to >>> address other viewer implementations. >> >> What do you mean by this? > > That we should be able to tell the VTK viewer to use MPI-IO in the same > way we tell the Binary viewer or any other viewer capable of using MPI-IO. Yes the correct PetscViewerFormat is limited in that there is only one supported at a time. We could go the horrible bit oring of options or we could collect multiple options instead of just one. > >>>>> PETSc devs, do you suppose MPI-IO support is stable enough that we could >>>>> make this a default? >>>> >>>> It is not so much a question of stability, it is more a question >>>> of (I believe) MPI IO is just loads slower for what might normal >>>> PETSc users use, it only pays off for large numbers of nodes. >>> >>> Do we have current data demonstrating that this is still the case? It's >>> really a failure of the implementations if they can't fall back to the >>> dumbest mechanism when it's faster. >> >> Oh, actually I just remembered what always drove me crazy with MPI >> IO is that it saves it in native format meaning it is not portable >> to all systems (which have different endian). Portability before >> performance :-) > > Uh, MPI_Pack_external and MPI_Unpack_external? Ok, so maybe the finally implemented them. We don?t use this BTW because Rajev told me (a long time ago) that MPICH doesn?t support it. Barry From jed at jedbrown.org Mon Jul 21 11:50:10 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 21 Jul 2014 10:50:10 -0600 Subject: [petsc-users] MPI-IO In-Reply-To: <39D00C64-E4B1-44A0-86B0-649E0322C048@mcs.anl.gov> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> <87siluk7bh.fsf@jedbrown.org> <39D00C64-E4B1-44A0-86B0-649E0322C048@mcs.anl.gov> Message-ID: <87oawik665.fsf@jedbrown.org> Barry Smith writes: > Yes the correct PetscViewerFormat is limited in that there is only > one supported at a time. We could go the horrible bit oring of > options or we could collect multiple options instead of just one. I think we should collect multiple options. >> Uh, MPI_Pack_external and MPI_Unpack_external? > > Ok, so maybe the finally implemented them. We don?t use this BTW > because Rajev told me (a long time ago) that MPICH doesn?t support > it. Must have been prior to MPI-2.1. Shall we test for MPI_Pack_external and use MPI-IO by default if it's available? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lawrence.mitchell at imperial.ac.uk Mon Jul 21 12:16:17 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 21 Jul 2014 18:16:17 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> Message-ID: To follow up, On 21 Jul 2014, at 13:11, Lawrence Mitchell wrote: > > On 21 Jul 2014, at 12:52, Dave May wrote: > >> >> -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi -mg_levels_ksp_max_it 2 >> >> then I get identical convergence in serial and parallel >> >> >> Good. That's the correct result. >> >> if, however, I run with >> >> -pc_type mg -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it 2 >> (the default according to -ksp_view) >> >> then I get very differing convergence in serial and parallel as described. >> >> >> It's normal that the behaviour is different. The PETSc SOR implementation is not parallel. It only performs SOR on your local subdomain. I think I've convinced myself that I was just getting unlucky with the chebyshev+sor smoothing combination when I looked, modifying the sor weighting, or choosing something other than 2 processes for the particular problem I was seeing bad 2-process convergence on, gives good convergence again. I'll keep an eye on it in case it turns out I have been doing something stupid. Thanks for the various pointers on debugging though. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jed at jedbrown.org Mon Jul 21 12:29:41 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 21 Jul 2014 11:29:41 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> Message-ID: <87k376k4ca.fsf@jedbrown.org> Lawrence Mitchell writes: > Below I show output from a run on 1 process and then two (along with ksp_view) for the following options: > > -pc_type mg -ksp_rtol 1e-8 -ksp_max_it 6 -pc_mg_levels 2 -mg_levels_pc_type sor -ksp_monitor > > On 1 process: > 0 KSP Residual norm 5.865090856053e+02 > 1 KSP Residual norm 1.293159126247e+01 > 2 KSP Residual norm 5.181199296299e-01 > 3 KSP Residual norm 1.268870802643e-02 > 4 KSP Residual norm 5.116058930806e-04 > 5 KSP Residual norm 3.735036960550e-05 > 6 KSP Residual norm 1.755288530515e-06 > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=6, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices How are you sure the rediscretized matrices are correct in parallel? I would stick with the redundant coarse solve and use -mg_levels_ksp_type chebyshev -mg_levels_pc_type jacobi -ksp_monitor_true_residual Use of Jacobi here is to make the smoother the same in parallel as serial. (Usually SOR is a bit stronger, though I think the Cheby/SOR combination is somewhat peculiar and usually overkill.) Compare convergence with and without -pc_mg_galerkin. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From dave.mayhem23 at gmail.com Mon Jul 21 12:33:25 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 21 Jul 2014 19:33:25 +0200 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> Message-ID: Hi Lawrence, I agree that you shouldn't expect magical things to work when using SOR in parallel, but I'm a bit surprised you see such variation for Poisson Take src/ksp/ksp/examples/tutorials/ex28.c for example Running with 1 and 16 cores I get very similar convergence histories mpiexec -n 1 ./ex29 -pc_type mg -pc_mg_levels 3 -da_grid_x 33 -da_grid_y 33 -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_levels_pc_type sor -ksp_type fgmres -ksp_monitor 0 KSP Residual norm 6.680980151738e-03 1 KSP Residual norm 2.600644743629e-04 2 KSP Residual norm 7.722227428855e-06 3 KSP Residual norm 2.001120894208e-07 4 KSP Residual norm 6.663821440723e-09 mpiexec -n 16 ./ex29 -pc_type mg -pc_mg_levels 3 -da_grid_x 33 -da_grid_y 33 -mg_coarse_ksp_type cg -mg_coarse_pc_type jacobi -mg_coarse_ksp_max_it 100 -mg_levels_pc_type sor -ksp_type fgmres -ksp_monitor 0 KSP Residual norm 6.680980151738e-03 1 KSP Residual norm 4.555242291797e-04 2 KSP Residual norm 1.508911073478e-05 3 KSP Residual norm 3.520772689849e-07 4 KSP Residual norm 1.128900788683e-08 On 21 July 2014 19:16, Lawrence Mitchell wrote: > To follow up, > > On 21 Jul 2014, at 13:11, Lawrence Mitchell < > lawrence.mitchell at imperial.ac.uk> wrote: > > > > > On 21 Jul 2014, at 12:52, Dave May wrote: > > > >> > >> -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi > -mg_levels_ksp_max_it 2 > >> > >> then I get identical convergence in serial and parallel > >> > >> > >> Good. That's the correct result. > >> > >> if, however, I run with > >> > >> -pc_type mg -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor > -mg_levels_ksp_max_it 2 > >> (the default according to -ksp_view) > >> > >> then I get very differing convergence in serial and parallel as > described. > >> > >> > >> It's normal that the behaviour is different. The PETSc SOR > implementation is not parallel. It only performs SOR on your local > subdomain. > > I think I've convinced myself that I was just getting unlucky with the > chebyshev+sor smoothing combination when I looked, modifying the sor > weighting, or choosing something other than 2 processes for the particular > problem I was seeing bad 2-process convergence on, gives good convergence > again. > > I'll keep an eye on it in case it turns out I have been doing something > stupid. Thanks for the various pointers on debugging though. > > Cheers, > > Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jul 21 16:32:38 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Jul 2014 16:32:38 -0500 Subject: [petsc-users] MPI-IO In-Reply-To: <87oawik665.fsf@jedbrown.org> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> <87siluk7bh.fsf@jedbrown.org> <39D00C64-E4B1-44A0-86B0-649E0322C048@mcs.anl.gov> <87oawik665.fsf@jedbrown.org> Message-ID: <6153543F-ED7C-459B-9A0E-968B3906E83B@mcs.anl.gov> On Jul 21, 2014, at 11:50 AM, Jed Brown wrote: > Barry Smith writes: >> Yes the correct PetscViewerFormat is limited in that there is only >> one supported at a time. We could go the horrible bit oring of >> options or we could collect multiple options instead of just one. > > I think we should collect multiple options. > >>> Uh, MPI_Pack_external and MPI_Unpack_external? >> >> Ok, so maybe the finally implemented them. We don?t use this BTW >> because Rajev told me (a long time ago) that MPICH doesn?t support >> it. > > Must have been prior to MPI-2.1. Shall we test for MPI_Pack_external > and use MPI-IO by default if it's available? I don?t care From lawrence.mitchell at imperial.ac.uk Tue Jul 22 06:43:56 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 22 Jul 2014 12:43:56 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <87k376k4ca.fsf@jedbrown.org> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> Message-ID: On 21 Jul 2014, at 18:29, Jed Brown wrote: > Lawrence Mitchell writes: > >> Below I show output from a run on 1 process and then two (along with ksp_view) for the following options: >> >> -pc_type mg -ksp_rtol 1e-8 -ksp_max_it 6 -pc_mg_levels 2 -mg_levels_pc_type sor -ksp_monitor >> >> On 1 process: >> 0 KSP Residual norm 5.865090856053e+02 >> 1 KSP Residual norm 1.293159126247e+01 >> 2 KSP Residual norm 5.181199296299e-01 >> 3 KSP Residual norm 1.268870802643e-02 >> 4 KSP Residual norm 5.116058930806e-04 >> 5 KSP Residual norm 3.735036960550e-05 >> 6 KSP Residual norm 1.755288530515e-06 >> KSP Object: 1 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=6, initial guess is zero >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: 1 MPI processes >> type: mg >> MG: type is MULTIPLICATIVE, levels=2 cycles=v >> Cycles per PCApply=1 >> Not using Galerkin computed coarse grid matrices > > How are you sure the rediscretized matrices are correct in parallel? I computed the leading few (10 or so) largest and smallest eigenvalues of the operators on each level, which agree in serial and parallel, so I'm reasonably happy that I'm solving the same problem. > I would stick with the redundant coarse solve and use > > -mg_levels_ksp_type chebyshev -mg_levels_pc_type jacobi -ksp_monitor_true_residual Chebyshev + jacobi appears not to be an effective smoother at all (in serial and parallel). For example, for a two-level cycle: -pc_type mg -ksp_rtol 1e-10 -ksp_max_it 2 -pc_mg_levels 2 -ksp_monitor_true_residual -mg_levels_ksp_type chebyshev -mg_levels_pc_type jacobi -mg_levels_ksp_max_it 3 -mg_levels_ksp_monitor_true_residual Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 1.362115349221e+02 true resid norm 1.602648950381e+02 ||r(i)||/||b|| 5.718551585231e-01 1 KSP none resid norm 3.635392745636e+01 true resid norm 8.483271949491e+01 ||r(i)||/||b|| 3.026990298979e-01 2 KSP none resid norm 2.480718743635e+01 true resid norm 5.297234693113e+01 ||r(i)||/||b|| 1.890152540545e-01 Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 8.050829954680e+00 true resid norm 2.187812818821e+01 ||r(i)||/||b|| 7.806525852269e-02 1 KSP none resid norm 9.600041408511e+00 true resid norm 3.264655957783e+01 ||r(i)||/||b|| 1.164890383398e-01 2 KSP none resid norm 2.246360204969e+01 true resid norm 7.338512212979e+01 ||r(i)||/||b|| 2.618518586918e-01 0 KSP preconditioned resid norm 5.699274467568e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 3.268903240705e-01 true resid norm 7.956741597771e-01 ||r(i)||/||b|| 1.322008721235e+00 1 KSP none resid norm 6.842996420984e-01 true resid norm 2.304432657016e+00 ||r(i)||/||b|| 3.828803578247e+00 2 KSP none resid norm 1.941611552825e+00 true resid norm 6.611568865604e+00 ||r(i)||/||b|| 1.098508930317e+01 Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 1.063939051808e+01 true resid norm 3.664397107500e+01 ||r(i)||/||b|| 6.088377855000e+01 1 KSP none resid norm 3.311047978681e+01 true resid norm 1.157817230412e+02 ||r(i)||/||b|| 1.923707660218e+02 2 KSP none resid norm 9.601250167498e+01 true resid norm 3.374117934071e+02 ||r(i)||/||b|| 5.606080429414e+02 1 KSP preconditioned resid norm 5.693881869205e+02 true resid norm 2.803424438322e+02 ||r(i)||/||b|| 1.000314339708e+00 Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 4.296484723989e+00 true resid norm 1.520508908495e+01 ||r(i)||/||b|| 2.232562919535e+00 1 KSP none resid norm 1.365259124519e+01 true resid norm 4.847930361817e+01 ||r(i)||/||b|| 7.118215159284e+00 2 KSP none resid norm 4.000954569964e+01 true resid norm 1.424890347852e+02 ||r(i)||/||b|| 2.092166206488e+01 Residual norms for mg_levels_1_ solve. 0 KSP none resid norm 2.315970399809e+02 true resid norm 8.307550866976e+02 ||r(i)||/||b|| 1.219797523982e+02 1 KSP none resid norm 7.371848722680e+02 true resid norm 2.669017652749e+03 ||r(i)||/||b|| 3.918918073954e+02 2 KSP none resid norm 2.174699749867e+03 true resid norm 7.893909006175e+03 ||r(i)||/||b|| 1.159062498016e+03 2 KSP preconditioned resid norm 5.682205494231e+02 true resid norm 2.798247056626e+02 ||r(i)||/||b|| 9.984669529615e-01 > Use of Jacobi here is to make the smoother the same in parallel as > serial. If I run the above in parallel I get the same behaviour, (I guess as expected). > (Usually SOR is a bit stronger, though I think the Cheby/SOR > combination is somewhat peculiar and usually overkill.) So I have noticed that I only see this problem of poor convergence on some meshes/decompositions. It also goes away if I apply more than one SOR sweep at each level. For example: -pc_type mg -ksp_rtol 1e-10 -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor_true_residual -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_pc_sor_omega 1 -mg_levels_pc_sor_its 2 produces on 1 process: 0 KSP preconditioned resid norm 5.883693224294e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.309534073571e+01 true resid norm 6.584338590739e+00 ||r(i)||/||b|| 2.349415314990e-02 2 KSP preconditioned resid norm 1.910365687382e-01 true resid norm 1.577734674543e-01 ||r(i)||/||b|| 5.629652783310e-04 3 KSP preconditioned resid norm 3.277350963687e-03 true resid norm 4.094543394403e-03 ||r(i)||/||b|| 1.461009762200e-05 4 KSP preconditioned resid norm 7.348080207899e-05 true resid norm 1.069159047866e-04 ||r(i)||/||b|| 3.814959705670e-07 5 KSP preconditioned resid norm 2.830825689894e-06 true resid norm 4.561049005693e-06 ||r(i)||/||b|| 1.627467700623e-08 6 KSP preconditioned resid norm 4.363171978244e-08 true resid norm 7.242601519032e-08 ||r(i)||/||b|| 2.584295855185e-10 and on 2: 0 KSP preconditioned resid norm 5.836547633061e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.096256903154e+01 true resid norm 6.241073205209e+00 ||r(i)||/||b|| 2.226931797055e-02 2 KSP preconditioned resid norm 1.417636317296e-01 true resid norm 2.236535290250e-01 ||r(i)||/||b|| 7.980376754651e-04 3 KSP preconditioned resid norm 7.600523523911e-03 true resid norm 9.340981660469e-03 ||r(i)||/||b|| 3.333037186303e-05 4 KSP preconditioned resid norm 2.109594208660e-04 true resid norm 3.284856206888e-04 ||r(i)||/||b|| 1.172098210571e-06 5 KSP preconditioned resid norm 4.640884789807e-06 true resid norm 1.425912597007e-05 ||r(i)||/||b|| 5.087923178735e-08 6 KSP preconditioned resid norm 2.250186110144e-07 true resid norm 5.621190025947e-07 ||r(i)||/||b|| 2.005745870056e-09 If I drop the number of sor its to 1, I reduce the residual by 10^8 in serial but 10^3 in parallel, but only on two processes, on 3 and more, I see reductions of around 10^8 as well. > Compare convergence with and without -pc_mg_galerkin. This was a little tricky, since I only have the action of the interpolation and restriction matrices. I coded up a "default" MatMatMatMult using a shell that can compute the appropriate matrix-vector multiply. Since I then don't have an explicit operator on each level, I restricted to a two-level method (where the fine grid operator is assembled). I then run with: -pc_type mg -ksp_rtol 1e-10 -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor_true_residual -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_pc_sor_omega 1 -mg_levels_pc_sor_its 1 -pc_mg_galerkin -mg_coarse_ksp_type cg -mg_coarse_pc_type none -mg_coarse_ksp_max_it 20 On one process I then have: 0 KSP preconditioned resid norm 5.658166234044e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.380224616829e+00 true resid norm 5.942695957168e+00 ||r(i)||/||b|| 2.120465207202e-02 2 KSP preconditioned resid norm 1.076966810659e-01 true resid norm 2.620990648687e-01 ||r(i)||/||b|| 9.352185471038e-04 3 KSP preconditioned resid norm 1.067766492637e-03 true resid norm 6.007393822966e-03 ||r(i)||/||b|| 2.143550617324e-05 4 KSP preconditioned resid norm 3.721153649133e-05 true resid norm 4.831899242645e-03 ||r(i)||/||b|| 1.724112137417e-05 5 KSP preconditioned resid norm 9.011620703273e-07 true resid norm 4.821878295079e-03 ||r(i)||/||b|| 1.720536475662e-05 6 KSP preconditioned resid norm 2.069209381617e-08 true resid norm 4.821642641207e-03 ||r(i)||/||b|| 1.720452389947e-05 On two: 0 KSP preconditioned resid norm 5.662903120832e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.643743888525e+00 true resid norm 1.378172146897e+01 ||r(i)||/||b|| 4.917576312322e-02 2 KSP preconditioned resid norm 6.976625171734e+00 true resid norm 1.228350334724e+01 ||r(i)||/||b|| 4.382984029151e-02 3 KSP preconditioned resid norm 6.830161637428e+00 true resid norm 1.239530026533e+01 ||r(i)||/||b|| 4.422875263161e-02 4 KSP preconditioned resid norm 1.043365955762e+00 true resid norm 7.550284766088e+00 ||r(i)||/||b|| 2.694082999761e-02 5 KSP preconditioned resid norm 7.461488225071e-01 true resid norm 5.937639295074e+00 ||r(i)||/||b|| 2.118660895470e-02 6 KSP preconditioned resid norm 7.370072731475e-01 true resid norm 5.668303344690e+00 ||r(i)||/||b|| 2.022556784482e-02 while on three I go back to decent convergence again: 0 KSP preconditioned resid norm 5.692627905911e+02 true resid norm 2.802543487620e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.768424422846e+00 true resid norm 8.666039768542e+00 ||r(i)||/||b|| 3.092205279534e-02 2 KSP preconditioned resid norm 1.308480914038e-01 true resid norm 1.498332207355e-01 ||r(i)||/||b|| 5.346329910577e-04 3 KSP preconditioned resid norm 1.029363320021e-03 true resid norm 3.954753776706e-03 ||r(i)||/||b|| 1.411130208747e-05 4 KSP preconditioned resid norm 2.495903055408e-05 true resid norm 6.469448501163e-04 ||r(i)||/||b|| 2.308420379466e-06 5 KSP preconditioned resid norm 8.019626928214e-07 true resid norm 6.361612256862e-04 ||r(i)||/||b|| 2.269942388036e-06 6 KSP preconditioned resid norm 1.669414544567e-08 true resid norm 6.360133846808e-04 ||r(i)||/||b|| 2.269414863642e-06 So I'm sort of none-the-wiser. I'm a little bit at a loss as to why this occurs, but either switching to Richardson+SOR or Cheby/SOR with more that one SOR sweep appears to fix the problems, so I might just punt for now. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dalcinl at gmail.com Tue Jul 22 14:23:48 2014 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 22 Jul 2014 16:23:48 -0300 Subject: [petsc-users] MPI-IO In-Reply-To: <87oawik665.fsf@jedbrown.org> References: <53CCE0E1.4040504@inria.fr> <871tteltga.fsf@jedbrown.org> <52150F61-E8FE-43DF-B198-F24D2042D3CC@mcs.anl.gov> <87y4vmk8on.fsf@jedbrown.org> <87siluk7bh.fsf@jedbrown.org> <39D00C64-E4B1-44A0-86B0-649E0322C048@mcs.anl.gov> <87oawik665.fsf@jedbrown.org> Message-ID: On 21 July 2014 13:50, Jed Brown wrote: > > Must have been prior to MPI-2.1. Shall we test for MPI_Pack_external > and use MPI-IO by default if it's available? Be prepared to file tickets for MPICH and OpenMPI. Pack/unpack_external with MPI_DOUBLE never worked for me: https://bitbucket.org/mpi4py/mpi4py/src/master/test/test_pack.py#cl-105 -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169 From blechta at karlin.mff.cuni.cz Wed Jul 23 05:51:18 2014 From: blechta at karlin.mff.cuni.cz (Jan Blechta) Date: Wed, 23 Jul 2014 12:51:18 +0200 Subject: [petsc-users] fieldsplit/richardson/gamg segfaults Message-ID: <20140723125118.2ff2ae8f@gott> Using RICHARDSON/GAMG as FIELDSPLIT subKSP results in a segfault on a second solve with backtrace like #1 0x00007fffe8da6c5b in MatResidual (mat=0x0, b=0x2a85e00, x=0x29c3360, r=0x4c56590) #2 0x00007fffe9765c1d in PCMGResidualDefault (mat=0x0, b=0x2a85e00, x=0x29c3360, r=0x4c56590) #3 0x00007fffe97552df in PCMGMCycle_Private (pc=0x2861520, mglevelsin=0x3d6dba0, reason=0x7fffffff8380) #4 0x00007fffe97567a1 in PCApplyRichardson_MG (pc=0x2861520, b=0x2a85e00, x=0x29c3360, w=0x498fd80, rtol=1.0000000000000001e-05, abstol=1e-50, dtol=10000, its=1, zeroguess=PETSC_TRUE, outits=0x285f474, reason=0x7fffffff8380) #5 0x00007fffe95e771b in PCApplyRichardson (pc=0x2861520, b=0x2a85e00, y=0x29c3360, w=0x498fd80, rtol=1.0000000000000001e-05, abstol=1e-50, dtol=10000, its=1, guesszero=PETSC_TRUE, outits=0x285f474, reason=0x7fffffff8380) #6 0x00007fffe97f0206 in KSPSolve_Richardson (ksp=0x285f0e0) ... Notice null mat on frame 1 and 2. This can be reproduced by adding -fieldsplit_phi_ksp_type richardson to runex73f90t target in src/snes/examples/tutorials/makefile on PETSc 3.5 release. Jan From jed at jedbrown.org Wed Jul 23 10:41:23 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 23 Jul 2014 09:41:23 -0600 Subject: [petsc-users] AGU Session: State of the Art in Computational Geoscience Message-ID: <87ha28axr0.fsf@jedbrown.org> If you are thinking about attending the American Geophysical Union Fall Meeting (Dec 15-19 in San Francisco), please consider submitting an abstract to this diverse session. State of the Art in Computational Geoscience Session Description: This session highlights computational advances in areas such as lithospheric and mantle dynamics, magma and fluid transport, landscape evolution, polar ice, subsurface flow, Earth structure inversion and Earth material properties. We seek contributions from all aspects of geophysical computation including: accurate, robust multiscale discretizations and efficient, scalable solvers; multilevel, block decomposed, and structure-preserving representations of operators, addressing model non-smoothness, and utilizing next generation hardware; dissemination of efficient/flexible/performant open source packages for the Earth science community; data assimilation with uncertainty and experimental design; application studies within Earth science utilizing High Performance Computing. Conveners: * Jed Brown (ANL and CU Boulder) * Dave May (ETH Z?rich) * Boris Kaus (University Mainz) * Richard Mills (Intel) https://agu.confex.com/agu/fm14/webprogrampreliminary/Session2730.html Abstracts are due August 6. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hus003 at ucsd.edu Wed Jul 23 12:36:02 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Wed, 23 Jul 2014 17:36:02 +0000 Subject: [petsc-users] why a certain option cannot be used In-Reply-To: <877g3ar37n.fsf@jedbrown.org> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CBFA3@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFB6@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFDB@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CBFF2@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC008@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC034@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB6CC06C@XMAIL-MBX-BH1.AD.UCSD.EDU> <877g3cwomy.fsf@jedbrown.org> <750 1CC2B7BBCC44A92ECEEC316170ECB6CC0B5@XMAIL-MBX-BH1.AD.UCSD.EDU>, <877g3ar37n.fsf@jedbrown.org> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC1F8@XMAIL-MBX-BH1.AD.UCSD.EDU> Dear Jed, Dave, Matt and others, I just want to let you know that I have successfully applied fieldsplit and Schur complement into the project I'm working on. The code I start up with is ex70.c under SNES/examples/tutorial. Thank you all for your kindly help. Best, Hui ________________________________________ From: Jed Brown [jed at jedbrown.org] Sent: Friday, July 18, 2014 10:23 AM To: Sun, Hui; Dave May Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] why a certain option cannot be used "Sun, Hui" writes: > Thank you Jed. After reading ex15, I got some idea of how to define the PCShell with KSP solver. However, I want to verify with you if the following procedure is normally what one should do. > 1. Define a ShellPC struct, which includes a MAT object, say Pmat, and a KSP object, say innerKsp. The context should indeed contain innerKSP. The ShellPC may not need a Mat (you can just pass it down to innerKSP). > 2. PCSetType(pc, PCSHELL) > 3. Create a context for the user defined PC. It is here that we call KSPSetOperators(innerKsp, Pmat, Pmat) You can do it eagerly or via PCShellSetSetUp(), if you want the shell PC to be more reusable/independent. > 4. Set the user defined routine for applying the preconditioner. It is in this user-define routine that we call KSPSolver(innerKsp, r, y) > > I'm wondering the way I set up the inner KSP for the PC is standard? Thank you! Pretty much. From qiyuelu1 at gmail.com Wed Jul 23 14:53:09 2014 From: qiyuelu1 at gmail.com (Qiyue Lu) Date: Wed, 23 Jul 2014 14:53:09 -0500 Subject: [petsc-users] PETSc Binary format to Matrix Market Message-ID: Dear All: My collaborators sent me some PETSc binary format files(matrix, RHS) with an convertor code which converts these binary file to Matrix Market. They did tests on their machine and it works. However, when I compile this convertor code and running it with their PETSc binary format matrix file, I got a message like: [0]PETSC ERROR: Unexpected data in file [0]PETSC ERROR: not matrix object Because there is no error in convertor code compilation and we are using the same version of PETSc, I am wondering, Is this due to 'binary format compatible problem', the PETSc binary matrix file is hardware-platform dependent? Thanks Qiyue Lu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jul 23 15:05:29 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 23 Jul 2014 15:05:29 -0500 Subject: [petsc-users] PETSc Binary format to Matrix Market In-Reply-To: References: Message-ID: <2CE9DFB9-8CF9-4F42-B3A9-F57A04E3D53D@mcs.anl.gov> On Jul 23, 2014, at 2:53 PM, Qiyue Lu wrote: > Dear All: > > My collaborators sent me some PETSc binary format files(matrix, RHS) with an convertor code which converts these binary file to Matrix Market. They did tests on their machine and it works. > > However, when I compile this convertor code and running it with their PETSc binary format matrix file, I got a message like: > [0]PETSC ERROR: Unexpected data in file > [0]PETSC ERROR: not matrix object > > > Because there is no error in convertor code compilation and we are using the same version of PETSc, I am wondering, Is this due to 'binary format compatible problem', the PETSc binary matrix file is hardware-platform dependent? It is suppose to be binary compatible on all machines. But you need the same PETSc numerical precision single, double, __float128, and the same PETSc index size 32 bit or 64 bit and the same real or complex Are both PETSc?s configured the same way? Barry > > Thanks > > Qiyue Lu From bsmith at mcs.anl.gov Wed Jul 23 16:35:53 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 23 Jul 2014 16:35:53 -0500 Subject: [petsc-users] fieldsplit/richardson/gamg segfaults In-Reply-To: <20140723125118.2ff2ae8f@gott> References: <20140723125118.2ff2ae8f@gott> Message-ID: <57B1A9B2-7A90-4A5E-921A-4F312719C33A@mcs.anl.gov> Jan, Thanks for reporting this. It was actually more generally and affected Richardson with GAMG always. Some refactoring of PCMG did not get completed for the Richardson case. This is now fixed in the maint, master, and next branch and will be in our next patch release. Barry ~/Src/petsc/src/snes/examples/tutorials maint $ git commit -a [maint a762d67] PCApplyRichardson_MG() did not properly set operators for all levels for residual computation Needed the same code as PCApply_MG() had 1 file changed, 7 insertions(+) On Jul 23, 2014, at 5:51 AM, Jan Blechta wrote: > Using RICHARDSON/GAMG as FIELDSPLIT subKSP results in a segfault on a > second solve with backtrace like > > #1 0x00007fffe8da6c5b in MatResidual (mat=0x0, b=0x2a85e00, x=0x29c3360, r=0x4c56590) > #2 0x00007fffe9765c1d in PCMGResidualDefault (mat=0x0, b=0x2a85e00, x=0x29c3360, r=0x4c56590) > #3 0x00007fffe97552df in PCMGMCycle_Private (pc=0x2861520, mglevelsin=0x3d6dba0, reason=0x7fffffff8380) > #4 0x00007fffe97567a1 in PCApplyRichardson_MG (pc=0x2861520, b=0x2a85e00, x=0x29c3360, w=0x498fd80, > rtol=1.0000000000000001e-05, abstol=1e-50, dtol=10000, its=1, zeroguess=PETSC_TRUE, outits=0x285f474, > reason=0x7fffffff8380) > #5 0x00007fffe95e771b in PCApplyRichardson (pc=0x2861520, b=0x2a85e00, y=0x29c3360, w=0x498fd80, > rtol=1.0000000000000001e-05, abstol=1e-50, dtol=10000, its=1, guesszero=PETSC_TRUE, outits=0x285f474, > reason=0x7fffffff8380) > #6 0x00007fffe97f0206 in KSPSolve_Richardson (ksp=0x285f0e0) > ... > > Notice null mat on frame 1 and 2. This can be reproduced by adding > > -fieldsplit_phi_ksp_type richardson > > to runex73f90t target in src/snes/examples/tutorials/makefile on PETSc 3.5 release. > > Jan From balay at mcs.anl.gov Thu Jul 24 11:47:13 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 24 Jul 2014 11:47:13 -0500 Subject: [petsc-users] petsc-3.5.1.tar.gz now available Message-ID: Dear PETSc users, The patch release petsc-3.5.1 is now available for download. http://www.mcs.anl.gov/petsc/download/index.html Some of the changes include: * PCApplyRichardson_MG() did not properly set operators for all levels for residual computation Needed the same code as PCApply_MG() had * if vector is of length 0 then VecSet() should set all norms to zero instead of triggering a divide by zero * allow complex numbers to be used from both C and C++ (when supported) not only the language PETSc was built with * improve the memory usage of PCGAMGCreateGraph() * did not mirror 1d DMDA values correctly due to previous change consulidating block size -options_table should have been -options_view twenty years ago * mpi4py: use current tarball mpi4py-1.3.1.tar.gz * bugfix for memory leak with >1 processes and calling KSPSetOperators(A) twice with same A (see Jose Roman's email on Jul 11, 2014) * At configure time determine if MPICH_NUMVERSION or OMPI_MAJOR/MINOR_VERSION or MPI-Uni, then at compile time make sure versions match to determine if incorrect mpi.h is being included * petsc4py.py: look for python dynamic library in correct locations on Apple based on python version Test did not work correctly for anaconda installed python on Apple since that does not install a generic libpython.dylib * Added code to properly preallocate "diagonal" portion of scalar matrix derived from vector matrix in PCGAMGCreateGraph() * KSPCreate() for TAO GPCG solver is now done at TaoCreate_GPCG() instead of TaoSetUp() this allows users to change KSP options from the code easily and matches the style of other Tao solvers that use KSP * TAO: Enable interface for complex scalar builds Satish From ksong at lbl.gov Thu Jul 24 17:09:31 2014 From: ksong at lbl.gov (Kai Song) Date: Thu, 24 Jul 2014 17:09:31 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed Message-ID: Hi Petsc community, I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and openmpi/1.6.5 The program built and installed fine, but make test failed on the following error: ================ ... mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o ex19 ex19.o -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl -lutil -lgcc_s -lpthread -ldl /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_aij_mumps' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_baij_mumps' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatPartitioningCreate_Parmetis' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_sbaij_mumps' collect2: ld returned 1 exit status gmake[3]: [ex19] Error 1 (ignored) ... ================ I used the following configure line: ./configure --prefix=/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --download-scalapack --download-metis=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ Is there anything that I am missing in the built process? Any help would be great appreciated! Thanks in advance! Kai -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jul 24 17:17:29 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 24 Jul 2014 17:17:29 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: Perhaps there were errors during the build. Can you send us make.log for this build? Satish On Thu, 24 Jul 2014, Kai Song wrote: > Hi Petsc community, > > I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and > openmpi/1.6.5 > > The program built and installed fine, but make test failed on the following > error: > ================ > ... > mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 > -O0 -o ex19 ex19.o > -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc > -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack > -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread > -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lgfortran -lm > -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl -lutil > -lgcc_s -lpthread -ldl > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > undefined reference to `MatGetFactor_aij_mumps' > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > undefined reference to `MatGetFactor_baij_mumps' > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > undefined reference to `MatPartitioningCreate_Parmetis' > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > undefined reference to `MatGetFactor_sbaij_mumps' > collect2: ld returned 1 exit status > gmake[3]: [ex19] Error 1 (ignored) > ... > ================ > > I used the following configure line: > ./configure --prefix=/software/modules/petsc/3.5.0 > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 > --download-scalapack --download-metis=1 > --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ > > Is there anything that I am missing in the built process? Any help would be > great appreciated! > > Thanks in advance! > > Kai > > From ksong at lbl.gov Thu Jul 24 17:33:10 2014 From: ksong at lbl.gov (Kai Song) Date: Thu, 24 Jul 2014 17:33:10 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: Hi Satish, Thanks for the quick response. I attached the make.log in this email, and a quick glance, I didn't see any obvious errors. Please let me know if you need any additional informations. Thanks, Kai On Thu, Jul 24, 2014 at 5:17 PM, Satish Balay wrote: > Perhaps there were errors during the build. Can you send us make.log > for this build? > > Satish > > On Thu, 24 Jul 2014, Kai Song wrote: > > > Hi Petsc community, > > > > I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and > > openmpi/1.6.5 > > > > The program built and installed fine, but make test failed on the > following > > error: > > ================ > > ... > > mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 > > -O0 -o ex19 ex19.o > > -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc > > -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib > > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack > > -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread > > -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib > > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lgfortran > -lm > > -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl -lutil > > -lgcc_s -lpthread -ldl > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > undefined reference to `MatGetFactor_aij_mumps' > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > undefined reference to `MatGetFactor_baij_mumps' > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > undefined reference to `MatPartitioningCreate_Parmetis' > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > undefined reference to `MatGetFactor_sbaij_mumps' > > collect2: ld returned 1 exit status > > gmake[3]: [ex19] Error 1 (ignored) > > ... > > ================ > > > > I used the following configure line: > > ./configure --prefix=/software/modules/petsc/3.5.0 > > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 > > --download-scalapack --download-metis=1 > > > --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ > > > > Is there anything that I am missing in the built process? Any help would > be > > great appreciated! > > > > Thanks in advance! > > > > Kai > > > > > > -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 90819 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jul 24 17:38:59 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 24 Jul 2014 17:38:59 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: Try: cd /clusterfs/voltaire/home/software/source/petsc-3.5.0 rm -rf /clusterfs/voltaire/home/software/modules/petsc/3.5.0 make PETSC_DIR=/clusterfs/voltaire/home/software/source/petsc-3.5.0 PETSC_ARCH=arch-linux2-c-debug install The previous build was with [default] --with-shared-libraries=1 - but the current built is using --with-shared-libraries=0. So the libraries installed at the prefix location are inconsistant. Satish On Thu, 24 Jul 2014, Kai Song wrote: > Hi Satish, > > Thanks for the quick response. I attached the make.log in this email, and a > quick glance, I didn't see any obvious errors. > > Please let me know if you need any additional informations. > > Thanks, > > Kai > > > > On Thu, Jul 24, 2014 at 5:17 PM, Satish Balay wrote: > > > Perhaps there were errors during the build. Can you send us make.log > > for this build? > > > > Satish > > > > On Thu, 24 Jul 2014, Kai Song wrote: > > > > > Hi Petsc community, > > > > > > I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and > > > openmpi/1.6.5 > > > > > > The program built and installed fine, but make test failed on the > > following > > > error: > > > ================ > > > ... > > > mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 > > > -O0 -o ex19 ex19.o > > > -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc > > > -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib > > > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack > > > -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread > > > -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib > > > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lgfortran > > -lm > > > -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl -lutil > > > -lgcc_s -lpthread -ldl > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > undefined reference to `MatGetFactor_aij_mumps' > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > undefined reference to `MatGetFactor_baij_mumps' > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > undefined reference to `MatPartitioningCreate_Parmetis' > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > undefined reference to `MatGetFactor_sbaij_mumps' > > > collect2: ld returned 1 exit status > > > gmake[3]: [ex19] Error 1 (ignored) > > > ... > > > ================ > > > > > > I used the following configure line: > > > ./configure --prefix=/software/modules/petsc/3.5.0 > > > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 > > > --download-scalapack --download-metis=1 > > > > > --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ > > > > > > Is there anything that I am missing in the built process? Any help would > > be > > > great appreciated! > > > > > > Thanks in advance! > > > > > > Kai > > > > > > > > > > > > > From ksong at lbl.gov Thu Jul 24 17:48:22 2014 From: ksong at lbl.gov (Kai Song) Date: Thu, 24 Jul 2014 17:48:22 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: Hi Satish, I actually changed it to "--with-shared-libraries=0" as my first attempt to fix. I just tried as you suggested, and I am still seeing the error: ... /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_aij_mumps' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_baij_mumps' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatPartitioningCreate_Parmetis' /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: undefined reference to `MatGetFactor_sbaij_mumps' .... Is there anything else that I should try? Thanks! Kai On Thu, Jul 24, 2014 at 5:38 PM, Satish Balay wrote: > Try: > > cd /clusterfs/voltaire/home/software/source/petsc-3.5.0 > rm -rf /clusterfs/voltaire/home/software/modules/petsc/3.5.0 > make PETSC_DIR=/clusterfs/voltaire/home/software/source/petsc-3.5.0 > PETSC_ARCH=arch-linux2-c-debug install > > > The previous build was with [default] --with-shared-libraries=1 - but > the current built is using --with-shared-libraries=0. So the libraries > installed at the prefix location are inconsistant. > > Satish > > On Thu, 24 Jul 2014, Kai Song wrote: > > > Hi Satish, > > > > Thanks for the quick response. I attached the make.log in this email, > and a > > quick glance, I didn't see any obvious errors. > > > > Please let me know if you need any additional informations. > > > > Thanks, > > > > Kai > > > > > > > > On Thu, Jul 24, 2014 at 5:17 PM, Satish Balay wrote: > > > > > Perhaps there were errors during the build. Can you send us make.log > > > for this build? > > > > > > Satish > > > > > > On Thu, 24 Jul 2014, Kai Song wrote: > > > > > > > Hi Petsc community, > > > > > > > > I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and > > > > openmpi/1.6.5 > > > > > > > > The program built and installed fine, but make test failed on the > > > following > > > > error: > > > > ================ > > > > ... > > > > mpicc -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -g3 > > > > -O0 -o ex19 ex19.o > > > > -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc > > > > -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib > > > > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack > > > > -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread > > > > > -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib > > > > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 > -lgfortran > > > -lm > > > > -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl > -lutil > > > > -lgcc_s -lpthread -ldl > > > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > > undefined reference to `MatGetFactor_aij_mumps' > > > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > > undefined reference to `MatGetFactor_baij_mumps' > > > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > > undefined reference to `MatPartitioningCreate_Parmetis' > > > > > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: > > > > undefined reference to `MatGetFactor_sbaij_mumps' > > > > collect2: ld returned 1 exit status > > > > gmake[3]: [ex19] Error 1 (ignored) > > > > ... > > > > ================ > > > > > > > > I used the following configure line: > > > > ./configure --prefix=/software/modules/petsc/3.5.0 > > > > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 > > > > --download-scalapack --download-metis=1 > > > > > > > > --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ > > > > > > > > Is there anything that I am missing in the built process? Any help > would > > > be > > > > great appreciated! > > > > > > > > Thanks in advance! > > > > > > > > Kai > > > > > > > > > > > > > > > > > > > > > > -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 25 04:48:35 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 25 Jul 2014 04:48:35 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > Hi Satish, > > Thanks for the quick response. I attached the make.log in this email, and > a quick glance, I didn't see any obvious errors. > This never built the MUMPS directory, even though you have PETSC_HAVE_MUMPS defined. I can see two possible reasons for this: 1) You distribution as missing files 2) You have two copies of PETSc fighting each other here (old PETSC_DIR?) Can you make sure you have the file mumps.c in src/mat/impls/aij/mpi/mumps Jed, how do we check what make is doing here? Matt > Please let me know if you need any additional informations. > > Thanks, > > Kai > > > > On Thu, Jul 24, 2014 at 5:17 PM, Satish Balay wrote: > >> Perhaps there were errors during the build. Can you send us make.log >> for this build? >> >> Satish >> >> On Thu, 24 Jul 2014, Kai Song wrote: >> >> > Hi Petsc community, >> > >> > I just installed petsc/3.5.0 on our RHEL-6 system, with gcc/4.4.7 and >> > openmpi/1.6.5 >> > >> > The program built and installed fine, but make test failed on the >> following >> > error: >> > ================ >> > ... >> > mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -g3 >> > -O0 -o ex19 ex19.o >> > -L/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib -lpetsc >> > -Wl,-rpath,/clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib >> > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack >> > -lflapack -lfblas -lparmetis -lmetis -lX11 -lssl -lcrypto -lpthread >> > -L/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/lib >> > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lgfortran >> -lm >> > -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnuma -lrt -lnsl >> -lutil >> > -lgcc_s -lpthread -ldl >> > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: >> > undefined reference to `MatGetFactor_aij_mumps' >> > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: >> > undefined reference to `MatGetFactor_baij_mumps' >> > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: >> > undefined reference to `MatPartitioningCreate_Parmetis' >> > /clusterfs/voltaire/home/software/modules/petsc/3.5.0/lib/libpetsc.so: >> > undefined reference to `MatGetFactor_sbaij_mumps' >> > collect2: ld returned 1 exit status >> > gmake[3]: [ex19] Error 1 (ignored) >> > ... >> > ================ >> > >> > I used the following configure line: >> > ./configure --prefix=/software/modules/petsc/3.5.0 >> > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 >> > --download-scalapack --download-metis=1 >> > >> --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ >> > >> > Is there anything that I am missing in the built process? Any help >> would be >> > great appreciated! >> > >> > Thanks in advance! >> > >> > Kai >> > >> > >> >> > > > -- > Kai Song > 1.510.495.2180 > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > High Performance Computing Services (HPCS) > Lawrence Berkeley National Laboratory - http://scs.lbl.gov > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 25 11:00:43 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 25 Jul 2014 10:00:43 -0600 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: Message-ID: <87y4vh77is.fsf@jedbrown.org> Matthew Knepley writes: > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > >> Hi Satish, >> >> Thanks for the quick response. I attached the make.log in this email, and >> a quick glance, I didn't see any obvious errors. >> > > This never built the MUMPS directory, even though you have PETSC_HAVE_MUMPS > defined. I can see two > possible reasons for this: > > 1) You distribution as missing files > > 2) You have two copies of PETSc fighting each other here (old PETSC_DIR?) > > Can you make sure you have the file mumps.c in > > src/mat/impls/aij/mpi/mumps > > Jed, how do we check what make is doing here? mumps.c should be listed in $PETSC_ARCH/conf/files after the code generator runs. Possibly a file system problem or tinkering caused it to appear up-to-date after reconfiguring? (Just speculating because we haven't seen this problem before.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From balay at mcs.anl.gov Fri Jul 25 11:06:53 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 25 Jul 2014 11:06:53 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: <87y4vh77is.fsf@jedbrown.org> References: <87y4vh77is.fsf@jedbrown.org> Message-ID: Its probably best to do a fresh build with the latest petsc-3.5.1 tarball and see if the problem is reproduceable [with --with-shared-libraries=0] Satish On Fri, 25 Jul 2014, Jed Brown wrote: > Matthew Knepley writes: > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > >> Hi Satish, > >> > >> Thanks for the quick response. I attached the make.log in this email, and > >> a quick glance, I didn't see any obvious errors. > >> > > > > This never built the MUMPS directory, even though you have PETSC_HAVE_MUMPS > > defined. I can see two > > possible reasons for this: > > > > 1) You distribution as missing files > > > > 2) You have two copies of PETSc fighting each other here (old PETSC_DIR?) > > > > Can you make sure you have the file mumps.c in > > > > src/mat/impls/aij/mpi/mumps > > > > Jed, how do we check what make is doing here? > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > generator runs. Possibly a file system problem or tinkering caused it > to appear up-to-date after reconfiguring? (Just speculating because we > haven't seen this problem before.) > From ksong at lbl.gov Fri Jul 25 12:23:46 2014 From: ksong at lbl.gov (Kai Song) Date: Fri, 25 Jul 2014 12:23:46 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: <87y4vh77is.fsf@jedbrown.org> Message-ID: Hi All, Thanks for all the suggestions! I took Satish's advice, removed the source, and did a fresh built. It got through the test compiling, but the tests failed as follow: ================== Running test examples to verify correct installation Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 and PETSC_ARCH=arch-linux2-c-debug Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html ... ================== I tried to go into src/snes/examples/tutorials/ex19, but didn't find any logs. Is this error for the test Okay? Thanks, Kai On Fri, Jul 25, 2014 at 11:06 AM, Satish Balay wrote: > Its probably best to do a fresh build with the latest petsc-3.5.1 tarball > and see > if the problem is reproduceable [with --with-shared-libraries=0] > > Satish > > On Fri, 25 Jul 2014, Jed Brown wrote: > > > Matthew Knepley writes: > > > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > > > >> Hi Satish, > > >> > > >> Thanks for the quick response. I attached the make.log in this email, > and > > >> a quick glance, I didn't see any obvious errors. > > >> > > > > > > This never built the MUMPS directory, even though you have > PETSC_HAVE_MUMPS > > > defined. I can see two > > > possible reasons for this: > > > > > > 1) You distribution as missing files > > > > > > 2) You have two copies of PETSc fighting each other here (old > PETSC_DIR?) > > > > > > Can you make sure you have the file mumps.c in > > > > > > src/mat/impls/aij/mpi/mumps > > > > > > Jed, how do we check what make is doing here? > > > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > > generator runs. Possibly a file system problem or tinkering caused it > > to appear up-to-date after reconfiguring? (Just speculating because we > > haven't seen this problem before.) > > > > -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jul 25 13:32:48 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 25 Jul 2014 13:32:48 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: <87y4vh77is.fsf@jedbrown.org> Message-ID: On Jul 25, 2014, at 12:23 PM, Kai Song wrote: > Hi All, > > Thanks for all the suggestions! I took Satish's advice, removed the source, and did a fresh built. > > It got through the test compiling, but the tests failed as follow: > ================== > Running test examples to verify correct installation > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 and PETSC_ARCH=arch-linux2-c-debug > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > ? Surely it did not print ? here? What was the complete message? Usually this happens due to an incompatible mpiexec being used because it appears earlier in the path. What happens if you do cd src/snes/examples/tutorials make ex19 mpiexec -n 2 ./ex19 for mpiexec use the one associated with the MPI that you built PETSc with Barry > ================== > > I tried to go into src/snes/examples/tutorials/ex19, but didn't find any logs. Is this error for the test Okay? > > Thanks, > > Kai > > > > On Fri, Jul 25, 2014 at 11:06 AM, Satish Balay wrote: > Its probably best to do a fresh build with the latest petsc-3.5.1 tarball and see > if the problem is reproduceable [with --with-shared-libraries=0] > > Satish > > On Fri, 25 Jul 2014, Jed Brown wrote: > > > Matthew Knepley writes: > > > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > > > >> Hi Satish, > > >> > > >> Thanks for the quick response. I attached the make.log in this email, and > > >> a quick glance, I didn't see any obvious errors. > > >> > > > > > > This never built the MUMPS directory, even though you have PETSC_HAVE_MUMPS > > > defined. I can see two > > > possible reasons for this: > > > > > > 1) You distribution as missing files > > > > > > 2) You have two copies of PETSc fighting each other here (old PETSC_DIR?) > > > > > > Can you make sure you have the file mumps.c in > > > > > > src/mat/impls/aij/mpi/mumps > > > > > > Jed, how do we check what make is doing here? > > > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > > generator runs. Possibly a file system problem or tinkering caused it > > to appear up-to-date after reconfiguring? (Just speculating because we > > haven't seen this problem before.) > > > > > > > -- > Kai Song > 1.510.495.2180 > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > High Performance Computing Services (HPCS) > Lawrence Berkeley National Laboratory - http://scs.lbl.gov From tlk0812 at hotmail.com Fri Jul 25 13:48:07 2014 From: tlk0812 at hotmail.com (LikunTan) Date: Sat, 26 Jul 2014 02:48:07 +0800 Subject: [petsc-users] parallel 3D convolution Message-ID: Dear all, I am trying to use FFT to calculate the demagnetizing field in micromagnetics, which is of a discrete convolution form, Hdemag(i,j,k)=\sum_{p=0}^Nx\sum_{q=0}^Ny\sum_{r=0}^Nz DemagTensor(i-p,j-q,k-r)aM(p,q,r) I wrote a subroutine based on the examples ex121.c(sequential convolution) and ex158.c (parallel FFT) in Petsc website, but the result is inconsistent with the one from direct method. I've attached my code. My concern is in the implementation of zero-padding and wrap-around order. I would very much appreciate your time to take a look and direct me to any possible problems. regards, Likun -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: parallel 3D convolution.pdf Type: application/pdf Size: 61377 bytes Desc: not available URL: From zinlin.zinlin at gmail.com Fri Jul 25 14:57:38 2014 From: zinlin.zinlin at gmail.com (Zin Lin) Date: Fri, 25 Jul 2014 15:57:38 -0400 Subject: [petsc-users] Problem with MATSOLVERPASTIX using KSPSolveTranspose Message-ID: Hi, I am having trouble calling KSPSolveTranspose under MATSOLVERPASTIX. The following is a very simple C code solving A^T x = b to illustrate the issue. The matrix A and vector b are built following the PETSC tutorial. KSPSolve works fine but when it comes to KSPSolveTranspose, it spits out the error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type mpiaij! The code runs fine if I change the solver package MATSOLVERPASTIX to MATSOLVERMUMPS. However, I need the PASTIX solver and the transpose solution for solving a large problem. Could you please let me know what is wrong with the PASTIX solver or with my usage? Best regards, Zin /******************/ #include #include #include #include int main(int argc, char **argv) { /* -------Initialize ------*/ PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); PetscPrintf(PETSC_COMM_WORLD,"--------Initializing------ \n"); PetscErrorCode ierr; Vec x,y; Mat A; int N=10; int low, high, i; double val; int start, end; double v[3]; int row, cols[3]; ierr = VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE,N, &x);CHKERRQ(ierr); ierr = VecDuplicate(x,&y); CHKERRQ(ierr); VecGetOwnershipRange(x, &low, &high); val = low*10.0; for(i = low; i < high; ++i) { VecSetValues(x, 1, &i, &val, INSERT_VALUES); val += 10.0; } VecAssemblyBegin(x); VecAssemblyEnd(x); MatCreate(PETSC_COMM_WORLD, &A); MatSetType(A,MATMPIAIJ); MatSetSizes(A,PETSC_DECIDE, PETSC_DECIDE, N, N); MatMPIAIJSetPreallocation(A, 3, PETSC_NULL, 3, PETSC_NULL); v[0] = -1.0; v[1] = 2.0; v[2] = -1.0; MatGetOwnershipRange(A,&start,&end); for(row = start; row < end; row++) { cols[0] = row-1; cols[1] = row; cols[2] = row+1; if (row == 0) { MatSetValues(A,1,&row,2,&cols[1],&v[1],INSERT_VALUES); } else if (row == N-1) { MatSetValues(A,1,&row,2,cols,v,INSERT_VALUES); } else { MatSetValues(A,1,&row,3,cols,v,INSERT_VALUES); } } MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); PetscPrintf(PETSC_COMM_WORLD,"A and x setup. \n"); KSP ksp; PC pc; ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc,PCLU);CHKERRQ(ierr); ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERPASTIX);CHKERRQ(ierr); ierr = PCSetFromOptions(pc); ierr = KSPSetTolerances(ksp,1e-14,PETSC_DEFAULT,PETSC_DEFAULT,20);CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSolve(ksp,x,y);CHKERRQ(ierr); double p; ierr = VecDot(x,y,&p); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"x dot y: %g. \n", p); Vec u; ierr = VecDuplicate(x,&u); CHKERRQ(ierr); ierr = KSPSolveTranspose(ksp,y,u);CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"Transpose solved. \n"); ierr = VecDestroy(&x); CHKERRQ(ierr); ierr = VecDestroy(&y); CHKERRQ(ierr); ierr = MatDestroy(&A); CHKERRQ(ierr); ierr = KSPDestroy(&ksp); CHKERRQ(ierr); { int rank; MPI_Comm_rank(PETSC_COMM_WORLD, &rank); MPI_Barrier(PETSC_COMM_WORLD); } ierr = PetscFinalize(); CHKERRQ(ierr); return 0; } /******************/ -- Zin Lin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ksong at lbl.gov Fri Jul 25 15:23:14 2014 From: ksong at lbl.gov (Kai Song) Date: Fri, 25 Jul 2014 15:23:14 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: <87y4vh77is.fsf@jedbrown.org> Message-ID: Hi Barry, I indeed deleted some of the OpenMPI related warning messages, it should be irrelevant. Here is the full output: ====================== [kaisong at n0009 petsc-3.5.0]$ sh petsc-3.5.0-gcc.sh Running test examples to verify correct installation Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 and PETSC_ARCH=arch-linux2-c-debug Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html -------------------------------------------------------------------------- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: n0009.scs00 -------------------------------------------------------------------------- lid velocity = 0.0016, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html -------------------------------------------------------------------------- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: n0009.scs00 -------------------------------------------------------------------------- lid velocity = 0.0016, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 [n0009.scs00:07638] 1 more process has sent help message help-mpi-btl-openib.txt / no active ports found [n0009.scs00:07638] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html -------------------------------------------------------------------------- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: n0009.scs00 -------------------------------------------------------------------------- Number of SNES iterations = 4 Completed test examples ========================================= Now to evaluate the computer systems you plan use - do: make PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 PETSC_ARCH=arch-linux2-c-debug streams NPMAX= ====================== I also followed your suggestion and built ex19 separately, and made sure using the same compiler that built petsc. Here is the complete output: ====================== [kaisong at n0009 tutorials]$ mpiexec -n 2 ./ex19 -------------------------------------------------------------------------- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: n0009.scs00 -------------------------------------------------------------------------- lid velocity = 0.0625, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 [n0009.scs00:09124] 1 more process has sent help message help-mpi-btl-openib.txt / no active ports found [n0009.scs00:09124] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ====================== Thanks, Kai On Fri, Jul 25, 2014 at 1:32 PM, Barry Smith wrote: > > On Jul 25, 2014, at 12:23 PM, Kai Song wrote: > > > Hi All, > > > > Thanks for all the suggestions! I took Satish's advice, removed the > source, and did a fresh built. > > > > It got through the test compiling, but the tests failed as follow: > > ================== > > Running test examples to verify correct installation > > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 > and PETSC_ARCH=arch-linux2-c-debug > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > > Number of SNES iterations = 2 > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI > processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > ? > > Surely it did not print ? here? What was the complete message? > > Usually this happens due to an incompatible mpiexec being used because > it appears earlier in the path. > > What happens if you do > > cd src/snes/examples/tutorials > make ex19 > mpiexec -n 2 ./ex19 > > for mpiexec use the one associated with the MPI that you built PETSc with > > Barry > > > ================== > > > > I tried to go into src/snes/examples/tutorials/ex19, but didn't find any > logs. Is this error for the test Okay? > > > > Thanks, > > > > Kai > > > > > > > > On Fri, Jul 25, 2014 at 11:06 AM, Satish Balay > wrote: > > Its probably best to do a fresh build with the latest petsc-3.5.1 > tarball and see > > if the problem is reproduceable [with --with-shared-libraries=0] > > > > Satish > > > > On Fri, 25 Jul 2014, Jed Brown wrote: > > > > > Matthew Knepley writes: > > > > > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > > > > > >> Hi Satish, > > > >> > > > >> Thanks for the quick response. I attached the make.log in this > email, and > > > >> a quick glance, I didn't see any obvious errors. > > > >> > > > > > > > > This never built the MUMPS directory, even though you have > PETSC_HAVE_MUMPS > > > > defined. I can see two > > > > possible reasons for this: > > > > > > > > 1) You distribution as missing files > > > > > > > > 2) You have two copies of PETSc fighting each other here (old > PETSC_DIR?) > > > > > > > > Can you make sure you have the file mumps.c in > > > > > > > > src/mat/impls/aij/mpi/mumps > > > > > > > > Jed, how do we check what make is doing here? > > > > > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > > > generator runs. Possibly a file system problem or tinkering caused it > > > to appear up-to-date after reconfiguring? (Just speculating because we > > > haven't seen this problem before.) > > > > > > > > > > > > > -- > > Kai Song > > 1.510.495.2180 > > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > > High Performance Computing Services (HPCS) > > Lawrence Berkeley National Laboratory - http://scs.lbl.gov > > -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 25 15:28:14 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 25 Jul 2014 14:28:14 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> Message-ID: <87a97x5gkh.fsf@jedbrown.org> Sorry about not following up. I also find these results peculiar. Lawrence Mitchell writes: > So I'm sort of none-the-wiser. I'm a little bit at a loss as to why > this occurs, but either switching to Richardson+SOR or Cheby/SOR with > more that one SOR sweep appears to fix the problems, so I might just > punt for now. What discretization and mesh is this running on? Is there something special about the decomposition with 2 subdomains? Are the Chebyshev estimates far from converging? Something is funky because if you have a discretization with good h-ellipticity, reliable convergence should not be a problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Fri Jul 25 15:36:25 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 25 Jul 2014 14:36:25 -0600 Subject: [petsc-users] Problem with MATSOLVERPASTIX using KSPSolveTranspose In-Reply-To: References: Message-ID: <877g315g6u.fsf@jedbrown.org> Zin Lin writes: > Hi, > > I am having trouble calling KSPSolveTranspose under MATSOLVERPASTIX. > The following is a very simple C code solving A^T x = b to illustrate the > issue. The matrix A and vector b are built following the PETSC tutorial. > KSPSolve works fine but when it comes to KSPSolveTranspose, it spits out > the error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type mpiaij! > > The code runs fine if I change the solver package MATSOLVERPASTIX to > MATSOLVERMUMPS. > However, I need the PASTIX solver and the transpose solution for solving a > large problem. > Could you please let me know what is wrong with the PASTIX solver or with > my usage? The contributed PaStiX interface does not include MatSolveTranspose_PaStiX. If there is a way to do this in PaStiX, it should be added to the interface. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Jul 25 15:50:18 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 25 Jul 2014 15:50:18 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: <87y4vh77is.fsf@jedbrown.org> Message-ID: Well it is running. It is just producing annoying warning messages. You need to talk to your local MPI expert on that system for how to get rid of the problem. Barry On Jul 25, 2014, at 3:23 PM, Kai Song wrote: > Hi Barry, > > I indeed deleted some of the OpenMPI related warning messages, it should be irrelevant. Here is the full output: > ====================== > [kaisong at n0009 petsc-3.5.0]$ sh petsc-3.5.0-gcc.sh > Running test examples to verify correct installation > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 and PETSC_ARCH=arch-linux2-c-debug > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > -------------------------------------------------------------------------- > WARNING: There is at least non-excluded one OpenFabrics device found, > but there are no active ports detected (or Open MPI was unable to use > them). This is most certainly not what you wanted. Check your > cables, subnet manager configuration, etc. The openib BTL will be > ignored for this job. > > Local host: n0009.scs00 > -------------------------------------------------------------------------- > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > -------------------------------------------------------------------------- > WARNING: There is at least non-excluded one OpenFabrics device found, > but there are no active ports detected (or Open MPI was unable to use > them). This is most certainly not what you wanted. Check your > cables, subnet manager configuration, etc. The openib BTL will be > ignored for this job. > > Local host: n0009.scs00 > -------------------------------------------------------------------------- > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > [n0009.scs00:07638] 1 more process has sent help message help-mpi-btl-openib.txt / no active ports found > [n0009.scs00:07638] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > -------------------------------------------------------------------------- > WARNING: There is at least non-excluded one OpenFabrics device found, > but there are no active ports detected (or Open MPI was unable to use > them). This is most certainly not what you wanted. Check your > cables, subnet manager configuration, etc. The openib BTL will be > ignored for this job. > > Local host: n0009.scs00 > -------------------------------------------------------------------------- > Number of SNES iterations = 4 > Completed test examples > ========================================= > Now to evaluate the computer systems you plan use - do: > make PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 PETSC_ARCH=arch-linux2-c-debug streams NPMAX= > ====================== > > I also followed your suggestion and built ex19 separately, and made sure using the same compiler that built petsc. Here is the complete output: > ====================== > [kaisong at n0009 tutorials]$ mpiexec -n 2 ./ex19 > -------------------------------------------------------------------------- > WARNING: There is at least non-excluded one OpenFabrics device found, > but there are no active ports detected (or Open MPI was unable to use > them). This is most certainly not what you wanted. Check your > cables, subnet manager configuration, etc. The openib BTL will be > ignored for this job. > > Local host: n0009.scs00 > -------------------------------------------------------------------------- > lid velocity = 0.0625, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > [n0009.scs00:09124] 1 more process has sent help message help-mpi-btl-openib.txt / no active ports found > [n0009.scs00:09124] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > ====================== > > Thanks, > > Kai > > > > On Fri, Jul 25, 2014 at 1:32 PM, Barry Smith wrote: > > On Jul 25, 2014, at 12:23 PM, Kai Song wrote: > > > Hi All, > > > > Thanks for all the suggestions! I took Satish's advice, removed the source, and did a fresh built. > > > > It got through the test compiling, but the tests failed as follow: > > ================== > > Running test examples to verify correct installation > > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 and PETSC_ARCH=arch-linux2-c-debug > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > > Number of SNES iterations = 2 > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > ? > > Surely it did not print ? here? What was the complete message? > > Usually this happens due to an incompatible mpiexec being used because it appears earlier in the path. > > What happens if you do > > cd src/snes/examples/tutorials > make ex19 > mpiexec -n 2 ./ex19 > > for mpiexec use the one associated with the MPI that you built PETSc with > > Barry > > > ================== > > > > I tried to go into src/snes/examples/tutorials/ex19, but didn't find any logs. Is this error for the test Okay? > > > > Thanks, > > > > Kai > > > > > > > > On Fri, Jul 25, 2014 at 11:06 AM, Satish Balay wrote: > > Its probably best to do a fresh build with the latest petsc-3.5.1 tarball and see > > if the problem is reproduceable [with --with-shared-libraries=0] > > > > Satish > > > > On Fri, 25 Jul 2014, Jed Brown wrote: > > > > > Matthew Knepley writes: > > > > > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > > > > > >> Hi Satish, > > > >> > > > >> Thanks for the quick response. I attached the make.log in this email, and > > > >> a quick glance, I didn't see any obvious errors. > > > >> > > > > > > > > This never built the MUMPS directory, even though you have PETSC_HAVE_MUMPS > > > > defined. I can see two > > > > possible reasons for this: > > > > > > > > 1) You distribution as missing files > > > > > > > > 2) You have two copies of PETSc fighting each other here (old PETSC_DIR?) > > > > > > > > Can you make sure you have the file mumps.c in > > > > > > > > src/mat/impls/aij/mpi/mumps > > > > > > > > Jed, how do we check what make is doing here? > > > > > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > > > generator runs. Possibly a file system problem or tinkering caused it > > > to appear up-to-date after reconfiguring? (Just speculating because we > > > haven't seen this problem before.) > > > > > > > > > > > > > -- > > Kai Song > > 1.510.495.2180 > > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > > High Performance Computing Services (HPCS) > > Lawrence Berkeley National Laboratory - http://scs.lbl.gov > > > > > -- > Kai Song > 1.510.495.2180 > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > High Performance Computing Services (HPCS) > Lawrence Berkeley National Laboratory - http://scs.lbl.gov From ksong at lbl.gov Fri Jul 25 15:55:08 2014 From: ksong at lbl.gov (Kai Song) Date: Fri, 25 Jul 2014 15:55:08 -0500 Subject: [petsc-users] Petsc/3.5.0 make test failed In-Reply-To: References: <87y4vh77is.fsf@jedbrown.org> Message-ID: Wonderful! Thanks so much for all of your help! Kai On Fri, Jul 25, 2014 at 3:50 PM, Barry Smith wrote: > > Well it is running. It is just producing annoying warning messages. You > need to talk to your local MPI expert on that system for how to get rid of > the problem. > > Barry > > On Jul 25, 2014, at 3:23 PM, Kai Song wrote: > > > Hi Barry, > > > > I indeed deleted some of the OpenMPI related warning messages, it should > be irrelevant. Here is the full output: > > ====================== > > [kaisong at n0009 petsc-3.5.0]$ sh petsc-3.5.0-gcc.sh > > Running test examples to verify correct installation > > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 > and PETSC_ARCH=arch-linux2-c-debug > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > -------------------------------------------------------------------------- > > WARNING: There is at least non-excluded one OpenFabrics device found, > > but there are no active ports detected (or Open MPI was unable to use > > them). This is most certainly not what you wanted. Check your > > cables, subnet manager configuration, etc. The openib BTL will be > > ignored for this job. > > > > Local host: n0009.scs00 > > > -------------------------------------------------------------------------- > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > > Number of SNES iterations = 2 > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI > processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > -------------------------------------------------------------------------- > > WARNING: There is at least non-excluded one OpenFabrics device found, > > but there are no active ports detected (or Open MPI was unable to use > > them). This is most certainly not what you wanted. Check your > > cables, subnet manager configuration, etc. The openib BTL will be > > ignored for this job. > > > > Local host: n0009.scs00 > > > -------------------------------------------------------------------------- > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > > Number of SNES iterations = 2 > > [n0009.scs00:07638] 1 more process has sent help message > help-mpi-btl-openib.txt / no active ports found > > [n0009.scs00:07638] Set MCA parameter "orte_base_help_aggregate" to 0 to > see all help / error messages > > Possible error running Fortran example src/snes/examples/tutorials/ex5f > with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > -------------------------------------------------------------------------- > > WARNING: There is at least non-excluded one OpenFabrics device found, > > but there are no active ports detected (or Open MPI was unable to use > > them). This is most certainly not what you wanted. Check your > > cables, subnet manager configuration, etc. The openib BTL will be > > ignored for this job. > > > > Local host: n0009.scs00 > > > -------------------------------------------------------------------------- > > Number of SNES iterations = 4 > > Completed test examples > > ========================================= > > Now to evaluate the computer systems you plan use - do: > > make PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 > PETSC_ARCH=arch-linux2-c-debug streams NPMAX= intend to use> > > ====================== > > > > I also followed your suggestion and built ex19 separately, and made sure > using the same compiler that built petsc. Here is the complete output: > > ====================== > > [kaisong at n0009 tutorials]$ mpiexec -n 2 ./ex19 > > > -------------------------------------------------------------------------- > > WARNING: There is at least non-excluded one OpenFabrics device found, > > but there are no active ports detected (or Open MPI was unable to use > > them). This is most certainly not what you wanted. Check your > > cables, subnet manager configuration, etc. The openib BTL will be > > ignored for this job. > > > > Local host: n0009.scs00 > > > -------------------------------------------------------------------------- > > lid velocity = 0.0625, prandtl # = 1, grashof # = 1 > > Number of SNES iterations = 2 > > [n0009.scs00:09124] 1 more process has sent help message > help-mpi-btl-openib.txt / no active ports found > > [n0009.scs00:09124] Set MCA parameter "orte_base_help_aggregate" to 0 to > see all help / error messages > > ====================== > > > > Thanks, > > > > Kai > > > > > > > > On Fri, Jul 25, 2014 at 1:32 PM, Barry Smith wrote: > > > > On Jul 25, 2014, at 12:23 PM, Kai Song wrote: > > > > > Hi All, > > > > > > Thanks for all the suggestions! I took Satish's advice, removed the > source, and did a fresh built. > > > > > > It got through the test compiling, but the tests failed as follow: > > > ================== > > > Running test examples to verify correct installation > > > Using PETSC_DIR=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 > and PETSC_ARCH=arch-linux2-c-debug > > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 > MPI process > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > > > Number of SNES iterations = 2 > > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 > MPI processes > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > ? > > > > Surely it did not print ? here? What was the complete message? > > > > Usually this happens due to an incompatible mpiexec being used > because it appears earlier in the path. > > > > What happens if you do > > > > cd src/snes/examples/tutorials > > make ex19 > > mpiexec -n 2 ./ex19 > > > > for mpiexec use the one associated with the MPI that you built PETSc with > > > > Barry > > > > > ================== > > > > > > I tried to go into src/snes/examples/tutorials/ex19, but didn't find > any logs. Is this error for the test Okay? > > > > > > Thanks, > > > > > > Kai > > > > > > > > > > > > On Fri, Jul 25, 2014 at 11:06 AM, Satish Balay > wrote: > > > Its probably best to do a fresh build with the latest petsc-3.5.1 > tarball and see > > > if the problem is reproduceable [with --with-shared-libraries=0] > > > > > > Satish > > > > > > On Fri, 25 Jul 2014, Jed Brown wrote: > > > > > > > Matthew Knepley writes: > > > > > > > > > On Thu, Jul 24, 2014 at 5:33 PM, Kai Song wrote: > > > > > > > > > >> Hi Satish, > > > > >> > > > > >> Thanks for the quick response. I attached the make.log in this > email, and > > > > >> a quick glance, I didn't see any obvious errors. > > > > >> > > > > > > > > > > This never built the MUMPS directory, even though you have > PETSC_HAVE_MUMPS > > > > > defined. I can see two > > > > > possible reasons for this: > > > > > > > > > > 1) You distribution as missing files > > > > > > > > > > 2) You have two copies of PETSc fighting each other here (old > PETSC_DIR?) > > > > > > > > > > Can you make sure you have the file mumps.c in > > > > > > > > > > src/mat/impls/aij/mpi/mumps > > > > > > > > > > Jed, how do we check what make is doing here? > > > > > > > > mumps.c should be listed in $PETSC_ARCH/conf/files after the code > > > > generator runs. Possibly a file system problem or tinkering caused > it > > > > to appear up-to-date after reconfiguring? (Just speculating because > we > > > > haven't seen this problem before.) > > > > > > > > > > > > > > > > > > > -- > > > Kai Song > > > 1.510.495.2180 > > > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > > > High Performance Computing Services (HPCS) > > > Lawrence Berkeley National Laboratory - http://scs.lbl.gov > > > > > > > > > > -- > > Kai Song > > 1.510.495.2180 > > 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 > > High Performance Computing Services (HPCS) > > Lawrence Berkeley National Laboratory - http://scs.lbl.gov > > -- Kai Song 1.510.495.2180 1 Cyclotron Rd. Berkeley, CA94720, MS-50B 3209 High Performance Computing Services (HPCS) Lawrence Berkeley National Laboratory - http://scs.lbl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Sat Jul 26 07:03:06 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Sat, 26 Jul 2014 13:03:06 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <87a97x5gkh.fsf@jedbrown.org> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> Message-ID: <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> On 25 Jul 2014, at 21:28, Jed Brown wrote: > Sorry about not following up. I also find these results peculiar. > > Lawrence Mitchell writes: >> So I'm sort of none-the-wiser. I'm a little bit at a loss as to why >> this occurs, but either switching to Richardson+SOR or Cheby/SOR with >> more that one SOR sweep appears to fix the problems, so I might just >> punt for now. > > What discretization and mesh is this running on? Bog-standard P1 on a pretty much regularly meshed square domain (i.e. no reentrant corners or bad elements). > Is there something special about the decomposition with 2 subdomains? It doesn't look like it, the two subdomains are about the same size. > Are the Chebyshev estimates far from converging? So for the two-level problem, if I compute the extremal eigenvalues of the preconditioned operator being used as a smoother I get (approximately): 1 process: 0.019, 1.0 2 processes: 0.016, 1.4 3 processes: 0.016, 1.36 The eigenvalue estimates (from ksp_view) are: 1 process: 0.09, 1.01 2 processes: 0.09, 1.01 3 processes: 0.13, 1.47 When I bump to more levels, the estimates are only bad on two processes on the finest grid. So for example running with something like (for a 3 level problem): -pc_type mg -mg_levels_2_ksp_chebyshev_eigenvalues 0.09,1.4 gives me good convergence on two processes where the extremal eigenvalues on the finest grid are: 0.016, 1.39 Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jifengzhao2015 at u.northwestern.edu Sat Jul 26 10:34:01 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Sat, 26 Jul 2014 10:34:01 -0500 Subject: [petsc-users] eps_target malloc issue Message-ID: Hello all, I am trying to use ex7 to solve series of generalized problem. The running commands are: ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij -eps_gen_hermitian -eps_type krylovschur -*eps_smallest_magnitude*. -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert It works fine, but when my problem contains zero eigenvalue, it fails to converge. Supposedly, I should use *eps_target -0.1* instead of *eps_smallest_magnitude*. However if eps_target is used, I run into a memory issue: PETSC ERROR: --------------------- Error Message ------------------------------------ PETSC ERROR: Argument out of range! PETSC ERROR: New nonzero at (0,6) caused a malloc! PETSC ERROR: ------------------------------------------------------------------------ Does anybody know how to solve this? Thanks! Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jul 26 12:04:21 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 26 Jul 2014 12:04:21 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: References: Message-ID: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> We need the entire error message. A matrix is being formed that wasn?t fully preallocated for so you need to determine which matrix and make sure either 1) it gets fully preallocated or 2) MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix Barry On Jul 26, 2014, at 10:34 AM, jifeng zhao wrote: > Hello all, > > I am trying to use ex7 to solve series of generalized problem. The running commands are: > > ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert > > It works fine, but when my problem contains zero eigenvalue, it fails to converge. > > Supposedly, I should use eps_target -0.1 instead of eps_smallest_magnitude. However if eps_target is used, I run into a memory issue: > > PETSC ERROR: --------------------- Error Message ------------------------------------ > PETSC ERROR: Argument out of range! > PETSC ERROR: New nonzero at (0,6) caused a malloc! > PETSC ERROR: ------------------------------------------------------------------------ > > Does anybody know how to solve this? Thanks! > > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program From jroman at dsic.upv.es Sat Jul 26 12:35:38 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sat, 26 Jul 2014 19:35:38 +0200 Subject: [petsc-users] eps_target malloc issue In-Reply-To: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> Message-ID: <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> The problem is that your stiffness and mass matrices have different nonzero pattern. The -eps_target configuration computes a MatAXPY on the first matrix (stiffness), so MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) could be called on this matrix, as Barry suggests. Another workaround would be to force both of them having the same nonzero pattern, maybe using MatSetOption(mat,MAT_IGNORE_ZERO_ENTRIES,PETSC_FALSE). Also, this problem would not appear with target=0.0. See details on section 3.4.2 of SLEPc's users guide. Jose El 26/07/2014, a las 19:04, Barry Smith escribi?: > > We need the entire error message. A matrix is being formed that wasn?t fully preallocated for so you need to determine which matrix and make sure either 1) it gets fully preallocated or 2) MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix > > Barry > > On Jul 26, 2014, at 10:34 AM, jifeng zhao wrote: > >> Hello all, >> >> I am trying to use ex7 to solve series of generalized problem. The running commands are: >> >> ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert >> >> It works fine, but when my problem contains zero eigenvalue, it fails to converge. >> >> Supposedly, I should use eps_target -0.1 instead of eps_smallest_magnitude. However if eps_target is used, I run into a memory issue: >> >> PETSC ERROR: --------------------- Error Message ------------------------------------ >> PETSC ERROR: Argument out of range! >> PETSC ERROR: New nonzero at (0,6) caused a malloc! >> PETSC ERROR: ------------------------------------------------------------------------ >> >> Does anybody know how to solve this? Thanks! >> >> Jifeng Zhao >> PhD candidate at Northwestern University, US >> Theoretical and Applied Mechanics Program > From hus003 at ucsd.edu Sun Jul 27 03:18:29 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 27 Jul 2014 08:18:29 +0000 Subject: [petsc-users] MATLAB viewer Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2BF@XMAIL-MBX-BH1.AD.UCSD.EDU> I try to create some binary output for MATLAB with the following code. PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); MatView(A,viewer); VecView(v,viewer); PetscViewerDestroy(&viewer); However it gives me an error, which is the following: interface_fs.c:215:26: warning: implicit declaration of function 'PETSC_VIEWER_MATLAB' is invalid in C99 [-Wimplicit-function-declaration] PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); ^ interface_fs.c:215:17: warning: incompatible integer to pointer conversion initializing 'PetscViewer' (aka 'struct _p_PetscViewer *') with an expression of type 'int' [-Wint-conversion] PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); I'm wondering what is the correct way to do this? Thank you! From patrick.sanan at gmail.com Sun Jul 27 06:19:16 2014 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Sun, 27 Jul 2014 13:19:16 +0200 Subject: [petsc-users] MATLAB viewer In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2BF@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2BF@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <461E0605-8050-4F35-BD7F-941B70E2E647@gmail.com> A simple way to proceed is to use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html On Jul 27, 2014, at 10:18 AM, Sun, Hui wrote: > I try to create some binary output for MATLAB with the following code. > > PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); > MatView(A,viewer); > VecView(v,viewer); > PetscViewerDestroy(&viewer); > > However it gives me an error, which is the following: > > interface_fs.c:215:26: warning: implicit declaration of function 'PETSC_VIEWER_MATLAB' is invalid in C99 [-Wimplicit-function-declaration] > PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); > ^ > interface_fs.c:215:17: warning: incompatible integer to pointer conversion initializing 'PetscViewer' (aka 'struct _p_PetscViewer *') with an expression of type 'int' > [-Wint-conversion] > PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); > > > I'm wondering what is the correct way to do this? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuiper at mpia.de Sun Jul 27 10:45:59 2014 From: kuiper at mpia.de (Rolf Kuiper) Date: Sun, 27 Jul 2014 17:45:59 +0200 Subject: [petsc-users] force KSP to perform at least a single iteration Message-ID: Dear PETSc users and developers, is there an easy way to set a minimum iteration number to a KSP context (such as a maximum iteration number in KSPSetTolerances())? I don't know, if the following explanation is useful, but I will try to give an example usage here: I need this to e.g. solve a linear system of equations several times (as the physical system proceeds in time); during a single call / timestep, the system, might not change stronger than the RTOL in use, but in the long-run (several hundreds to thousands of calls), the system will actually change by several RTOL; nonetheless, the default KSP solver stops without doing any iteration (= 0), i.e. the system does not change at all. I have the feeling that it might be possible by using my own convergence test with KSPSetConvergenceTest(), but I am not 100% sure and would prefer to go with the standard PETSc convergence test, just adding the minimum iteration number. Thanks a lot, Rolf -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 27 11:01:45 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 27 Jul 2014 11:01:45 -0500 Subject: [petsc-users] force KSP to perform at least a single iteration In-Reply-To: References: Message-ID: On Sun, Jul 27, 2014 at 10:45 AM, Rolf Kuiper wrote: > Dear PETSc users and developers, > > is there an easy way to set a *minimum* iteration number to a KSP context > (such as a maximum iteration number in KSPSetTolerances())? > > I don't know, if the following explanation is useful, but I will try to > give an example usage here: > I need this to e.g. solve a linear system of equations several times (as > the physical system proceeds in time); during a single call / timestep, the > system, might not change stronger than the RTOL in use, but in the long-run > (several hundreds to thousands of calls), the system will actually change > by several RTOL; nonetheless, the default KSP solver stops without doing > any iteration (= 0), i.e. the system does not change at all. > 1) This explanation makes no sense to me. Either you have a difference greater than the tolerance or not. > I have the feeling that it might be possible by using my own convergence > test with KSPSetConvergenceTest(), but I am not 100% sure and would prefer > to go with the standard PETSc convergence test, just adding the *minimum* iteration > number. > 2) The easiest thing to do is make a custom convergence test that immediately returns no if its the first iterate, and otherwise calls the default test. Thanks, Matt > Thanks a lot, > Rolf > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jul 27 12:51:28 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 27 Jul 2014 12:51:28 -0500 Subject: [petsc-users] MATLAB viewer In-Reply-To: <461E0605-8050-4F35-BD7F-941B70E2E647@gmail.com> References: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2BF@XMAIL-MBX-BH1.AD.UCSD.EDU> <461E0605-8050-4F35-BD7F-941B70E2E647@gmail.com> Message-ID: On Jul 27, 2014, at 6:19 AM, Patrick Sanan wrote: > A simple way to proceed is to use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html > > On Jul 27, 2014, at 10:18 AM, Sun, Hui wrote: > >> I try to create some binary output for MATLAB with the following code. >> >> PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); or you are missing the _ at the end of PETSC_VIEWER_BINARY_ >> MatView(A,viewer); >> VecView(v,viewer); >> PetscViewerDestroy(&viewer); >> >> However it gives me an error, which is the following: >> >> interface_fs.c:215:26: warning: implicit declaration of function 'PETSC_VIEWER_MATLAB' is invalid in C99 [-Wimplicit-function-declaration] >> PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); >> ^ >> interface_fs.c:215:17: warning: incompatible integer to pointer conversion initializing 'PetscViewer' (aka 'struct _p_PetscViewer *') with an expression of type 'int' >> [-Wint-conversion] >> PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); >> >> >> I'm wondering what is the correct way to do this? Thank you! > From jifengzhao2015 at u.northwestern.edu Sun Jul 27 15:37:05 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Sun, 27 Jul 2014 15:37:05 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> Message-ID: Thank you all for the insights. I tried to add MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE). It does solve the malloc error. However there appears to be another error: In addition, 66324 appears to be the number of index that is assigned to each processor. Any more insights? Thanks a lot! PETSC ERROR: Argument out of range! PETSC ERROR: *Column too large: col 935748 max 66324!* PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014 PETSC ERROR: See docs/changes/index.html for recent updates. PETSC ERROR: See docs/faq.html for hints about trouble shooting. PETSC ERROR: See docs/index.html for manual pages. PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: eigen_solver20 on a arch-linux2-c-debug named qnode4091 by jzh953 Sun Jul 27 15:17:28 2014 PETSC ERROR: Libraries linked from /software/petsc/3.4.4/lib PETSC ERROR: Configure run at Tue Jul 8 14:20:27 2014 ]PETSC ERROR: Configure options --prefix=/software/petsc/3.4.4 --with-blas-lib=/software/lapack/3.5.0/lib/libblas.so --with-lapack-lib=/software/lapack/3.5.0/liblapack.so --with-hdf5-dir=/software/hdf5/1.8.12 with-boost=1 --with-boost-dir=/software/hdf5/1.8.12 --with-fftw=1 --with-fftw-dir=/software/FFTW/3.3.3-gcc-threads --with-valgrind-dir=/software/valgrind/3.8.1 --with-netcdf=1 --with-netcdf-dir=/software/netCDF/4.2 --with-hdf5=1 --with-debugging=1 --with-mpi-dir=/software/mpi/openmpi-1.6.3-gcc-4.6.3-trq4 --download-scalapack --download-mumps --download-hypre --download-parmetis --download-metis PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: MatSetValues_SeqSBAIJ() line 986 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/seq/sbaij.c PETSC ERROR: MatDisAssemble_MPISBAIJ() line 356 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mmsbaij.c PETSC ERROR: MatSetValues_MPISBAIJ() line 213 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c PETSC ERROR: MatSetValues() line 1106 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/interface/matrix.c PETSC ERROR: MatAXPY_Basic() line 78 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c PETSC ERROR: MatAXPY_MPISBAIJ() line 1398 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c PETSC ERROR: MatAXPY() line 39 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c PETSC ERROR: STMatGAXPY_Private() line 375 in src/st/interface/stsolve.c PETSC ERROR: STSetUp_Sinvert() line 138 in src/st/impls/sinvert/sinvert.c PETSC ERROR: STSetUp() line 294 in src/st/interface/stsolve.c PETSC ERROR: EPSSetUp() line 215 in src/eps/interface/setup.c PETSC ERROR: EPSSolve() line 90 in src/eps/interface/solve.c PETSC ERROR: -------------------------------------------------------------------------- On Sat, Jul 26, 2014 at 12:35 PM, Jose E. Roman wrote: > The problem is that your stiffness and mass matrices have different > nonzero pattern. > > The -eps_target configuration computes a MatAXPY on the first matrix > (stiffness), so MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) could be called on this matrix, > as Barry suggests. > > Another workaround would be to force both of them having the same nonzero > pattern, maybe using MatSetOption(mat,MAT_IGNORE_ZERO_ENTRIES,PETSC_FALSE). > > Also, this problem would not appear with target=0.0. > > See details on section 3.4.2 of SLEPc's users guide. > > Jose > > > > El 26/07/2014, a las 19:04, Barry Smith escribi?: > > > > > We need the entire error message. A matrix is being formed that wasn?t > fully preallocated for so you need to determine which matrix and make sure > either 1) it gets fully preallocated or 2) MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix > > > > Barry > > > > On Jul 26, 2014, at 10:34 AM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > > > >> Hello all, > >> > >> I am trying to use ex7 to solve series of generalized problem. The > running commands are: > >> > >> ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij > -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. > -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type > icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert > >> > >> It works fine, but when my problem contains zero eigenvalue, it fails > to converge. > >> > >> Supposedly, I should use eps_target -0.1 instead of > eps_smallest_magnitude. However if eps_target is used, I run into a memory > issue: > >> > >> PETSC ERROR: --------------------- Error Message > ------------------------------------ > >> PETSC ERROR: Argument out of range! > >> PETSC ERROR: New nonzero at (0,6) caused a malloc! > >> PETSC ERROR: > ------------------------------------------------------------------------ > >> > >> Does anybody know how to solve this? Thanks! > >> > >> Jifeng Zhao > >> PhD candidate at Northwestern University, US > >> Theoretical and Applied Mechanics Program > > > > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Sun Jul 27 18:38:10 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 27 Jul 2014 23:38:10 +0000 Subject: [petsc-users] MATLAB viewer In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2BF@XMAIL-MBX-BH1.AD.UCSD.EDU> <461E0605-8050-4F35-BD7F-941B70E2E647@gmail.com>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB6CC2DF@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Barry and Patrick. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Sunday, July 27, 2014 10:51 AM To: Patrick Sanan Cc: Sun, Hui; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] MATLAB viewer On Jul 27, 2014, at 6:19 AM, Patrick Sanan wrote: > A simple way to proceed is to use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html > > On Jul 27, 2014, at 10:18 AM, Sun, Hui wrote: > >> I try to create some binary output for MATLAB with the following code. >> >> PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); or you are missing the _ at the end of PETSC_VIEWER_BINARY_ >> MatView(A,viewer); >> VecView(v,viewer); >> PetscViewerDestroy(&viewer); >> >> However it gives me an error, which is the following: >> >> interface_fs.c:215:26: warning: implicit declaration of function 'PETSC_VIEWER_MATLAB' is invalid in C99 [-Wimplicit-function-declaration] >> PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); >> ^ >> interface_fs.c:215:17: warning: incompatible integer to pointer conversion initializing 'PetscViewer' (aka 'struct _p_PetscViewer *') with an expression of type 'int' >> [-Wint-conversion] >> PetscViewer viewer = PETSC_VIEWER_MATLAB(PETSC_COMM_WORLD); >> >> >> I'm wondering what is the correct way to do this? Thank you! > From oliver.browne at upm.es Mon Jul 28 07:43:45 2014 From: oliver.browne at upm.es (Oliver Browne) Date: Mon, 28 Jul 2014 14:43:45 +0200 Subject: [petsc-users] ILU - factor fill ratio Message-ID: <55e383160aa11f44017a9d41b6dac510@upm.es> Hi, I am using the ILU preconditioner. I am looking at "factor fill ratio" in the output. Is there a way to a priori calculate what this value may be to determine whether enough memory is available? How does PETSc determine this value? Thanks, Ollie From knepley at gmail.com Mon Jul 28 07:47:49 2014 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 28 Jul 2014 07:47:49 -0500 Subject: [petsc-users] ILU - factor fill ratio In-Reply-To: <55e383160aa11f44017a9d41b6dac510@upm.es> References: <55e383160aa11f44017a9d41b6dac510@upm.es> Message-ID: On Mon, Jul 28, 2014 at 7:43 AM, Oliver Browne wrote: > Hi, > > I am using the ILU preconditioner. I am looking at "factor fill ratio" in > the output. Is there a way to a priori calculate what this value may be to > determine whether enough memory is available? How does PETSc determine this > value? > Calculating this is as difficult as doing the symbolic factorization, so we guess and check. Thanks, Matt > Thanks, > > Ollie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jifengzhao2015 at u.northwestern.edu Mon Jul 28 10:27:37 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Mon, 28 Jul 2014 10:27:37 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> Message-ID: Hello everyone, Are there more ideas why when using MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE), the processor seems trying to access elements that are out of local range? Anybody can help? Thank you so much. Best regards, Jifeng Zhao On Sun, Jul 27, 2014 at 3:37 PM, jifeng zhao < jifengzhao2015 at u.northwestern.edu> wrote: > Thank you all for the insights. > > I tried to add MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE). > It does solve the malloc error. However there appears to be another error: > > In addition, 66324 appears to be the number of index that is assigned to > each processor. > > Any more insights? Thanks a lot! > > PETSC ERROR: Argument out of range! > PETSC ERROR: *Column too large: col 935748 max 66324!* > PETSC ERROR: > ------------------------------------------------------------------------ > PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014 > PETSC ERROR: See docs/changes/index.html for recent updates. > PETSC ERROR: See docs/faq.html for hints about trouble shooting. > PETSC ERROR: See docs/index.html for manual pages. > PETSC ERROR: > ------------------------------------------------------------------------ > PETSC ERROR: eigen_solver20 on a arch-linux2-c-debug named qnode4091 by > jzh953 Sun Jul 27 15:17:28 2014 > PETSC ERROR: Libraries linked from /software/petsc/3.4.4/lib > PETSC ERROR: Configure run at Tue Jul 8 14:20:27 2014 > ]PETSC ERROR: Configure options --prefix=/software/petsc/3.4.4 > --with-blas-lib=/software/lapack/3.5.0/lib/libblas.so > --with-lapack-lib=/software/lapack/3.5.0/liblapack.so > --with-hdf5-dir=/software/hdf5/1.8.12 with-boost=1 > --with-boost-dir=/software/hdf5/1.8.12 --with-fftw=1 > --with-fftw-dir=/software/FFTW/3.3.3-gcc-threads > --with-valgrind-dir=/software/valgrind/3.8.1 --with-netcdf=1 > --with-netcdf-dir=/software/netCDF/4.2 --with-hdf5=1 --with-debugging=1 > --with-mpi-dir=/software/mpi/openmpi-1.6.3-gcc-4.6.3-trq4 > --download-scalapack --download-mumps --download-hypre --download-parmetis > --download-metis > > PETSC ERROR: > ------------------------------------------------------------------------ > PETSC ERROR: MatSetValues_SeqSBAIJ() line 986 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/seq/sbaij.c > PETSC ERROR: MatDisAssemble_MPISBAIJ() line 356 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mmsbaij.c > PETSC ERROR: MatSetValues_MPISBAIJ() line 213 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > PETSC ERROR: MatSetValues() line 1106 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/interface/matrix.c > PETSC ERROR: MatAXPY_Basic() line 78 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > PETSC ERROR: MatAXPY_MPISBAIJ() line 1398 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > PETSC ERROR: MatAXPY() line 39 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > PETSC ERROR: STMatGAXPY_Private() line 375 in src/st/interface/stsolve.c > PETSC ERROR: STSetUp_Sinvert() line 138 in src/st/impls/sinvert/sinvert.c > PETSC ERROR: STSetUp() line 294 in src/st/interface/stsolve.c > PETSC ERROR: EPSSetUp() line 215 in src/eps/interface/setup.c > PETSC ERROR: EPSSolve() line 90 in src/eps/interface/solve.c > PETSC ERROR: > -------------------------------------------------------------------------- > > > On Sat, Jul 26, 2014 at 12:35 PM, Jose E. Roman > wrote: > >> The problem is that your stiffness and mass matrices have different >> nonzero pattern. >> >> The -eps_target configuration computes a MatAXPY on the first matrix >> (stiffness), so MatSetOption(mat, >> MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) could be called on this matrix, >> as Barry suggests. >> >> Another workaround would be to force both of them having the same nonzero >> pattern, maybe using MatSetOption(mat,MAT_IGNORE_ZERO_ENTRIES,PETSC_FALSE). >> >> Also, this problem would not appear with target=0.0. >> >> See details on section 3.4.2 of SLEPc's users guide. >> >> Jose >> >> >> >> El 26/07/2014, a las 19:04, Barry Smith escribi?: >> >> > >> > We need the entire error message. A matrix is being formed that wasn?t >> fully preallocated for so you need to determine which matrix and make sure >> either 1) it gets fully preallocated or 2) MatSetOption(mat, >> MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix >> > >> > Barry >> > >> > On Jul 26, 2014, at 10:34 AM, jifeng zhao < >> jifengzhao2015 at u.northwestern.edu> wrote: >> > >> >> Hello all, >> >> >> >> I am trying to use ex7 to solve series of generalized problem. The >> running commands are: >> >> >> >> ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij >> -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. >> -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type >> icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert >> >> >> >> It works fine, but when my problem contains zero eigenvalue, it fails >> to converge. >> >> >> >> Supposedly, I should use eps_target -0.1 instead of >> eps_smallest_magnitude. However if eps_target is used, I run into a memory >> issue: >> >> >> >> PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> >> PETSC ERROR: Argument out of range! >> >> PETSC ERROR: New nonzero at (0,6) caused a malloc! >> >> PETSC ERROR: >> ------------------------------------------------------------------------ >> >> >> >> Does anybody know how to solve this? Thanks! >> >> >> >> Jifeng Zhao >> >> PhD candidate at Northwestern University, US >> >> Theoretical and Applied Mechanics Program >> > >> >> > > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 28 10:37:50 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 28 Jul 2014 09:37:50 -0600 Subject: [petsc-users] force KSP to perform at least a single iteration In-Reply-To: References: Message-ID: <87mwbt1okx.fsf@jedbrown.org> Rolf Kuiper writes: > I need this to e.g. solve a linear system of equations several times > (as the physical system proceeds in time); during a single call / > timestep, the system, might not change stronger than the RTOL in use, > but in the long-run (several hundreds to thousands of calls), the > system will actually change by several RTOL; nonetheless, the default > KSP solver stops without doing any iteration (= 0), i.e. the system > does not change at all. When you use -ksp_converged_reason, is it completing due to RTOL or ATOL? For transient problems, it's often good to set an absolute tolerance. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From kuiper at mpia.de Mon Jul 28 10:53:20 2014 From: kuiper at mpia.de (Rolf Kuiper) Date: Mon, 28 Jul 2014 17:53:20 +0200 Subject: [petsc-users] force KSP to perform at least a single iteration In-Reply-To: <87mwbt1okx.fsf@jedbrown.org> References: <87mwbt1okx.fsf@jedbrown.org> Message-ID: <7A9FC572-BF38-47F3-87F8-57A2CA623ED7@mpia.de> Hi Jed, in general, I am using only RTOL with ATOL=0, sometimes (if the range of the physical solution is known a priori) I use both criteria. Actually, Matthew already helped me to build my own KSPConverged routine including the minimum iteration number. Thanks a lot for all your effort and great support, Rolf Am 28.07.2014 um 17:37 schrieb Jed Brown: > Rolf Kuiper writes: >> I need this to e.g. solve a linear system of equations several times >> (as the physical system proceeds in time); during a single call / >> timestep, the system, might not change stronger than the RTOL in use, >> but in the long-run (several hundreds to thousands of calls), the >> system will actually change by several RTOL; nonetheless, the default >> KSP solver stops without doing any iteration (= 0), i.e. the system >> does not change at all. > > When you use -ksp_converged_reason, is it completing due to RTOL or > ATOL? For transient problems, it's often good to set an absolute > tolerance. From bsmith at mcs.anl.gov Mon Jul 28 12:07:43 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 28 Jul 2014 12:07:43 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> Message-ID: <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> On Jul 28, 2014, at 10:27 AM, jifeng zhao wrote: > Hello everyone, > > Are there more ideas why when using MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE), the processor seems trying to access elements that are out of local range? Add the option has NOTHING to do with the ?processor seems trying to access elements out of local range?. It is just that adding that option allows the code to run further until it hits this other bug. So they second bug needs to be tracked down. Barry > > Anybody can help? Thank you so much. > > Best regards, > Jifeng Zhao > > > On Sun, Jul 27, 2014 at 3:37 PM, jifeng zhao wrote: > Thank you all for the insights. > > I tried to add MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE). It does solve the malloc error. However there appears to be another error: > > In addition, 66324 appears to be the number of index that is assigned to each processor. > > Any more insights? Thanks a lot! > > PETSC ERROR: Argument out of range! > PETSC ERROR: Column too large: col 935748 max 66324! > PETSC ERROR: ------------------------------------------------------------------------ > PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014 > PETSC ERROR: See docs/changes/index.html for recent updates. > PETSC ERROR: See docs/faq.html for hints about trouble shooting. > PETSC ERROR: See docs/index.html for manual pages. > PETSC ERROR: ------------------------------------------------------------------------ > PETSC ERROR: eigen_solver20 on a arch-linux2-c-debug named qnode4091 by jzh953 Sun Jul 27 15:17:28 2014 > PETSC ERROR: Libraries linked from /software/petsc/3.4.4/lib > PETSC ERROR: Configure run at Tue Jul 8 14:20:27 2014 > ]PETSC ERROR: Configure options --prefix=/software/petsc/3.4.4 --with-blas-lib=/software/lapack/3.5.0/lib/libblas.so --with-lapack-lib=/software/lapack/3.5.0/liblapack.so --with-hdf5-dir=/software/hdf5/1.8.12 with-boost=1 --with-boost-dir=/software/hdf5/1.8.12 --with-fftw=1 --with-fftw-dir=/software/FFTW/3.3.3-gcc-threads --with-valgrind-dir=/software/valgrind/3.8.1 --with-netcdf=1 --with-netcdf-dir=/software/netCDF/4.2 --with-hdf5=1 --with-debugging=1 --with-mpi-dir=/software/mpi/openmpi-1.6.3-gcc-4.6.3-trq4 --download-scalapack --download-mumps --download-hypre --download-parmetis --download-metis > > PETSC ERROR: ------------------------------------------------------------------------ > PETSC ERROR: MatSetValues_SeqSBAIJ() line 986 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/seq/sbaij.c > PETSC ERROR: MatDisAssemble_MPISBAIJ() line 356 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mmsbaij.c > PETSC ERROR: MatSetValues_MPISBAIJ() line 213 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > PETSC ERROR: MatSetValues() line 1106 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/interface/matrix.c > PETSC ERROR: MatAXPY_Basic() line 78 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > PETSC ERROR: MatAXPY_MPISBAIJ() line 1398 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > PETSC ERROR: MatAXPY() line 39 in /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > PETSC ERROR: STMatGAXPY_Private() line 375 in src/st/interface/stsolve.c > PETSC ERROR: STSetUp_Sinvert() line 138 in src/st/impls/sinvert/sinvert.c > PETSC ERROR: STSetUp() line 294 in src/st/interface/stsolve.c > PETSC ERROR: EPSSetUp() line 215 in src/eps/interface/setup.c > PETSC ERROR: EPSSolve() line 90 in src/eps/interface/solve.c > PETSC ERROR: -------------------------------------------------------------------------- > > > On Sat, Jul 26, 2014 at 12:35 PM, Jose E. Roman wrote: > The problem is that your stiffness and mass matrices have different nonzero pattern. > > The -eps_target configuration computes a MatAXPY on the first matrix (stiffness), so MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) could be called on this matrix, as Barry suggests. > > Another workaround would be to force both of them having the same nonzero pattern, maybe using MatSetOption(mat,MAT_IGNORE_ZERO_ENTRIES,PETSC_FALSE). > > Also, this problem would not appear with target=0.0. > > See details on section 3.4.2 of SLEPc's users guide. > > Jose > > > > El 26/07/2014, a las 19:04, Barry Smith escribi?: > > > > > We need the entire error message. A matrix is being formed that wasn?t fully preallocated for so you need to determine which matrix and make sure either 1) it gets fully preallocated or 2) MatSetOption(mat, MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix > > > > Barry > > > > On Jul 26, 2014, at 10:34 AM, jifeng zhao wrote: > > > >> Hello all, > >> > >> I am trying to use ex7 to solve series of generalized problem. The running commands are: > >> > >> ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert > >> > >> It works fine, but when my problem contains zero eigenvalue, it fails to converge. > >> > >> Supposedly, I should use eps_target -0.1 instead of eps_smallest_magnitude. However if eps_target is used, I run into a memory issue: > >> > >> PETSC ERROR: --------------------- Error Message ------------------------------------ > >> PETSC ERROR: Argument out of range! > >> PETSC ERROR: New nonzero at (0,6) caused a malloc! > >> PETSC ERROR: ------------------------------------------------------------------------ > >> > >> Does anybody know how to solve this? Thanks! > >> > >> Jifeng Zhao > >> PhD candidate at Northwestern University, US > >> Theoretical and Applied Mechanics Program > > > > > > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program > > > > -- > Jifeng Zhao > PhD candidate at Northwestern University, US > Theoretical and Applied Mechanics Program From jifengzhao2015 at u.northwestern.edu Mon Jul 28 16:48:12 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Mon, 28 Jul 2014 16:48:12 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> Message-ID: Hi Barry, Thanks for the reply. I see. I am still testing it and looking for the bug. It is pretty weird that the error appears, since the code I am running is ex7 and I didn't change anything. My matrices are assembled in binary files separately. As I explained earlier, I have a series of matrices to solve, each of them have the same patterns but with different values. The way I assembly the matrix has proven to be correct for all the other cases. Only for this one, when the matrix has an zero eigenvalue, ex7 failed to solve it. In fact, I don't understand why this "Out of range" error could possibly appear? 1. my matrices have the correct parrellel lay up. 2. In ex7, I am not explicitly accessing to any elements at all! All I did is EPSSolve(); Any ideas? Best regards, Jifeng Zhao On Mon, Jul 28, 2014 at 12:07 PM, Barry Smith wrote: > > On Jul 28, 2014, at 10:27 AM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > > > Hello everyone, > > > > Are there more ideas why when using MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE), the processor seems trying to > access elements that are out of local range? > > Add the option has NOTHING to do with the ?processor seems trying to > access elements out of local range?. It is just that adding that option > allows the code to run further until it hits this other bug. So they > second bug needs to be tracked down. > > Barry > > > > > Anybody can help? Thank you so much. > > > > Best regards, > > Jifeng Zhao > > > > > > On Sun, Jul 27, 2014 at 3:37 PM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > > Thank you all for the insights. > > > > I tried to add MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE). It does solve the malloc > error. However there appears to be another error: > > > > In addition, 66324 appears to be the number of index that is assigned to > each processor. > > > > Any more insights? Thanks a lot! > > > > PETSC ERROR: Argument out of range! > > PETSC ERROR: Column too large: col 935748 max 66324! > > PETSC ERROR: > ------------------------------------------------------------------------ > > PETSC ERROR: Petsc Release Version 3.4.4, Mar, 13, 2014 > > PETSC ERROR: See docs/changes/index.html for recent updates. > > PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > PETSC ERROR: See docs/index.html for manual pages. > > PETSC ERROR: > ------------------------------------------------------------------------ > > PETSC ERROR: eigen_solver20 on a arch-linux2-c-debug named qnode4091 by > jzh953 Sun Jul 27 15:17:28 2014 > > PETSC ERROR: Libraries linked from /software/petsc/3.4.4/lib > > PETSC ERROR: Configure run at Tue Jul 8 14:20:27 2014 > > ]PETSC ERROR: Configure options --prefix=/software/petsc/3.4.4 > --with-blas-lib=/software/lapack/3.5.0/lib/libblas.so > --with-lapack-lib=/software/lapack/3.5.0/liblapack.so > --with-hdf5-dir=/software/hdf5/1.8.12 with-boost=1 > --with-boost-dir=/software/hdf5/1.8.12 --with-fftw=1 > --with-fftw-dir=/software/FFTW/3.3.3-gcc-threads > --with-valgrind-dir=/software/valgrind/3.8.1 --with-netcdf=1 > --with-netcdf-dir=/software/netCDF/4.2 --with-hdf5=1 --with-debugging=1 > --with-mpi-dir=/software/mpi/openmpi-1.6.3-gcc-4.6.3-trq4 > --download-scalapack --download-mumps --download-hypre --download-parmetis > --download-metis > > > > PETSC ERROR: > ------------------------------------------------------------------------ > > PETSC ERROR: MatSetValues_SeqSBAIJ() line 986 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/seq/sbaij.c > > PETSC ERROR: MatDisAssemble_MPISBAIJ() line 356 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mmsbaij.c > > PETSC ERROR: MatSetValues_MPISBAIJ() line 213 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > > PETSC ERROR: MatSetValues() line 1106 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/interface/matrix.c > > PETSC ERROR: MatAXPY_Basic() line 78 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > > PETSC ERROR: MatAXPY_MPISBAIJ() line 1398 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/impls/sbaij/mpi/mpisbaij.c > > PETSC ERROR: MatAXPY() line 39 in > /hpc/software/sources/builds/petsc-3.4.4/src/mat/utils/axpy.c > > PETSC ERROR: STMatGAXPY_Private() line 375 in src/st/interface/stsolve.c > > PETSC ERROR: STSetUp_Sinvert() line 138 in src/st/impls/sinvert/sinvert.c > > PETSC ERROR: STSetUp() line 294 in src/st/interface/stsolve.c > > PETSC ERROR: EPSSetUp() line 215 in src/eps/interface/setup.c > > PETSC ERROR: EPSSolve() line 90 in src/eps/interface/solve.c > > PETSC ERROR: > -------------------------------------------------------------------------- > > > > > > On Sat, Jul 26, 2014 at 12:35 PM, Jose E. Roman > wrote: > > The problem is that your stiffness and mass matrices have different > nonzero pattern. > > > > The -eps_target configuration computes a MatAXPY on the first matrix > (stiffness), so MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) could be called on this matrix, > as Barry suggests. > > > > Another workaround would be to force both of them having the same > nonzero pattern, maybe using > MatSetOption(mat,MAT_IGNORE_ZERO_ENTRIES,PETSC_FALSE). > > > > Also, this problem would not appear with target=0.0. > > > > See details on section 3.4.2 of SLEPc's users guide. > > > > Jose > > > > > > > > El 26/07/2014, a las 19:04, Barry Smith escribi?: > > > > > > > > We need the entire error message. A matrix is being formed that > wasn?t fully preallocated for so you need to determine which matrix and > make sure either 1) it gets fully preallocated or 2) MatSetOption(mat, > MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) is called on the matrix > > > > > > Barry > > > > > > On Jul 26, 2014, at 10:34 AM, jifeng zhao < > jifengzhao2015 at u.northwestern.edu> wrote: > > > > > >> Hello all, > > >> > > >> I am trying to use ex7 to solve series of generalized problem. The > running commands are: > > >> > > >> ./ex7 -f1 petsc_stiff19.dat -f2 petsc_mass19.dat -mat_type sbaij > -eps_gen_hermitian -eps_type krylovschur -eps_smallest_magnitude. > -eps_monitor_conv -st_k sp_type bcgs -st_pc_type bjacobi -st_sub_pc_type > icc -st_ksp_rtol 1.e-4 -eps_tol 1.e-4 -eps_nev 40 -st_type sinvert > > >> > > >> It works fine, but when my problem contains zero eigenvalue, it fails > to converge. > > >> > > >> Supposedly, I should use eps_target -0.1 instead of > eps_smallest_magnitude. However if eps_target is used, I run into a memory > issue: > > >> > > >> PETSC ERROR: --------------------- Error Message > ------------------------------------ > > >> PETSC ERROR: Argument out of range! > > >> PETSC ERROR: New nonzero at (0,6) caused a malloc! > > >> PETSC ERROR: > ------------------------------------------------------------------------ > > >> > > >> Does anybody know how to solve this? Thanks! > > >> > > >> Jifeng Zhao > > >> PhD candidate at Northwestern University, US > > >> Theoretical and Applied Mechanics Program > > > > > > > > > > > > > -- > > Jifeng Zhao > > PhD candidate at Northwestern University, US > > Theoretical and Applied Mechanics Program > > > > > > > > -- > > Jifeng Zhao > > PhD candidate at Northwestern University, US > > Theoretical and Applied Mechanics Program > > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Jul 28 16:52:57 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 28 Jul 2014 23:52:57 +0200 Subject: [petsc-users] eps_target malloc issue In-Reply-To: References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> Message-ID: <7379E8E2-0D4F-4FB6-B886-3469A1F9BCE8@dsic.upv.es> El 28/07/2014, a las 23:48, jifeng zhao escribi?: > Hi Barry, > > Thanks for the reply. I see. I am still testing it and looking for the bug. > > It is pretty weird that the error appears, since the code I am running is ex7 and I didn't change anything. > > My matrices are assembled in binary files separately. As I explained earlier, I have a series of matrices to solve, each of them have the same patterns but with different values. The way I assembly the matrix has proven to be correct for all the other cases. Only for this one, when the matrix has an zero eigenvalue, ex7 failed to solve it. > > In fact, I don't understand why this "Out of range" error could possibly appear? 1. my matrices have the correct parrellel lay up. 2. In ex7, I am not explicitly accessing to any elements at all! All I did is EPSSolve(); > > Any ideas? > > Best regards, > Jifeng Zhao > Tomorrow we will release slepc-3.5. I would suggest trying with the new version. If the problem persist we can help to track down the problem. Jose From jed at jedbrown.org Mon Jul 28 17:27:21 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 28 Jul 2014 16:27:21 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> Message-ID: <87zjftyv92.fsf@jedbrown.org> Lawrence Mitchell writes: > Bog-standard P1 on a pretty much regularly meshed square domain (i.e. no reentrant corners or bad elements). What interpolation is being used? The finite-element embedding should work well. >> Is there something special about the decomposition with 2 subdomains? > > It doesn't look like it, the two subdomains are about the same size. > >> Are the Chebyshev estimates far from converging? > > So for the two-level problem, if I compute the extremal eigenvalues of the preconditioned operator being used as a smoother I get (approximately): > > 1 process: > 0.019, 1.0 > > 2 processes: > 0.016, 1.4 > > 3 processes: > 0.016, 1.36 > > The eigenvalue estimates (from ksp_view) are: > > 1 process: > 0.09, 1.01 > > 2 processes: > 0.09, 1.01 > > 3 processes: > 0.13, 1.47 > > > When I bump to more levels, the estimates are only bad on two processes on the finest grid. Can you add -ksp_chebyshev_estimate_eigenvalues_random I wonder if you have a degenerate right hand side that is not exciting the largest eigenvalues on two processes. Anyway, try switching from GMRES to CG for computing the eigenvalue estimates and also using more iterations. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Tue Jul 29 07:37:47 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 06:37:47 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> Message-ID: <8761igz6g4.fsf@jedbrown.org> Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Lawrence Mitchell writes: >> On 28 Jul 2014, at 23:27, Jed Brown wrote: >> >> Lawrence Mitchell writes: >>> Bog-standard P1 on a pretty much regularly meshed square domain (i.e. no reentrant corners or bad elements). >> >> What interpolation is being used? The finite-element embedding should >> work well. > > Coarse to fine is just identity. Fine to coarse a lumped L2 projection. By "identity", do you mean in terms of continuous functions (the finite-element embedding) or something on C-points? Fine-to-coarse is generally taken to be the transpose of the natural prolongation, which is integration. >>>> Is there something special about the decomposition with 2 subdomains? >>> >>> It doesn't look like it, the two subdomains are about the same size. >>> >>>> Are the Chebyshev estimates far from converging? >>> >>> So for the two-level problem, if I compute the extremal eigenvalues of the preconditioned operator being used as a smoother I get (approximately): >>> >>> 1 process: >>> 0.019, 1.0 >>> >>> 2 processes: >>> 0.016, 1.4 >>> >>> 3 processes: >>> 0.016, 1.36 >>> >>> The eigenvalue estimates (from ksp_view) are: >>> >>> 1 process: >>> 0.09, 1.01 >>> >>> 2 processes: >>> 0.09, 1.01 >>> >>> 3 processes: >>> 0.13, 1.47 >>> >>> >>> When I bump to more levels, the estimates are only bad on two processes on the finest grid. >> >> Can you add >> >> -ksp_chebyshev_estimate_eigenvalues_random > > This doesn't help. > >> I wonder if you have a degenerate right hand side that is not exciting >> the largest eigenvalues on two processes. Anyway, try switching from >> GMRES to CG for computing the eigenvalue estimates and also using more >> iterations. > > Switching from GMRES to CG has no real effect. However bumping the number of iterations used for eigenvalue estimation from 10 to 20 on the finest grid gives me much better convergence on 2 processes. With fewer iterations the largest eigenvalue estimate gets stick around 1, once I jump to 20 it goes up to around 1.33, more in line with the exact 1.39. > > So this looks to be the original culprit. Thanks. It's odd for the estimates to require that many iterations unless the RHS is somehow degenerate. I was hoping that a random right hand side would excite the higher mode sooner. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lawrence.mitchell at imperial.ac.uk Tue Jul 29 08:26:57 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 29 Jul 2014 14:26:57 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <8761igz6g4.fsf@jedbrown.org> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> Message-ID: <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> On 29 Jul 2014, at 13:37, Jed Brown wrote: > Please always use "reply-all" so that your messages go to the list. Sorry, fat-fingered the buttons. > > Lawrence Mitchell writes: > >>> On 28 Jul 2014, at 23:27, Jed Brown wrote: >>> >>> Lawrence Mitchell writes: >>>> Bog-standard P1 on a pretty much regularly meshed square domain (i.e. no reentrant corners or bad elements). >>> >>> What interpolation is being used? The finite-element embedding should >>> work well. >> >> Coarse to fine is just identity. Fine to coarse a lumped L2 projection. > > By "identity", do you mean in terms of continuous functions (the > finite-element embedding) or something on C-points? Fine-to-coarse is > generally taken to be the transpose of the natural prolongation, which > is integration. So my coarse space is spanned by the fine one, so I copy coarse dofs to the corresponding fine ones and then linearly interpolate to get the coefficient value at the missing fine dofs. >> >> Switching from GMRES to CG has no real effect. However bumping the number of iterations used for eigenvalue estimation from 10 to 20 on the finest grid gives me much better convergence on 2 processes. With fewer iterations the largest eigenvalue estimate gets stick around 1, once I jump to 20 it goes up to around 1.33, more in line with the exact 1.39. >> >> So this looks to be the original culprit. Thanks. > > It's odd for the estimates to require that many iterations unless the > RHS is somehow degenerate. I was hoping that a random right hand side > would excite the higher mode sooner. I had another look and it turns out this does work, I was driving the options database incorrectly the first time round. I needed to run with both estimate_eigenvalues and estimate_eigenvalues_random. If I run with: -pc_type mg -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor -mg_levels_1_ksp_chebyshev_estimate_eigenvalues_random -mg_levels_1_ksp_chebyshev_estimate_eigenvalues Then I get decent estimation of the higher modes and good convergence. Thanks, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jed at jedbrown.org Tue Jul 29 08:35:05 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 07:35:05 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> Message-ID: <87wqawxp86.fsf@jedbrown.org> Lawrence Mitchell writes: > So my coarse space is spanned by the fine one, so I copy coarse dofs to the corresponding fine ones and then linearly interpolate to get the coefficient value at the missing fine dofs. Good, and is restriction the transpose? Some people mistakenly use nodal injection and not surprisingly see poor convergence because the restriction operator aliases high frequencies at full amplitude and the prolongation is accurate for neither high or low frequencies. (HPCG does this, for example.) >> It's odd for the estimates to require that many iterations unless the >> RHS is somehow degenerate. I was hoping that a random right hand side >> would excite the higher mode sooner. > > I had another look and it turns out this does work, I was driving the > options database incorrectly the first time round. I needed to run > with both estimate_eigenvalues and estimate_eigenvalues_random. > > If I run with: > > -pc_type mg -ksp_max_it 6 -pc_mg_levels 2 -ksp_monitor -mg_levels_1_ksp_chebyshev_estimate_eigenvalues_random -mg_levels_1_ksp_chebyshev_estimate_eigenvalues > > Then I get decent estimation of the higher modes and good convergence. Excellent. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lawrence.mitchell at imperial.ac.uk Tue Jul 29 10:40:11 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 29 Jul 2014 16:40:11 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <87wqawxp86.fsf@jedbrown.org> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> <87wqawxp86.fsf@jedbrown.org> Message-ID: On 29/07/14 14:35, Jed Brown wrote: > Lawrence Mitchell writes: >> So my coarse space is spanned by the fine one, so I copy coarse dofs to the corresponding fine ones and then linearly interpolate to get the coefficient value at the missing fine dofs. > > Good, and is restriction the transpose? No, I'm L2-projecting (with mass-lumping) for the restriction. So if I weren't lumping, I think this is the dual of the prolongation. The lumping seems to be alright (although I'm aware that the preconditioned operator is then not necessarily symmetric) in the sense that convergence is good irrespective of mesh resolution/MG levels. > Some people mistakenly use nodal injection and not surprisingly see poor > convergence because the restriction operator aliases high frequencies at > full amplitude and the prolongation is accurate for neither high or low > frequencies. (HPCG does this, for example.) Sure. Anyway, I think for now problem addressed. Thanks for the continued advice! Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jed at jedbrown.org Tue Jul 29 10:58:37 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 09:58:37 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> <87wqawxp86.fsf@jedbrown.org> Message-ID: <877g2wxiky.fsf@jedbrown.org> Lawrence Mitchell writes: > No, I'm L2-projecting (with mass-lumping) for the restriction. So if I > weren't lumping, I think this is the dual of the prolongation. A true L2 projection is a dense operation (involves the inverse mass matrix). But here, we're trying to map residuals, not states, so _integration_ rather than "projection" is the desired operation. Integration is the transpose of your prolongation. We can do a precise analysis, but I expect that you'll see slightly better convergence if you modify this operator. I would also recommend trying FMG soon. You should be able to converge in one F-cycle, and that will be faster than V-cycles with Krylov. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From quecat001 at gmail.com Tue Jul 29 12:02:36 2014 From: quecat001 at gmail.com (Que Cat) Date: Tue, 29 Jul 2014 12:02:36 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds Message-ID: Dear Petsc-Users, I called the SNESVISetVariableBounds and received the following error: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 .... [0]PETSC ERROR: Configure options --with-x=0 -with-pic --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 --with-scalar-type=real --with-shared-libraries=1 --with-matlab --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 --download-spai --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-superlu=1 --download-superlu --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 --with-debugging=no --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= [ 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c Could anyone give me a hint to the source of the error? Thank you. Que -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 29 12:09:06 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 11:09:06 -0600 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: Message-ID: <87oaw8w0r1.fsf@jedbrown.org> Que Cat writes: > Dear Petsc-Users, > > I called the SNESVISetVariableBounds > > and > received the following error: > > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 > .... It looks like you are truncating the error message. Please don't ever do that. > [0]PETSC ERROR: Configure options --with-x=0 -with-pic > --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 > --with-scalar-type=real --with-shared-libraries=1 --with-matlab > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 --download-spai > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps > --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs > --with-spooles=1 --download-spooles --with-superlu=1 --download-superlu > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 > --with-debugging=no > --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= > [ > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c > > Could anyone give me a hint to the source of the error? Thank you. > > Que -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From quecat001 at gmail.com Tue Jul 29 12:25:29 2014 From: quecat001 at gmail.com (Que Cat) Date: Tue, 29 Jul 2014 12:25:29 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: <87oaw8w0r1.fsf@jedbrown.org> References: <87oaw8w0r1.fsf@jedbrown.org> Message-ID: Hi Jed, I removed only one line from the error message that indicated the unnecessary information like directory path, program name, processor type. In this problem, I created the vector using VecCreateMPI, not with DM. Does it cause such problem? I have the unstructured grid and can not use DM Que On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: > Que Cat writes: > > > Dear Petsc-Users, > > > > I called the SNESVISetVariableBounds > > < > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds > > > > and > > received the following error: > > > > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > > DMShellSetCreateGlobalVector() > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for > > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 > > .... > > It looks like you are truncating the error message. Please don't ever > do that. > > > [0]PETSC ERROR: Configure options --with-x=0 -with-pic > > > --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages > > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 > > --with-scalar-type=real --with-shared-libraries=1 --with-matlab > > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 --download-spai > > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps > > --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs > > --with-spooles=1 --download-spooles --with-superlu=1 --download-superlu > > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 > > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 > > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 > > --with-debugging=no > > > --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 > > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= > > [ > > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in > > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c > > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in > > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c > > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in > > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c > > > > Could anyone give me a hint to the source of the error? Thank you. > > > > Que > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 29 12:29:46 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 11:29:46 -0600 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: <87oaw8w0r1.fsf@jedbrown.org> Message-ID: <87iomgvzsl.fsf@jedbrown.org> Que Cat writes: > Hi Jed, > > I removed only one line from the error message that indicated the > unnecessary information like directory path, program name, processor type. > > In this problem, I created the vector using VecCreateMPI, not with DM. Does > it cause such problem? I have the unstructured grid and can not use DM There should be more lines, like who is calling DMGetGlobalVector(). Maybe that is your code and you don't check errors, but why are you calling that function if you don't use DM (so the DMShell is created automatically)? > > Que > > > On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: > >> Que Cat writes: >> >> > Dear Petsc-Users, >> > >> > I called the SNESVISetVariableBounds >> > < >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds >> > >> > and >> > received the following error: >> > >> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >> > DMShellSetCreateGlobalVector() >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for >> > trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 >> > .... >> >> It looks like you are truncating the error message. Please don't ever >> do that. >> >> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic >> > >> --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages >> > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 >> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab >> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 --download-spai >> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps >> > --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs >> > --with-spooles=1 --download-spooles --with-superlu=1 --download-superlu >> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 >> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 >> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 >> > --with-debugging=no >> > >> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 >> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= >> > [ >> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c >> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c >> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c >> > >> > Could anyone give me a hint to the source of the error? Thank you. >> > >> > Que >> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From quecat001 at gmail.com Tue Jul 29 12:42:10 2014 From: quecat001 at gmail.com (Que Cat) Date: Tue, 29 Jul 2014 12:42:10 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: <87iomgvzsl.fsf@jedbrown.org> References: <87oaw8w0r1.fsf@jedbrown.org> <87iomgvzsl.fsf@jedbrown.org> Message-ID: Yes, I don't use DM at all and no information about who is calling DMGetGlobalVector(). When I use snesnewtonls (without constraint on the solution), the code worked fine. I only changed one of code : ierr = SNESSetType(snes,SNESNEWTONTR );CHKERRQ(ierr); --- > ierr = SNESSetType(snes,SNESVINEWTONSSLS );CHKERRQ(ierr); and added : ierr = SNESVISetVariableBounds(snes, low_b, up_b);CHKERRQ(ierr); and the errors appeared. Que On Tue, Jul 29, 2014 at 12:29 PM, Jed Brown wrote: > Que Cat writes: > > > Hi Jed, > > > > I removed only one line from the error message that indicated the > > unnecessary information like directory path, program name, processor > type. > > > > In this problem, I created the vector using VecCreateMPI, not with DM. > Does > > it cause such problem? I have the unstructured grid and can not use DM > > There should be more lines, like who is calling DMGetGlobalVector(). > Maybe that is your code and you don't check errors, but why are you > calling that function if you don't use DM (so the DMShell is created > automatically)? > > > > > Que > > > > > > On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: > > > >> Que Cat writes: > >> > >> > Dear Petsc-Users, > >> > > >> > I called the SNESVISetVariableBounds > >> > < > >> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds > >> > > >> > and > >> > received the following error: > >> > > >> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > >> > DMShellSetCreateGlobalVector() > >> > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > >> for > >> > trouble shooting. > >> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 > >> > .... > >> > >> It looks like you are truncating the error message. Please don't ever > >> do that. > >> > >> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic > >> > > >> > --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages > >> > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 > >> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab > >> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 > --download-spai > >> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps > >> > --with-scalapack=1 --download-scalapack --with-blacs=1 > --download-blacs > >> > --with-spooles=1 --download-spooles --with-superlu=1 > --download-superlu > >> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 > >> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 > >> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 > >> > --with-debugging=no > >> > > >> > --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 > >> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= > >> > [ > >> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in > >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c > >> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in > >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c > >> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in > >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c > >> > > >> > Could anyone give me a hint to the source of the error? Thank you. > >> > > >> > Que > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Tue Jul 29 13:12:10 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 29 Jul 2014 19:12:10 +0100 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <877g2wxiky.fsf@jedbrown.org> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> <87wqawxp86.fsf@jedbrown.org> <877g2wxiky.fsf@jedbrown.org> Message-ID: <0B6F7157-F008-41FE-BCFD-A5B42E16CA15@imperial.ac.uk> On 29 Jul 2014, at 16:58, Jed Brown wrote: > Lawrence Mitchell writes: >> No, I'm L2-projecting (with mass-lumping) for the restriction. So if I >> weren't lumping, I think this is the dual of the prolongation. > > A true L2 projection is a dense operation (involves the inverse mass > matrix). Sure, hence the lumping. > But here, we're trying to map residuals, not states, so > _integration_ rather than "projection" is the desired operation. So my approach was to transfer using projection and then use riesz representation to get the residual from the dual space back into the primal space so I can apply the operator at the next level. Is there an obvious reason why this is worse than the transpose of prolongation? The analysis I've read typically approaches from an FD angle, which doesn't always obviously (at least to me) map into FE speak. In particular, moving forward, I'm interested in GMG for H(div) problems, for which the analysis I'm aware of uses projections to move residuals between levels (for example, Arnold, Falk and Winther Num Math 85:197 (2000)). > Integration is the transpose of your prolongation. We can do a precise > analysis, but I expect that you'll see slightly better convergence if > you modify this operator. So I coded this up, I only have the matrix-free application of the prolongation, so it was not just taking the transpose. I get marginally better convergence for this problem, although it's almost a wash. > I would also recommend trying FMG soon. You should be able to converge > in one F-cycle, and that will be faster than V-cycles with Krylov. This is just -pc_mg_type full, right? Given that I only have matrix-free transfer operators do I need to explicitly set do PCMGSetRscale such that the state is restricted correctly? If I just select -pc_mg_type full, I take more than one Krylov iteration to converge. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From quecat001 at gmail.com Tue Jul 29 13:19:48 2014 From: quecat001 at gmail.com (Que Cat) Date: Tue, 29 Jul 2014 13:19:48 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: <87oaw8w0r1.fsf@jedbrown.org> <87iomgvzsl.fsf@jedbrown.org> Message-ID: I also used the ghosted vector (VecCreateGhost). Does it have any relation to the DMGetGlobalVector() ? Que On Tue, Jul 29, 2014 at 12:42 PM, Que Cat wrote: > Yes, I don't use DM at all and no information about who is calling > DMGetGlobalVector(). When I use snesnewtonls (without constraint on the > solution), the code worked fine. I only changed one of code : > ierr = SNESSetType(snes,SNESNEWTONTR );CHKERRQ(ierr); > --- > ierr = SNESSetType(snes,SNESVINEWTONSSLS );CHKERRQ(ierr); > and added : > ierr = SNESVISetVariableBounds(snes, low_b, up_b);CHKERRQ(ierr); > > and the errors appeared. > > Que > > > On Tue, Jul 29, 2014 at 12:29 PM, Jed Brown wrote: > >> Que Cat writes: >> >> > Hi Jed, >> > >> > I removed only one line from the error message that indicated the >> > unnecessary information like directory path, program name, processor >> type. >> > >> > In this problem, I created the vector using VecCreateMPI, not with DM. >> Does >> > it cause such problem? I have the unstructured grid and can not use DM >> >> There should be more lines, like who is calling DMGetGlobalVector(). >> Maybe that is your code and you don't check errors, but why are you >> calling that function if you don't use DM (so the DMShell is created >> automatically)? >> >> > >> > Que >> > >> > >> > On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: >> > >> >> Que Cat writes: >> >> >> >> > Dear Petsc-Users, >> >> > >> >> > I called the SNESVISetVariableBounds >> >> > < >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds >> >> > >> >> > and >> >> > received the following error: >> >> > >> >> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >> >> > DMShellSetCreateGlobalVector() >> >> > [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> for >> >> > trouble shooting. >> >> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 >> >> > .... >> >> >> >> It looks like you are truncating the error message. Please don't ever >> >> do that. >> >> >> >> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic >> >> > >> >> >> --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages >> >> > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 >> >> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab >> >> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 >> --download-spai >> >> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps >> >> > --with-scalapack=1 --download-scalapack --with-blacs=1 >> --download-blacs >> >> > --with-spooles=1 --download-spooles --with-superlu=1 >> --download-superlu >> >> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 >> >> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 >> >> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 >> >> > --with-debugging=no >> >> > >> >> >> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 >> >> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= >> >> > [ >> >> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in >> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c >> >> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in >> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c >> >> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in >> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c >> >> > >> >> > Could anyone give me a hint to the source of the error? Thank you. >> >> > >> >> > Que >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 29 13:21:45 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 29 Jul 2014 13:21:45 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: <87oaw8w0r1.fsf@jedbrown.org> <87iomgvzsl.fsf@jedbrown.org> Message-ID: On Tue, Jul 29, 2014 at 1:19 PM, Que Cat wrote: > I also used the ghosted vector (VecCreateGhost). Does it have any relation > to the DMGetGlobalVector() ? > We clearly have the problem that a full stack in not being printed. I am open to the idea that it is a problem in our SNESVI, but we need a way to track it down. Could you produce a test example small enough to send us? Thanks, Matt > Que > > > On Tue, Jul 29, 2014 at 12:42 PM, Que Cat wrote: > >> Yes, I don't use DM at all and no information about who is calling >> DMGetGlobalVector(). When I use snesnewtonls (without constraint on the >> solution), the code worked fine. I only changed one of code : >> ierr = SNESSetType(snes,SNESNEWTONTR );CHKERRQ(ierr); >> --- > ierr = SNESSetType(snes,SNESVINEWTONSSLS );CHKERRQ(ierr); >> and added : >> ierr = SNESVISetVariableBounds(snes, low_b, up_b);CHKERRQ(ierr); >> >> and the errors appeared. >> >> Que >> >> >> On Tue, Jul 29, 2014 at 12:29 PM, Jed Brown wrote: >> >>> Que Cat writes: >>> >>> > Hi Jed, >>> > >>> > I removed only one line from the error message that indicated the >>> > unnecessary information like directory path, program name, processor >>> type. >>> > >>> > In this problem, I created the vector using VecCreateMPI, not with DM. >>> Does >>> > it cause such problem? I have the unstructured grid and can not use DM >>> >>> There should be more lines, like who is calling DMGetGlobalVector(). >>> Maybe that is your code and you don't check errors, but why are you >>> calling that function if you don't use DM (so the DMShell is created >>> automatically)? >>> >>> > >>> > Que >>> > >>> > >>> > On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: >>> > >>> >> Que Cat writes: >>> >> >>> >> > Dear Petsc-Users, >>> >> > >>> >> > I called the SNESVISetVariableBounds >>> >> > < >>> >> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds >>> >> > >>> >> > and >>> >> > received the following error: >>> >> > >>> >> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >>> >> > DMShellSetCreateGlobalVector() >>> >> > [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >> for >>> >> > trouble shooting. >>> >> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 >>> >> > .... >>> >> >>> >> It looks like you are truncating the error message. Please don't ever >>> >> do that. >>> >> >>> >> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic >>> >> > >>> >> >>> --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages >>> >> > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 >>> >> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab >>> >> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 >>> --download-spai >>> >> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps >>> >> > --with-scalapack=1 --download-scalapack --with-blacs=1 >>> --download-blacs >>> >> > --with-spooles=1 --download-spooles --with-superlu=1 >>> --download-superlu >>> >> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 >>> >> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 >>> >> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 >>> >> > --with-debugging=no >>> >> > >>> >> >>> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 >>> >> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= >>> >> > [ >>> >> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in >>> >> > >>> /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c >>> >> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in >>> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c >>> >> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in >>> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c >>> >> > >>> >> > Could anyone give me a hint to the source of the error? Thank you. >>> >> > >>> >> > Que >>> >> >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 29 13:37:12 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Jul 2014 12:37:12 -0600 Subject: [petsc-users] Poor multigrid convergence in parallel In-Reply-To: <0B6F7157-F008-41FE-BCFD-A5B42E16CA15@imperial.ac.uk> References: <2E0E15DE-D0B0-4C8F-B2A2-261EC17AE363@imperial.ac.uk> <9CA13970-C325-4E00-974F-9D7063CF519F@imperial.ac.uk> <87k376k4ca.fsf@jedbrown.org> <87a97x5gkh.fsf@jedbrown.org> <33E8941A-694F-458C-AD80-2C7A959B3F33@imperial.ac.uk> <87zjftyv92.fsf@jedbrown.org> <8761igz6g4.fsf@jedbrown.org> <22BD05FD-4F43-4494-88EB-3763A6A89A22@imperial.ac.uk> <87wqawxp86.fsf@jedbrown.org> <877g2wxiky.fsf@jedbrown.org> <0B6F7157-F008-41FE-BCFD-A5B42E16CA15@imperial.ac.uk> Message-ID: <874my0vwo7.fsf@jedbrown.org> Lawrence Mitchell writes: > So my approach was to transfer using projection and then use riesz > representation to get the residual from the dual space back into the > primal space so I can apply the operator at the next level. Is there > an obvious reason why this is worse than the transpose of > prolongation? The analysis I've read typically approaches from an FD > angle, which doesn't always obviously (at least to me) map into FE > speak. In particular, moving forward, I'm interested in GMG for > H(div) problems, for which the analysis I'm aware of uses projections > to move residuals between levels (for example, Arnold, Falk and > Winther Num Math 85:197 (2000)). FE multigrid usually talks about embedding of spaces. If I have a continuous function f and want to evaluate \int v_i f for each coarse basis function, I can do it by quadrature directly or I can choose a large space containing v_i, integrate for all basis functions w_j in that larger space, then sum up those fine-grid contributions. This amounts to the transpose of interpolation, but is exact in the sense that it is equivalent to integrating against v_i directly using a suitable quadrature. >> Integration is the transpose of your prolongation. We can do a precise >> analysis, but I expect that you'll see slightly better convergence if >> you modify this operator. > > So I coded this up, I only have the matrix-free application of the > prolongation, so it was not just taking the transpose. I get > marginally better convergence for this problem, although it's almost a > wash. Good to hear it's at least as good. Now to make FMG work... >> I would also recommend trying FMG soon. You should be able to converge >> in one F-cycle, and that will be faster than V-cycles with Krylov. > > This is just -pc_mg_type full, right? Given that I only have > matrix-free transfer operators do I need to explicitly set do > PCMGSetRscale such that the state is restricted correctly? If you want FAS, you should implement state restriction as nodal injection. State restriction should be as accurate as possible on low-frequency modes, but does not require damping aliased high-frequency modes. > If I just select -pc_mg_type full, I take more than one Krylov > iteration to converge. You have to compare to discretization error. For example, HPGMG-FE computes errors against an analytic solution, and delivers the design third-order accuracy using 3,1 cycles. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From abhyshr at mcs.anl.gov Tue Jul 29 14:06:49 2014 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Tue, 29 Jul 2014 19:06:49 +0000 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: Message-ID: This error is also seen in src/snes/examples/tutorials/ex53.c. [Shri at Shrirangs-MacBook-Pro tutorials (master)]$ ./ex53 Vec Object: 1 MPI processes type: seq 1 1 1 1 1 -1 -1 0 -1 -1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 [0]PETSC ERROR: ./ex53 on a debug-master named Shrirangs-MacBook-Pro.local by Shri Tue Jul 29 14:00:33 2014 [0]PETSC ERROR: Configure options PETSC_ARCH=debug-master [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in /Users/Shri/Documents/petsc/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in /Users/Shri/Documents/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in /Users/Shri/Documents/petsc/src/dm/interface/dmget.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix is missing diagonal entry 0 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: 2014-07-07 14:50:15 +0200 [0]PETSC ERROR: ./ex53 on a debug-master named Shrirangs-MacBook-Pro.local by Shri Tue Jul 29 14:00:33 2014 [0]PETSC ERROR: Configure options PETSC_ARCH=debug-master [0]PETSC ERROR: #4 MatILUFactorSymbolic_SeqAIJ() line 1738 in /Users/Shri/Documents/petsc/src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: #5 MatILUFactorSymbolic() line 6203 in /Users/Shri/Documents/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #6 PCSetUp_ILU() line 206 in /Users/Shri/Documents/petsc/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #7 PCSetUp() line 902 in /Users/Shri/Documents/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 305 in /Users/Shri/Documents/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 SNESSolve_VINEWTONRSLS() line 506 in /Users/Shri/Documents/petsc/src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: #10 SNESSolve() line 3743 in /Users/Shri/Documents/petsc/src/snes/interface/snes.c [0]PETSC ERROR: #11 main() line 49 in /Users/Shri/Documents/petsc/src/snes/examples/tutorials/ex53.c [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 From: Matthew Knepley Date: Tue, 29 Jul 2014 13:21:45 -0500 To: Que Cat Cc: petsc-users Subject: Re: [petsc-users] Error with SNESVISetVariableBounds >On Tue, Jul 29, 2014 at 1:19 PM, Que Cat wrote: > >I also used the ghosted vector (VecCreateGhost). Does it have any >relation to the DMGetGlobalVector() ? > > > >We clearly have the problem that a full stack in not being printed. I am >open to the idea that it is a problem >in our SNESVI, but we need a way to track it down. Could you produce a >test example small enough to send >us? > > Thanks, > > Matt > > >Que > > > >On Tue, Jul 29, 2014 at 12:42 PM, Que Cat wrote: > >Yes, I don't use DM at all and no information about who is calling >DMGetGlobalVector(). When I use snesnewtonls (without constraint on the >solution), the code worked fine. I only changed one of code : > > >ierr = SNESSetType(snes,SNESNEWTONTR );CHKERRQ(ierr); >--- > ierr = SNESSetType(snes,SNESVINEWTONSSLS );CHKERRQ(ierr); >and added : >ierr = SNESVISetVariableBounds(snes, low_b, up_b);CHKERRQ(ierr); > > >and the errors appeared. > >Que > > >On Tue, Jul 29, 2014 at 12:29 PM, Jed Brown wrote: > >Que Cat writes: > >> Hi Jed, >> >> I removed only one line from the error message that indicated the >> unnecessary information like directory path, program name, processor >>type. >> >> In this problem, I created the vector using VecCreateMPI, not with DM. >>Does >> it cause such problem? I have the unstructured grid and can not use DM > > >There should be more lines, like who is calling DMGetGlobalVector(). >Maybe that is your code and you don't check errors, but why are you >calling that function if you don't use DM (so the DMShell is created >automatically)? > >> >> Que >> >> >> On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: >> >>> Que Cat writes: >>> >>> > Dear Petsc-Users, >>> > >>> > I called the SNESVISetVariableBounds >>> > < >>> >>>http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVIS >>>etVariableBounds.html#SNESVISetVariableBounds > > > >>> > >>> > and >>> > received the following error: >>> > >>> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >>> > DMShellSetCreateGlobalVector() >>> > [0]PETSC ERROR: See >>>http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for >>> > trouble shooting. >>> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 >>> > .... >>> >>> It looks like you are truncating the error message. Please don't ever >>> do that. >>> >>> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic >>> > >>> >>>--with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/ex >>>ternalpackages >>> > --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 >>> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab >>> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 >>>--download-spai >>> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps >>> > --with-scalapack=1 --download-scalapack --with-blacs=1 >>>--download-blacs >>> > --with-spooles=1 --download-spooles --with-superlu=1 >>>--download-superlu >>> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 >>> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 >>> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 >>> > --with-debugging=no >>> > >>> >>>--with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib >>>/intel64 >>> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= >>> > [ >>> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in >>> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c >>> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in >>> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c >>> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in >>> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c >>> > >>> > Could anyone give me a hint to the source of the error? Thank you. >>> > >>> > Que >>> > > > > > > > > > > > > > > > > > > > > > >-- >What most experimenters take for granted before they begin their >experiments is infinitely more interesting than any results to which >their experiments lead. >-- Norbert Wiener From quecat001 at gmail.com Tue Jul 29 14:22:05 2014 From: quecat001 at gmail.com (Que Cat) Date: Tue, 29 Jul 2014 14:22:05 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: Message-ID: I received the same error message for example 53 in my machine as well. ./ex53 Vec Object: 1 MPI processes type: seq 1 1 1 1 1 -1 -1 0 -1 -1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 [0]PETSC ERROR: ./ex53 on a arch-linux2-c-debug named qc by qc Tue Jul 29 14:20:17 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-parmetis --download-metis --download-hdf5 [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in /home/qc/bin/petsc-3.5.1/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in /home/qc/bin/petsc-3.5.1/src/dm/interface/dm.c [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in /home/qc/bin/petsc-3.5.1/src/dm/interface/dmget.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix is missing diagonal entry 0 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 [0]PETSC ERROR: ./ex53 on a arch-linux2-c-debug named qc by qc Tue Jul 29 14:20:17 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-parmetis --download-metis --download-hdf5 [0]PETSC ERROR: #4 MatILUFactorSymbolic_SeqAIJ() line 1738 in /home/qc/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: #5 MatILUFactorSymbolic() line 6203 in /home/qc/bin/petsc-3.5.1/src/mat/interface/matrix.c [0]PETSC ERROR: #6 PCSetUp_ILU() line 206 in /home/qc/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #7 PCSetUp() line 902 in /home/qc/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 305 in /home/qc/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 SNESSolve_VINEWTONRSLS() line 506 in /home/qc/bin/petsc-3.5.1/src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: #10 SNESSolve() line 3743 in /home/qc/bin/petsc-3.5.1/src/snes/interface/snes.c [0]PETSC ERROR: #11 main() line 49 in src/main.cpp [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From altriaex86 at gmail.com Wed Jul 30 01:20:13 2014 From: altriaex86 at gmail.com (=?UTF-8?B?5byg5Zu954aZ?=) Date: Wed, 30 Jul 2014 16:20:13 +1000 Subject: [petsc-users] same code, different machine, different performance Message-ID: Hi,all Recently I am using SLEPc to solve an sparse eigenvalue problem. I use krylov-shur and mumps as solvers. My codes work well my corei5 dual core laptop. Compared to my old sequential solution ARPACK/UMFPACK, running with 2 process could give me 1.3x speedup. However when I install my code on another station with quad core CPU, the performance change significantly. My parallel code runs 2-3 times slower than sequential code. Is there any one who knows what happened here? Thanks a lot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Jul 30 01:28:40 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 30 Jul 2014 00:28:40 -0600 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: References: Message-ID: <87a97ruzqf.fsf@jedbrown.org> ??? writes: > Hi,all > > Recently I am using SLEPc to solve an sparse eigenvalue problem. I use > krylov-shur and mumps as solvers. > > My codes work well my corei5 dual core laptop. Compared to my old > sequential solution ARPACK/UMFPACK, running with 2 process could give me > 1.3x speedup. However when I install my code on another station with quad > core CPU, the performance change significantly. My parallel code runs 2-3 > times slower than sequential code. Run "make streams NPMAX=4" and send the output. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From altriaex86 at gmail.com Wed Jul 30 02:10:13 2014 From: altriaex86 at gmail.com (=?UTF-8?B?5byg5Zu954aZ?=) Date: Wed, 30 Jul 2014 17:10:13 +1000 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: <877g2vuz02.fsf@jedbrown.org> References: <87a97ruzqf.fsf@jedbrown.org> <877g2vuz02.fsf@jedbrown.org> Message-ID: I am using putty so I could only give you screenshots. The attachments are full output. The output [image: ???? 1] 2014-07-30 16:44 GMT+10:00 Jed Brown : > ??? writes: > > > Do you mean compling and running the file "noname" attached? > > Is it a source code? After opened it I only see the following. > > > > *-----BEGIN PGP SIGNATURE-----* > > It isn't called "noname", it is a PGP/MIME signature > (application/pgp-signature) and your mail client refuses to recognize a > 15-year old standard and instead writes "noname". Just ignore my > signatures and do what I suggested, then reply-all (do not drop > petsc-users). > > > *Version: GnuPG v2* > > > > *iQIcBAEBAgAGBQJT2JCbAAoJEM+2iNHeMalNOhQQAJ9xVEBPCJhEB3PkVhCwNg/e* > > *djetm4dAcWoRtv262lGU+yoe1W88WVNQBrHyJCXn7SBZ7sYvUM7lUgDxbBhWIILj* > > *CwasnKDj2/r/TjmNmxwaP/xtKAJya6lfzKCQr6wymOQRCikGGCKZxNz+XfD5HI7j* > > *wF9kjfUI/oDTm7beBi21z+GXaDHx0uq9rIIrn2kG2Bkv2OFYbAa/YKi6SuPxF7aa* > > *kxbEchusp9WPChBqlmB6e+E991YwvUPQuaeXdN9do1YLuk9SPZYVmVbj7fdBrkB6* > > *tfgoJIQW80mOCJFu13HqA22MzaQikaztW4B8zRguL1YIDhKmu+moi8gvlbkH+HjZ* > > *L+PuIIKaozlkiHSy7KzhHpP8m6JuJqNoI7nkMwd2Ye4AF9xUFBiuzf426xafroF0* > > *cLXLR1AMFXpRy4fRwZpiFdiyu4ats6jPOQ2WsJhS147WldShG+dMgo6acWaWq30v* > > *RPqyWJV478cjZ8ZFsf6hdHoiMjfqPDxi4XF+b0+OeY+gM8CG+1h9HYpo6E+ALo1Q* > > *IVKLI5cLfvshACrCPY1vnehDYSJMCPD91X+zlfjm/wrWb4Ci+RH+WGovIW0Smb72* > > *NaGu1EDqZnUIFbjQA8wX1/E31uwVmr18Z2OJnqaWaas4Fnc53aro02Lya2Xn7/yf* > > */kBlARwztc6/U6oUHFf+* > > *=qGvR* > > *-----END PGP SIGNATURE-----* > > > > > > 2014-07-30 16:28 GMT+10:00 Jed Brown : > > > >> ??? writes: > >> > >> > Hi,all > >> > > >> > Recently I am using SLEPc to solve an sparse eigenvalue problem. I use > >> > krylov-shur and mumps as solvers. > >> > > >> > My codes work well my corei5 dual core laptop. Compared to my old > >> > sequential solution ARPACK/UMFPACK, running with 2 process could give > me > >> > 1.3x speedup. However when I install my code on another station with > quad > >> > core CPU, the performance change significantly. My parallel code runs > 2-3 > >> > times slower than sequential code. > >> > >> Run "make streams NPMAX=4" and send the output. > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68095 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1.png Type: image/png Size: 147467 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2.png Type: image/png Size: 72913 bytes Desc: not available URL: From altriaex86 at gmail.com Wed Jul 30 02:11:03 2014 From: altriaex86 at gmail.com (=?UTF-8?B?5byg5Zu954aZ?=) Date: Wed, 30 Jul 2014 17:11:03 +1000 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: References: <87a97ruzqf.fsf@jedbrown.org> <877g2vuz02.fsf@jedbrown.org> Message-ID: And here is the lscpu output [image: ???? 1] 2014-07-30 17:10 GMT+10:00 ??? : > I am using putty so I could only give you screenshots. > The attachments are full output. > The output > [image: ???? 1] > > > 2014-07-30 16:44 GMT+10:00 Jed Brown : > > ??? writes: >> >> > Do you mean compling and running the file "noname" attached? >> > Is it a source code? After opened it I only see the following. >> > >> > *-----BEGIN PGP SIGNATURE-----* >> >> It isn't called "noname", it is a PGP/MIME signature >> (application/pgp-signature) and your mail client refuses to recognize a >> 15-year old standard and instead writes "noname". Just ignore my >> signatures and do what I suggested, then reply-all (do not drop >> petsc-users). >> >> > *Version: GnuPG v2* >> > >> > *iQIcBAEBAgAGBQJT2JCbAAoJEM+2iNHeMalNOhQQAJ9xVEBPCJhEB3PkVhCwNg/e* >> > *djetm4dAcWoRtv262lGU+yoe1W88WVNQBrHyJCXn7SBZ7sYvUM7lUgDxbBhWIILj* >> > *CwasnKDj2/r/TjmNmxwaP/xtKAJya6lfzKCQr6wymOQRCikGGCKZxNz+XfD5HI7j* >> > *wF9kjfUI/oDTm7beBi21z+GXaDHx0uq9rIIrn2kG2Bkv2OFYbAa/YKi6SuPxF7aa* >> > *kxbEchusp9WPChBqlmB6e+E991YwvUPQuaeXdN9do1YLuk9SPZYVmVbj7fdBrkB6* >> > *tfgoJIQW80mOCJFu13HqA22MzaQikaztW4B8zRguL1YIDhKmu+moi8gvlbkH+HjZ* >> > *L+PuIIKaozlkiHSy7KzhHpP8m6JuJqNoI7nkMwd2Ye4AF9xUFBiuzf426xafroF0* >> > *cLXLR1AMFXpRy4fRwZpiFdiyu4ats6jPOQ2WsJhS147WldShG+dMgo6acWaWq30v* >> > *RPqyWJV478cjZ8ZFsf6hdHoiMjfqPDxi4XF+b0+OeY+gM8CG+1h9HYpo6E+ALo1Q* >> > *IVKLI5cLfvshACrCPY1vnehDYSJMCPD91X+zlfjm/wrWb4Ci+RH+WGovIW0Smb72* >> > *NaGu1EDqZnUIFbjQA8wX1/E31uwVmr18Z2OJnqaWaas4Fnc53aro02Lya2Xn7/yf* >> > */kBlARwztc6/U6oUHFf+* >> > *=qGvR* >> > *-----END PGP SIGNATURE-----* >> > >> > >> > 2014-07-30 16:28 GMT+10:00 Jed Brown : >> > >> >> ??? writes: >> >> >> >> > Hi,all >> >> > >> >> > Recently I am using SLEPc to solve an sparse eigenvalue problem. I >> use >> >> > krylov-shur and mumps as solvers. >> >> > >> >> > My codes work well my corei5 dual core laptop. Compared to my old >> >> > sequential solution ARPACK/UMFPACK, running with 2 process could >> give me >> >> > 1.3x speedup. However when I install my code on another station with >> quad >> >> > core CPU, the performance change significantly. My parallel code >> runs 2-3 >> >> > times slower than sequential code. >> >> >> >> Run "make streams NPMAX=4" and send the output. >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 24559 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68095 bytes Desc: not available URL: From knepley at gmail.com Wed Jul 30 05:54:58 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jul 2014 05:54:58 -0500 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: References: <87a97ruzqf.fsf@jedbrown.org> <877g2vuz02.fsf@jedbrown.org> Message-ID: On Wed, Jul 30, 2014 at 2:10 AM, ??? wrote: > I am using putty so I could only give you screenshots. > The attachments are full output. > As we can see below, the memory bus on this machine can only really drive 1.5 processors. For that algorithm, you will see some speedup on 2 procs, and nothing after that. It would be nice if manufacturers reported STREAMS along with their CPU benchmarks. Thanks, Matt > The output > [image: ???? 1] > > > 2014-07-30 16:44 GMT+10:00 Jed Brown : > >> ??? writes: >> >> > Do you mean compling and running the file "noname" attached? >> > Is it a source code? After opened it I only see the following. >> > >> > *-----BEGIN PGP SIGNATURE-----* >> >> It isn't called "noname", it is a PGP/MIME signature >> (application/pgp-signature) and your mail client refuses to recognize a >> 15-year old standard and instead writes "noname". Just ignore my >> signatures and do what I suggested, then reply-all (do not drop >> petsc-users). >> >> > *Version: GnuPG v2* >> > >> > *iQIcBAEBAgAGBQJT2JCbAAoJEM+2iNHeMalNOhQQAJ9xVEBPCJhEB3PkVhCwNg/e* >> > *djetm4dAcWoRtv262lGU+yoe1W88WVNQBrHyJCXn7SBZ7sYvUM7lUgDxbBhWIILj* >> > *CwasnKDj2/r/TjmNmxwaP/xtKAJya6lfzKCQr6wymOQRCikGGCKZxNz+XfD5HI7j* >> > *wF9kjfUI/oDTm7beBi21z+GXaDHx0uq9rIIrn2kG2Bkv2OFYbAa/YKi6SuPxF7aa* >> > *kxbEchusp9WPChBqlmB6e+E991YwvUPQuaeXdN9do1YLuk9SPZYVmVbj7fdBrkB6* >> > *tfgoJIQW80mOCJFu13HqA22MzaQikaztW4B8zRguL1YIDhKmu+moi8gvlbkH+HjZ* >> > *L+PuIIKaozlkiHSy7KzhHpP8m6JuJqNoI7nkMwd2Ye4AF9xUFBiuzf426xafroF0* >> > *cLXLR1AMFXpRy4fRwZpiFdiyu4ats6jPOQ2WsJhS147WldShG+dMgo6acWaWq30v* >> > *RPqyWJV478cjZ8ZFsf6hdHoiMjfqPDxi4XF+b0+OeY+gM8CG+1h9HYpo6E+ALo1Q* >> > *IVKLI5cLfvshACrCPY1vnehDYSJMCPD91X+zlfjm/wrWb4Ci+RH+WGovIW0Smb72* >> > *NaGu1EDqZnUIFbjQA8wX1/E31uwVmr18Z2OJnqaWaas4Fnc53aro02Lya2Xn7/yf* >> > */kBlARwztc6/U6oUHFf+* >> > *=qGvR* >> > *-----END PGP SIGNATURE-----* >> > >> > >> > 2014-07-30 16:28 GMT+10:00 Jed Brown : >> >> > >> >> ??? writes: >> >> >> >> > Hi,all >> >> > >> >> > Recently I am using SLEPc to solve an sparse eigenvalue problem. I >> use >> >> > krylov-shur and mumps as solvers. >> >> > >> >> > My codes work well my corei5 dual core laptop. Compared to my old >> >> > sequential solution ARPACK/UMFPACK, running with 2 process could >> give me >> >> > 1.3x speedup. However when I install my code on another station with >> quad >> >> > core CPU, the performance change significantly. My parallel code >> runs 2-3 >> >> > times slower than sequential code. >> >> >> >> Run "make streams NPMAX=4" and send the output. >> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68095 bytes Desc: not available URL: From khaipham at utexas.edu Wed Jul 30 07:26:28 2014 From: khaipham at utexas.edu (Khai Hong Pham) Date: Wed, 30 Jul 2014 07:26:28 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: Message-ID: Does anyone have an idea about the error of example 53 reported by Abhyankar ? Thanks On Tue, Jul 29, 2014 at 2:22 PM, Que Cat wrote: > I received the same error message for example 53 in my machine as well. > > ./ex53 > Vec Object: 1 MPI processes > type: seq > 1 > 1 > 1 > 1 > 1 > -1 > -1 > 0 > -1 > -1 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 > [0]PETSC ERROR: ./ex53 on a arch-linux2-c-debug named qc by qc Tue Jul 29 > 14:20:17 2014 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > --download-parmetis --download-metis --download-hdf5 > [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in > /home/qc/bin/petsc-3.5.1/src/dm/impls/shell/dmshell.c > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in > /home/qc/bin/petsc-3.5.1/src/dm/interface/dm.c > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in > /home/qc/bin/petsc-3.5.1/src/dm/interface/dmget.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 > [0]PETSC ERROR: ./ex53 on a arch-linux2-c-debug named qc by qc Tue Jul 29 > 14:20:17 2014 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > --download-parmetis --download-metis --download-hdf5 > [0]PETSC ERROR: #4 MatILUFactorSymbolic_SeqAIJ() line 1738 in > /home/qc/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: #5 MatILUFactorSymbolic() line 6203 in > /home/qc/bin/petsc-3.5.1/src/mat/interface/matrix.c > [0]PETSC ERROR: #6 PCSetUp_ILU() line 206 in > /home/qc/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #7 PCSetUp() line 902 in > /home/qc/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 305 in > /home/qc/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 SNESSolve_VINEWTONRSLS() line 506 in > /home/qc/bin/petsc-3.5.1/src/snes/impls/vi/rs/virs.c > [0]PETSC ERROR: #10 SNESSolve() line 3743 in > /home/qc/bin/petsc-3.5.1/src/snes/interface/snes.c > [0]PETSC ERROR: #11 main() line 49 in src/main.cpp > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From altriaex86 at gmail.com Wed Jul 30 07:32:46 2014 From: altriaex86 at gmail.com (=?UTF-8?B?5byg5Zu954aZ?=) Date: Wed, 30 Jul 2014 22:32:46 +1000 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: References: <87a97ruzqf.fsf@jedbrown.org> <877g2vuz02.fsf@jedbrown.org> Message-ID: Thank you for answering. I have one more thing to ask. The first computer I used is i5 2.4g dualcore. The second I used is i7 3.2g quad core. Both sequencial case. My codes tend to be slower on the second one(compared with a same ARPACK/UMFPACK combination) but faster on the first one. I thought at least the two comparison should be similar for the second machine is faster. Is there any options that may affect performance on different machine in PETSc or SLEPc? Thanks a lot. 2014?7?30? ??8:54? "Matthew Knepley" ??? > On Wed, Jul 30, 2014 at 2:10 AM, ??? wrote: > >> I am using putty so I could only give you screenshots. >> The attachments are full output. >> > > As we can see below, the memory bus on this machine can only really drive > 1.5 processors. For that algorithm, you will see some speedup on 2 procs, > and nothing after that. It would be nice if manufacturers reported STREAMS > along with their CPU benchmarks. > > Thanks, > > Matt > > >> The output >> [image: ???? 1] >> >> >> 2014-07-30 16:44 GMT+10:00 Jed Brown : >> >>> ??? writes: >>> >>> > Do you mean compling and running the file "noname" attached? >>> > Is it a source code? After opened it I only see the following. >>> > >>> > *-----BEGIN PGP SIGNATURE-----* >>> >>> It isn't called "noname", it is a PGP/MIME signature >>> (application/pgp-signature) and your mail client refuses to recognize a >>> 15-year old standard and instead writes "noname". Just ignore my >>> signatures and do what I suggested, then reply-all (do not drop >>> petsc-users). >>> >>> > *Version: GnuPG v2* >>> > >>> > *iQIcBAEBAgAGBQJT2JCbAAoJEM+2iNHeMalNOhQQAJ9xVEBPCJhEB3PkVhCwNg/e* >>> > *djetm4dAcWoRtv262lGU+yoe1W88WVNQBrHyJCXn7SBZ7sYvUM7lUgDxbBhWIILj* >>> > *CwasnKDj2/r/TjmNmxwaP/xtKAJya6lfzKCQr6wymOQRCikGGCKZxNz+XfD5HI7j* >>> > *wF9kjfUI/oDTm7beBi21z+GXaDHx0uq9rIIrn2kG2Bkv2OFYbAa/YKi6SuPxF7aa* >>> > *kxbEchusp9WPChBqlmB6e+E991YwvUPQuaeXdN9do1YLuk9SPZYVmVbj7fdBrkB6* >>> > *tfgoJIQW80mOCJFu13HqA22MzaQikaztW4B8zRguL1YIDhKmu+moi8gvlbkH+HjZ* >>> > *L+PuIIKaozlkiHSy7KzhHpP8m6JuJqNoI7nkMwd2Ye4AF9xUFBiuzf426xafroF0* >>> > *cLXLR1AMFXpRy4fRwZpiFdiyu4ats6jPOQ2WsJhS147WldShG+dMgo6acWaWq30v* >>> > *RPqyWJV478cjZ8ZFsf6hdHoiMjfqPDxi4XF+b0+OeY+gM8CG+1h9HYpo6E+ALo1Q* >>> > *IVKLI5cLfvshACrCPY1vnehDYSJMCPD91X+zlfjm/wrWb4Ci+RH+WGovIW0Smb72* >>> > *NaGu1EDqZnUIFbjQA8wX1/E31uwVmr18Z2OJnqaWaas4Fnc53aro02Lya2Xn7/yf* >>> > */kBlARwztc6/U6oUHFf+* >>> > *=qGvR* >>> > *-----END PGP SIGNATURE-----* >>> > >>> > >>> > 2014-07-30 16:28 GMT+10:00 Jed Brown : >>> >>> > >>> >> ??? writes: >>> >> >>> >> > Hi,all >>> >> > >>> >> > Recently I am using SLEPc to solve an sparse eigenvalue problem. I >>> use >>> >> > krylov-shur and mumps as solvers. >>> >> > >>> >> > My codes work well my corei5 dual core laptop. Compared to my old >>> >> > sequential solution ARPACK/UMFPACK, running with 2 process could >>> give me >>> >> > 1.3x speedup. However when I install my code on another station >>> with quad >>> >> > core CPU, the performance change significantly. My parallel code >>> runs 2-3 >>> >> > times slower than sequential code. >>> >> >>> >> Run "make streams NPMAX=4" and send the output. >>> >> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 68095 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Jul 30 09:41:34 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 30 Jul 2014 09:41:34 -0500 Subject: [petsc-users] same code, different machine, different performance In-Reply-To: References: <87a97ruzqf.fsf@jedbrown.org> <877g2vuz02.fsf@jedbrown.org> Message-ID: On Wed, 30 Jul 2014, ??? wrote: > I am using putty so I could only give you screenshots. You can copy/paste text from putty. [and even copy/paste from windows 'cmd'] Satish From khaipham at utexas.edu Wed Jul 30 10:41:19 2014 From: khaipham at utexas.edu (Khai Hong Pham) Date: Wed, 30 Jul 2014 10:41:19 -0500 Subject: [petsc-users] Error with SNESVISetVariableBounds In-Reply-To: References: <87oaw8w0r1.fsf@jedbrown.org> <87iomgvzsl.fsf@jedbrown.org> Message-ID: Dear Matt, I tried to find the source of the error from ex53. If I add *"CHKERRQ(ierr);"* after line 926 in the file src/ksp/ksp/interface/iterativ.c : "*ierr = DMGetGlobalVector(ksp->dm,&vecr); /* don't check for errors -- if any errors, pass down to next block */ *". The error message becomes: ---------------------------------------- ./ex53 Vec Object: 1 MPI processes type: seq 1 1 1 1 1 -1 -1 0 -1 -1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 [0]PETSC ERROR: ./ex53 on a arch-linux2-c-debug named khp by khp Wed Jul 30 10:24:05 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-parmetis --download-metis [0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in /home/petsc-3.5.1/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in /home/petsc-3.5.1/src/dm/interface/dm.c [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in /home/petsc-3.5.1/src/dm/interface/dmget.c [0]PETSC ERROR: #4 KSPGetVecs() line 926 in /home/petsc-3.5.1/src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: #5 KSPSetUp_GMRES() line 87 in /home/petsc-3.5.1/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #6 KSPSetUp() line 268 in /home/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 SNESSolve_VINEWTONRSLS() line 507 in /home/petsc-3.5.1/src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: #8 SNESSolve() line 3743 in /home/petsc-3.5.1/src/snes/interface/snes.c [0]PETSC ERROR: #9 main() line 49 in src/main.cpp [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- ---------------------------- Does it say anything about the source of the error? Thanks. Khai On Tue, Jul 29, 2014 at 1:21 PM, Matthew Knepley wrote: > On Tue, Jul 29, 2014 at 1:19 PM, Que Cat wrote: > >> I also used the ghosted vector (VecCreateGhost). Does it have any >> relation to the DMGetGlobalVector() ? >> > > We clearly have the problem that a full stack in not being printed. I am > open to the idea that it is a problem > in our SNESVI, but we need a way to track it down. Could you produce a > test example small enough to send > us? > > Thanks, > > Matt > > >> Que >> >> >> On Tue, Jul 29, 2014 at 12:42 PM, Que Cat wrote: >> >>> Yes, I don't use DM at all and no information about who is calling >>> DMGetGlobalVector(). When I use snesnewtonls (without constraint on the >>> solution), the code worked fine. I only changed one of code : >>> ierr = SNESSetType(snes,SNESNEWTONTR );CHKERRQ(ierr); >>> --- > ierr = SNESSetType(snes,SNESVINEWTONSSLS );CHKERRQ(ierr); >>> and added : >>> ierr = SNESVISetVariableBounds(snes, low_b, up_b);CHKERRQ(ierr); >>> >>> and the errors appeared. >>> >>> Que >>> >>> >>> On Tue, Jul 29, 2014 at 12:29 PM, Jed Brown wrote: >>> >>>> Que Cat writes: >>>> >>>> > Hi Jed, >>>> > >>>> > I removed only one line from the error message that indicated the >>>> > unnecessary information like directory path, program name, processor >>>> type. >>>> > >>>> > In this problem, I created the vector using VecCreateMPI, not with >>>> DM. Does >>>> > it cause such problem? I have the unstructured grid and can not use DM >>>> >>>> There should be more lines, like who is calling DMGetGlobalVector(). >>>> Maybe that is your code and you don't check errors, but why are you >>>> calling that function if you don't use DM (so the DMShell is created >>>> automatically)? >>>> >>>> > >>>> > Que >>>> > >>>> > >>>> > On Tue, Jul 29, 2014 at 12:09 PM, Jed Brown wrote: >>>> > >>>> >> Que Cat writes: >>>> >> >>>> >> > Dear Petsc-Users, >>>> >> > >>>> >> > I called the SNESVISetVariableBounds >>>> >> > < >>>> >> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESVISetVariableBounds.html#SNESVISetVariableBounds >>>> >> > >>>> >> > and >>>> >> > received the following error: >>>> >> > >>>> >> > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >>>> >> > DMShellSetCreateGlobalVector() >>>> >> > [0]PETSC ERROR: See >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> >> for >>>> >> > trouble shooting. >>>> >> > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 >>>> >> > .... >>>> >> >>>> >> It looks like you are truncating the error message. Please don't >>>> ever >>>> >> do that. >>>> >> >>>> >> > [0]PETSC ERROR: Configure options --with-x=0 -with-pic >>>> >> > >>>> >> >>>> --with-external-packages-dir=/opt/apps/intel13/mvapich2_1_9/petsc/3.5/externalpackages >>>> >> > --with-mpi-compilers=1 >>>> --with-mpi-dir=/opt/apps/intel13/mvapich2/1.9 >>>> >> > --with-scalar-type=real --with-shared-libraries=1 --with-matlab >>>> >> > --with-matlab-dir=/work/apps/matlab/2013a --with-spai=1 >>>> --download-spai >>>> >> > --with-hypre=1 --download-hypre --with-mumps=1 --download-mumps >>>> >> > --with-scalapack=1 --download-scalapack --with-blacs=1 >>>> --download-blacs >>>> >> > --with-spooles=1 --download-spooles --with-superlu=1 >>>> --download-superlu >>>> >> > --with-superlu_dist=1 --download-superlu_dist --with-parmetis=1 >>>> >> > --download-parmetis --with-metis=1 --download-metis --with-hdf5=1 >>>> >> > --with-hdf5-dir=/opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 >>>> >> > --with-debugging=no >>>> >> > >>>> >> >>>> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64 >>>> >> > --with-mpiexec=mpirun_rsh --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS= >>>> >> > [ >>>> >> > 0]PETSC ERROR: #1 DMCreateGlobalVector_Shell() line 245 in >>>> >> > >>>> /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/impls/shell/dmshell.c >>>> >> > [0]PETSC ERROR: #2 DMCreateGlobalVector() line 681 in >>>> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dm.c >>>> >> > [0]PETSC ERROR: #3 DMGetGlobalVector() line 154 in >>>> >> > /opt/apps/intel13/mvapich2_1_9/petsc/3.5/src/dm/interface/dmget.c >>>> >> > >>>> >> > Could anyone give me a hint to the source of the error? Thank you. >>>> >> > >>>> >> > Que >>>> >> >>>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Wed Jul 30 11:08:20 2014 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Wed, 30 Jul 2014 09:08:20 -0700 Subject: [petsc-users] Partition of parallel AIJ sparce matrix Message-ID: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> Hello, ? Does a process have to own consecutive rows of the matrix? For example,?suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own?rows 1 and?2, process 2 own rows 3 and 4? Or process1 may own rows 1 and 3, and process?2?own row 2 and 4? ? Thanks?a?lot for your help! ? Regards, Qin? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hong at aspiritech.org Wed Jul 30 13:16:43 2014 From: hong at aspiritech.org (hong at aspiritech.org) Date: Wed, 30 Jul 2014 13:16:43 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: <7379E8E2-0D4F-4FB6-B886-3469A1F9BCE8@dsic.upv.es> References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> <7379E8E2-0D4F-4FB6-B886-3469A1F9BCE8@dsic.upv.es> Message-ID: Jifeng, Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2 $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type aij -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1 -st_ksp_type minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2 -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee log.mat20.ex7.np32.tol-2 The output is attached. When using '-mat_type sbaij', I got same error as reported, which might be a bug in our libraries. I'll investigate it. Switching to '-mat_type aij' works (I'm not sure if the computed eigenvalues are meaningful though), but the linear solver minres/bjacobi/icc converges very slow, takes around 500 - 1000 linear iterations for each eps iteration. You may experiment with stronger pc_type, e.g., '-pc_type asm' etc. I'll let you know why sbaij crashes. Hong On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman wrote: > > El 28/07/2014, a las 23:48, jifeng zhao escribi?: > >> Hi Barry, >> >> Thanks for the reply. I see. I am still testing it and looking for the bug. >> >> It is pretty weird that the error appears, since the code I am running is ex7 and I didn't change anything. >> >> My matrices are assembled in binary files separately. As I explained earlier, I have a series of matrices to solve, each of them have the same patterns but with different values. The way I assembly the matrix has proven to be correct for all the other cases. Only for this one, when the matrix has an zero eigenvalue, ex7 failed to solve it. >> >> In fact, I don't understand why this "Out of range" error could possibly appear? 1. my matrices have the correct parrellel lay up. 2. In ex7, I am not explicitly accessing to any elements at all! All I did is EPSSolve(); >> >> Any ideas? >> >> Best regards, >> Jifeng Zhao >> > > Tomorrow we will release slepc-3.5. I would suggest trying with the new version. If the problem persist we can help to track down the problem. > > Jose > -------------- next part -------------- A non-text attachment was scrubbed... Name: log.mat20.ex7.np32.tol-2 Type: application/octet-stream Size: 15481 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Jul 30 14:49:51 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Jul 2014 14:49:51 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> Message-ID: <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > Hello, > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? Yes > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. Barry > > Thanks a lot for your help! > > Regards, > Qin From jifengzhao2015 at u.northwestern.edu Wed Jul 30 14:53:16 2014 From: jifengzhao2015 at u.northwestern.edu (jifeng zhao) Date: Wed, 30 Jul 2014 14:53:16 -0500 Subject: [petsc-users] eps_target malloc issue In-Reply-To: References: <42B8A7D6-2001-48BC-8E5C-B68CEB1394DD@mcs.anl.gov> <63FDFE2D-2B08-4B30-BF39-C0F314751E19@dsic.upv.es> <8C032529-BA15-4457-8E89-EDC50A2A5DBB@mcs.anl.gov> <7379E8E2-0D4F-4FB6-B886-3469A1F9BCE8@dsic.upv.es> Message-ID: Hello Hong, Thanks a lot. The result makes sense, but I think I will need higher accuracy (smaller tolerance). I will take a try myself at the meanwhile. Best regards, Jifeng Zhao On Wed, Jul 30, 2014 at 1:16 PM, hong at aspiritech.org wrote: > Jifeng, > Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices > > mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2 > $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type aij > -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1 -st_ksp_type > minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2 > -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee > log.mat20.ex7.np32.tol-2 > > The output is attached. > > When using '-mat_type sbaij', I got same error as reported, which > might be a bug in our libraries. I'll investigate it. > Switching to '-mat_type aij' works (I'm not sure if the computed > eigenvalues are meaningful though), but the linear solver > minres/bjacobi/icc converges very slow, takes around 500 - 1000 linear > iterations for each eps iteration. You may experiment with stronger > pc_type, e.g., '-pc_type asm' etc. > > I'll let you know why sbaij crashes. > > Hong > > On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman wrote: > > > > El 28/07/2014, a las 23:48, jifeng zhao escribi?: > > > >> Hi Barry, > >> > >> Thanks for the reply. I see. I am still testing it and looking for the > bug. > >> > >> It is pretty weird that the error appears, since the code I am running > is ex7 and I didn't change anything. > >> > >> My matrices are assembled in binary files separately. As I explained > earlier, I have a series of matrices to solve, each of them have the same > patterns but with different values. The way I assembly the matrix has > proven to be correct for all the other cases. Only for this one, when the > matrix has an zero eigenvalue, ex7 failed to solve it. > >> > >> In fact, I don't understand why this "Out of range" error could > possibly appear? 1. my matrices have the correct parrellel lay up. 2. In > ex7, I am not explicitly accessing to any elements at all! All I did is > EPSSolve(); > >> > >> Any ideas? > >> > >> Best regards, > >> Jifeng Zhao > >> > > > > Tomorrow we will release slepc-3.5. I would suggest trying with the new > version. If the problem persist we can help to track down the problem. > > > > Jose > > > -- Jifeng Zhao PhD candidate at Northwestern University, US Theoretical and Applied Mechanics Program -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Wed Jul 30 14:53:36 2014 From: jychang48 at gmail.com (Justin Chang) Date: Wed, 30 Jul 2014 13:53:36 -0600 Subject: [petsc-users] DMPlex and Tao Message-ID: Hi, This might be a silly question, but I am developing a finite element code to solve the non-negative diffusion equation. I am using the DMPlex structure and plan on subjecting the problem to a bounded constraint optimization to ensure non-negative concentrations. Say for instance, I want to solve: min || K*u - F ||^2 subject to u >= 0 M*u = 0 Where K is the Jacobian, F the residual, u the concentration, and M some equality constraint matrix. I know how to do this via Tao, but my question is can Tao handle Mats and Vecs created via DMPlex? Because from the SNES and TS examples I have seen in ex12, ex62, and ex11, it seems there are solvers tailored to handle DMPlex created data structures. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 30 15:03:40 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jul 2014 15:03:40 -0500 Subject: [petsc-users] DMPlex and Tao In-Reply-To: References: Message-ID: On Wed, Jul 30, 2014 at 2:53 PM, Justin Chang wrote: > Hi, > > This might be a silly question, but I am developing a finite element code > to solve the non-negative diffusion equation. I am using the DMPlex > structure and plan on subjecting the problem to a bounded constraint > optimization to ensure non-negative concentrations. Say for instance, I > want to solve: > > min || K*u - F ||^2 > subject to u >= 0 > M*u = 0 > > Where K is the Jacobian, F the residual, u the concentration, and M some > equality constraint matrix. I know how to do this via Tao, but my question > is can Tao handle Mats and Vecs created via DMPlex? Because from the SNES > and TS examples I have seen in ex12, ex62, and ex11, it seems there are > solvers tailored to handle DMPlex created data structures. > Yes, TAO can handle objects created by DMPlex since they are just Vec and Mat structures. The additions to ex12, ex62, and ex11 concern the callbacks from the solver to the constructions routines for residual and Jacobian. Right now, none of the callbacks are automatic for TAO so you would have to code them yourself, probably by just calling the routines from SNES or TS so it should not be that hard. We are working to get everything integrated as it is in the other solvers. Thanks, Matt > Thanks, > Justin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Wed Jul 30 15:30:30 2014 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Wed, 30 Jul 2014 13:30:30 -0700 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> Message-ID: <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> In the context of domain decompostion, if the?unknowns are ordered (to reduce the number of infills, for instance) in the way that?each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? ? For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain?2 (assigned to process 2) owns unknowns?2 and 4 (associated with equation?2 and 4) , how can I make each process own consecutive rows? ? Thanks, Qin From: Barry Smith To: Qin Lu Cc: petsc-users Sent: Wednesday, July 30, 2014 2:49 PM Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > Hello, >? > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? ? Yes > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? ? However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. ? Barry >? > Thanks a lot for your help! >? > Regards, > Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 30 15:50:17 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jul 2014 15:50:17 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > In the context of domain decompostion, if the unknowns are ordered (to > reduce the number of infills, for instance) in the way that each subdomain > may not own consecutive unknown index, does this mean the partition of the > domain will be different from the partition of the matrix? > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 > (associated with equation 1 and 3), subdomain 2 (assigned to process 2) > owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make > each process own consecutive rows? > You renumber the rows once they are partitioned. Matt > Thanks, > Qin > > *From:* Barry Smith > *To:* Qin Lu > *Cc:* petsc-users > *Sent:* Wednesday, July 30, 2014 2:49 PM > *Subject:* Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > Hello, > > > > Does a process have to own consecutive rows of the matrix? For example, > suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 > have to own rows 1 and 2, process 2 own rows 3 and 4? > > Yes > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > However, the numbering of degrees of freedom is arbitrary. Just renumber > you degrees of freedom so the first set is on process 0, the next on > process 1 etc. > > Barry > > > > > > Thanks a lot for your help! > > > > Regards, > > Qin > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Wed Jul 30 15:59:15 2014 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Wed, 30 Jul 2014 13:59:15 -0700 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> Renumbering?row will make an unknown not associate with its row (equation), will this affect the solver?performance? For example, will the infill pattern change in ILU factorization? ? Thanks, Qin? ________________________________ From: Matthew Knepley To: Qin Lu Cc: Barry Smith ; petsc-users Sent: Wednesday, July 30, 2014 3:50 PM Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: In the context of domain decompostion, if the?unknowns are ordered (to reduce the number of infills, for instance) in the way that?each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? > >For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain?2 (assigned to process 2) owns unknowns?2 and 4 (associated with equation?2 and 4) , how can I make each process own consecutive rows? You renumber the rows once they are partitioned. ? ?Matt Thanks, >Qin > > > From: Barry Smith >To: Qin Lu >Cc: petsc-users >Sent: Wednesday, July 30, 2014 2:49 PM >Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > >On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > >> Hello, >>? >> Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > >? Yes > >> Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > >? However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > >? Barry > > >>? >> Thanks a lot for your help! >>? >> Regards, >> Qin > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 30 16:02:58 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 30 Jul 2014 16:02:58 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: On Wed, Jul 30, 2014 at 3:59 PM, Qin Lu wrote: > Renumbering row will make an unknown not associate with its row > (equation), will this affect the solver performance? For example, will the > infill pattern change in ILU factorization? > 1) If you are doing parallel LU, all those packages (MUMPS, SuperLU) reorder anyway 2) If you are doing LU on a domain, you already have a contiguous order Matt > > Thanks, > Qin > > *From:* Matthew Knepley > *To:* Qin Lu > *Cc:* Barry Smith ; petsc-users < > petsc-users at mcs.anl.gov> > *Sent:* Wednesday, July 30, 2014 3:50 PM > *Subject:* Re: [petsc-users] Partition of parallel AIJ sparce matrix > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > > In the context of domain decompostion, if the unknowns are ordered (to > reduce the number of infills, for instance) in the way that each subdomain > may not own consecutive unknown index, does this mean the partition of the > domain will be different from the partition of the matrix? > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 > (associated with equation 1 and 3), subdomain 2 (assigned to process 2) > owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make > each process own consecutive rows? > > > You renumber the rows once they are partitioned. > > Matt > > > > > Thanks, > Qin > > *From:* Barry Smith > *To:* Qin Lu > *Cc:* petsc-users > *Sent:* Wednesday, July 30, 2014 2:49 PM > *Subject:* Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > Hello, > > > > Does a process have to own consecutive rows of the matrix? For example, > suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 > have to own rows 1 and 2, process 2 own rows 3 and 4? > > Yes > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > However, the numbering of degrees of freedom is arbitrary. Just renumber > you degrees of freedom so the first set is on process 0, the next on > process 1 etc. > > Barry > > > > > > Thanks a lot for your help! > > > > Regards, > > Qin > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Wed Jul 30 17:01:17 2014 From: jychang48 at gmail.com (Justin Chang) Date: Wed, 30 Jul 2014 16:01:17 -0600 Subject: [petsc-users] DMPlex and Tao In-Reply-To: References: Message-ID: That's good to know thanks. I have another question somewhat related. In that quadratic programming problem I have shown earlier, my K matrix is of global size *Ndof* by *Ndof* where *Ndof* is number of free dofs (for instance, if I have a 2x2 quadrilateral mesh and want to evaluate concentration at all the verticies, I will have nine total dofs but if I have Dirichlet constraints all along the boundary, only my center vertex is free so *Ndof* = 1. Thus when I simply call DMCreateMatrix(...) it would give me a 1x1 matrix.) However, I want my M constraint matrix to be of size *Nele* by *Ndof* where *Nele* is the total number of elements. How would I create this matrix? Because ultimately, I am subjecting all of my free u's to *Nele* number of equality constraints, and if possible I want this M matrix to be compatible with the layout from Vec u. On Wed, Jul 30, 2014 at 2:03 PM, Matthew Knepley wrote: > On Wed, Jul 30, 2014 at 2:53 PM, Justin Chang wrote: > >> Hi, >> >> This might be a silly question, but I am developing a finite element code >> to solve the non-negative diffusion equation. I am using the DMPlex >> structure and plan on subjecting the problem to a bounded constraint >> optimization to ensure non-negative concentrations. Say for instance, I >> want to solve: >> >> min || K*u - F ||^2 >> subject to u >= 0 >> M*u = 0 >> >> Where K is the Jacobian, F the residual, u the concentration, and M some >> equality constraint matrix. I know how to do this via Tao, but my question >> is can Tao handle Mats and Vecs created via DMPlex? Because from the SNES >> and TS examples I have seen in ex12, ex62, and ex11, it seems there are >> solvers tailored to handle DMPlex created data structures. >> > > Yes, TAO can handle objects created by DMPlex since they are just Vec and > Mat structures. The additions > to ex12, ex62, and ex11 concern the callbacks from the solver to the > constructions routines for residual and > Jacobian. Right now, none of the callbacks are automatic for TAO so you > would have to code them yourself, > probably by just calling the routines from SNES or TS so it should not be > that hard. We are working to get > everything integrated as it is in the other solvers. > > Thanks, > > Matt > > >> Thanks, >> Justin >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Wed Jul 30 18:09:12 2014 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Wed, 30 Jul 2014 23:09:12 +0000 Subject: [petsc-users] TAOGetKSP fails if TAOSolve has not been called. Message-ID: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> Hi, TAOGetKSP returns a null pointer when called before TAOSolve (swap the calls in eptorsion1.c for instance). Is this expected? I want to override the default KSP in my application. Blaise -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From bourdin at lsu.edu Wed Jul 30 18:19:07 2014 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Wed, 30 Jul 2014 23:19:07 +0000 Subject: [petsc-users] TAOGetKSP fails if TAOSolve has not been called. In-Reply-To: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> References: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> Message-ID: <8DED8E37-7AE3-4ADC-8B13-229BEF9AB588@lsu.edu> Responding to myself to clarify my question: This is with petsc-maint (3.5) up to date bourdin at galerkin:tutorials (maint)$ git show commit 587bed1b5ce23d03610a5e586bde2d77cb223937 Merge: ac2cab2 4856ba2 Author: Satish Balay Date: Wed Jul 30 09:53:57 2014 -0500 Merge branch 'barry/check-mpi-include' into maint I see in the release notes of 3.5.1 a reference to TAOSetUp()being needed, but calling TaoSetUp after TAOSetFromOptions but before TaoGetKSP does not make any difference. Breakpoint 1.1, main (argc=3, argv=0x7fff5fbfe9f8) at /opt/HPC/petsc-3.5/src/tao/unconstrained/examples/tutorials/eptorsion1.c:148 148 ierr = TaoSetFromOptions(tao);CHKERRQ(ierr); (idb) n 150 ierr = TaoSetUp(tao); (idb) 151 ierr = TaoGetKSP(tao,&ksp); (idb) 153 ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); (idb) print tao $1 = (struct _p_Tao *) 0x10885c860 (idb) print ksp $2 = (struct _p_KSP *) 0x0 On Jul 30, 2014, at 6:09 PM, Blaise A Bourdin wrote: > Hi, > > TAOGetKSP returns a null pointer when called before TAOSolve (swap the calls in eptorsion1.c for instance). Is this expected? I want to override the default KSP in my application. > > Blaise > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > > -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From jonathantu at gmail.com Wed Jul 30 18:29:18 2014 From: jonathantu at gmail.com (Jonathan Tu) Date: Wed, 30 Jul 2014 16:29:18 -0700 Subject: [petsc-users] Shell matrix in petsc4py? Message-ID: Hi, I would like to solve a large linear system. I have a linear operator that operates on a "state" vector. The linear operator does things like take the sum of the state elements, take the cross product of the state with a known vector, and then multiply the state by a large (distributed) matrix. You could imagine it like this: [sum of elements, cross product with other vector, multiplication with large matrix] = A * x Now, I realize it is possible to implement all of this using a large matrix. After all, these are all linear operations. However, I thought it might be faster and less error-prone to use some of numpy's built-in operations, for the sums and cross products, and only use a distributed matrix for the last part of the output. >From the PETSc documentation, it seems like a matrix shell would be perfect for this. I could define my own mult operator that does all of these things in steps, using built-in numpy functions when appropriate and direct matrix multiplication when appropriate. However, I cannot figure out how to do this in petsc4py. I found somewhere online that perhaps createPython is the way I need to go about this, but I couldn't figure out the details. Any ideas? Jonathan Tu -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetank at gmail.com Wed Jul 30 18:48:00 2014 From: gaetank at gmail.com (Gaetan Kenway) Date: Wed, 30 Jul 2014 19:48:00 -0400 Subject: [petsc-users] Shell matrix in petsc4py? In-Reply-To: References: Message-ID: Hi Jonathan I've successfully used petsc shell matrices from Python. There is an example of how to do it in demo/poisson2d/poisson2d.py. The example is a little sparse, but it has all the important information. The most important is to define the def mult(self, mat, x, y): routine inside of a Python class that is given in mat.setPythonContext() command. Hope that helps. Gaetan On Wed, Jul 30, 2014 at 7:29 PM, Jonathan Tu wrote: > Hi, > > I would like to solve a large linear system. I have a linear operator > that operates on a "state" vector. The linear operator does things like > take the sum of the state elements, take the cross product of the state > with a known vector, and then multiply the state by a large (distributed) > matrix. You could imagine it like this: > > [sum of elements, cross product with other vector, multiplication with > large matrix] = A * x > > Now, I realize it is possible to implement all of this using a large > matrix. After all, these are all linear operations. However, I thought it > might be faster and less error-prone to use some of numpy's built-in > operations, for the sums and cross products, and only use a distributed > matrix for the last part of the output. > > From the PETSc documentation, it seems like a matrix shell would be > perfect for this. I could define my own mult operator that does all of > these things in steps, using built-in numpy functions when appropriate and > direct matrix multiplication when appropriate. However, I cannot figure > out how to do this in petsc4py. I found somewhere online that perhaps > createPython is the way I need to go about this, but I couldn't figure out > the details. > > Any ideas? > > > > Jonathan Tu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathantu at gmail.com Wed Jul 30 18:56:56 2014 From: jonathantu at gmail.com (Jonathan Tu) Date: Wed, 30 Jul 2014 16:56:56 -0700 Subject: [petsc-users] Shell matrix in petsc4py? In-Reply-To: References: Message-ID: <2C743D64-94DB-4D36-BCB9-7D7CCB0E3E04@gmail.com> Hi Gaetan, Thanks for pointing me to the example. I think I understand more now. It seems that setPythonContext should point to an object that performs the multi(x, y) operation, which takes x and modifies y inside the multi() function. What does setUp() do? Also, can I assume that the implementation is Pythonic, such that the shell can operate on my own custom vector objects, rather than PETSc Vecs? Defining the multi function to do this would be easy, but I wanted to make sure that the KSP solver wouldn't complain for some reason. Thanks, Jonathan Tu On Jul 30, 2014, at 4:48 PM, Gaetan Kenway wrote: > Hi Jonathan > > I've successfully used petsc shell matrices from Python. There is an example of how to do it in demo/poisson2d/poisson2d.py. The example is a little sparse, but it has all the important information. The most important is to define the > > def mult(self, mat, x, y): > > routine inside of a Python class that is given in mat.setPythonContext() command. > > Hope that helps. > > Gaetan > > > On Wed, Jul 30, 2014 at 7:29 PM, Jonathan Tu wrote: > Hi, > > I would like to solve a large linear system. I have a linear operator that operates on a "state" vector. The linear operator does things like take the sum of the state elements, take the cross product of the state with a known vector, and then multiply the state by a large (distributed) matrix. You could imagine it like this: > > [sum of elements, cross product with other vector, multiplication with large matrix] = A * x > > Now, I realize it is possible to implement all of this using a large matrix. After all, these are all linear operations. However, I thought it might be faster and less error-prone to use some of numpy's built-in operations, for the sums and cross products, and only use a distributed matrix for the last part of the output. > > From the PETSc documentation, it seems like a matrix shell would be perfect for this. I could define my own mult operator that does all of these things in steps, using built-in numpy functions when appropriate and direct matrix multiplication when appropriate. However, I cannot figure out how to do this in petsc4py. I found somewhere online that perhaps createPython is the way I need to go about this, but I couldn't figure out the details. > > Any ideas? > > > > Jonathan Tu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.sarich at gmail.com Wed Jul 30 19:01:27 2014 From: jason.sarich at gmail.com (Jason Sarich) Date: Wed, 30 Jul 2014 19:01:27 -0500 Subject: [petsc-users] TAOGetKSP fails if TAOSolve has not been called. In-Reply-To: <5b888e0000c643d2b6629765c262f1d6@GEORGE.anl.gov> References: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> <5b888e0000c643d2b6629765c262f1d6@GEORGE.anl.gov> Message-ID: Hi Blaise, I need to update this so that it TaoGetKSP mimics the behavior of SNESGetKSP (ie, creates a KSP object if it doesn't exist already, even if it will never get used). Jason On Wed, Jul 30, 2014 at 6:19 PM, Blaise A Bourdin wrote: > Responding to myself to clarify my question: > > This is with petsc-maint (3.5) up to date > > bourdin at galerkin:tutorials (maint)$ git show > commit 587bed1b5ce23d03610a5e586bde2d77cb223937 > Merge: ac2cab2 4856ba2 > Author: Satish Balay > Date: Wed Jul 30 09:53:57 2014 -0500 > > Merge branch 'barry/check-mpi-include' into maint > > I see in the release notes of 3.5.1 a reference to TAOSetUp()being needed, > but calling TaoSetUp after TAOSetFromOptions but before TaoGetKSP does not > make any difference. > > Breakpoint 1.1, main (argc=3, argv=0x7fff5fbfe9f8) at > /opt/HPC/petsc-3.5/src/tao/unconstrained/examples/tutorials/eptorsion1.c:148 > 148 ierr = TaoSetFromOptions(tao);CHKERRQ(ierr); > (idb) n > 150 ierr = TaoSetUp(tao); > (idb) > 151 ierr = TaoGetKSP(tao,&ksp); > (idb) > 153 ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > (idb) print tao > $1 = (struct _p_Tao *) 0x10885c860 > (idb) print ksp > $2 = (struct _p_KSP *) 0x0 > > > On Jul 30, 2014, at 6:09 PM, Blaise A Bourdin wrote: > > > Hi, > > > > TAOGetKSP returns a null pointer when called before TAOSolve (swap the > calls in eptorsion1.c for instance). Is this expected? I want to override > the default KSP in my application. > > > > Blaise > > -- > > Department of Mathematics and Center for Computation & Technology > > Louisiana State University, Baton Rouge, LA 70803, USA > > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > > > > > > > > > > > > > > > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jul 30 19:37:48 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Jul 2014 19:37:48 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> On Jul 30, 2014, at 3:59 PM, Qin Lu wrote: > Renumbering row will make an unknown not associate with its row (equation), It is a symmetric reordering. In matrix terms both the rows and columns are reordered the same way. In terms of spaces both the domain and range spaces are reordered the same way. Barry > will this affect the solver performance? For example, will the infill pattern change in ILU factorization? > > Thanks, > Qin > > From: Matthew Knepley > To: Qin Lu > Cc: Barry Smith ; petsc-users > Sent: Wednesday, July 30, 2014 3:50 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > In the context of domain decompostion, if the unknowns are ordered (to reduce the number of infills, for instance) in the way that each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain 2 (assigned to process 2) owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make each process own consecutive rows? > > You renumber the rows once they are partitioned. > > Matt > > > > Thanks, > Qin > > From: Barry Smith > To: Qin Lu > Cc: petsc-users > Sent: Wednesday, July 30, 2014 2:49 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > Hello, > > > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > > Yes > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > > Barry > > > > > > Thanks a lot for your help! > > > > Regards, > > Qin > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > From alp at dener.me Wed Jul 30 19:49:34 2014 From: alp at dener.me (Alp Dener) Date: Wed, 30 Jul 2014 20:49:34 -0400 Subject: [petsc-users] Configure with IBM XL toolchain quietly ignores request to build shared libraries Message-ID: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> Hi, I?m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using the XL toolchain. I need to build the PETSc shared library and to accomplish that, I use the following configure options: ./configure \ --with-shared-libraries=1 \ --with-debugging=1 \ --with-fortran-interfaces=1 \ --with-mpi=/bgsys/drivers/ppcfloor/comm \ --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \ --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack" \ --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis" \ --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lparmetis" \ --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \ --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz \ --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \ --PETSC_ARCH=xl-dbg \ FFLAGS="-qextname? Despite specifying the ??with-shared-libraries? option, the configuration report shows ?shared libraries : disabled?. The configure log doesn?t appear to contain any warnings or errors about this (or maybe I couldn?t find it). Normally, compiling shared objects with XL compilers require a -qmkshrobj flag. I?ve attempted passing this to PETSc using LDFLAGS but this causes the configuration to fail with an error, stating that it cannot run executables created with the C compiler. This problem seems to be persistent across a number of PETSc versions. I need to use 3.4.5 for compatibility reasons, but I?ve attempted the same build with 3.5, 3.3 and 3.0 with the exact same results. Would really appreciate it if anyone can offer guidance on how to force PETSc to build its shared library. The configure log from the above options is provided in the attachment. Thanks! ?????????????????? Alp Dener -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3494103 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Jul 30 19:51:15 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Jul 2014 19:51:15 -0500 Subject: [petsc-users] TAOGetKSP fails if TAOSolve has not been called. In-Reply-To: <8DED8E37-7AE3-4ADC-8B13-229BEF9AB588@lsu.edu> References: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> <8DED8E37-7AE3-4ADC-8B13-229BEF9AB588@lsu.edu> Message-ID: <176D72A6-1AD4-4318-BFE1-89B1A41042FB@mcs.anl.gov> Hmm, which solver are you using? I fixed this for one solver already. Most solvers do return the KSP at this stage. For solvers that do not use a KSP I would prefer that it never be created. Barry On Jul 30, 2014, at 6:19 PM, Blaise A Bourdin wrote: > Responding to myself to clarify my question: > > This is with petsc-maint (3.5) up to date > > bourdin at galerkin:tutorials (maint)$ git show > commit 587bed1b5ce23d03610a5e586bde2d77cb223937 > Merge: ac2cab2 4856ba2 > Author: Satish Balay > Date: Wed Jul 30 09:53:57 2014 -0500 > > Merge branch 'barry/check-mpi-include' into maint > > I see in the release notes of 3.5.1 a reference to TAOSetUp()being needed, but calling TaoSetUp after TAOSetFromOptions but before TaoGetKSP does not make any difference. > > Breakpoint 1.1, main (argc=3, argv=0x7fff5fbfe9f8) at /opt/HPC/petsc-3.5/src/tao/unconstrained/examples/tutorials/eptorsion1.c:148 > 148 ierr = TaoSetFromOptions(tao);CHKERRQ(ierr); > (idb) n > 150 ierr = TaoSetUp(tao); > (idb) > 151 ierr = TaoGetKSP(tao,&ksp); > (idb) > 153 ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > (idb) print tao > $1 = (struct _p_Tao *) 0x10885c860 > (idb) print ksp > $2 = (struct _p_KSP *) 0x0 > > > On Jul 30, 2014, at 6:09 PM, Blaise A Bourdin wrote: > >> Hi, >> >> TAOGetKSP returns a null pointer when called before TAOSolve (swap the calls in eptorsion1.c for instance). Is this expected? I want to override the default KSP in my application. >> >> Blaise >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > > From gaetank at gmail.com Wed Jul 30 20:01:22 2014 From: gaetank at gmail.com (Gaetan Kenway) Date: Wed, 30 Jul 2014 21:01:22 -0400 Subject: [petsc-users] Shell matrix in petsc4py? In-Reply-To: <2C743D64-94DB-4D36-BCB9-7D7CCB0E3E04@gmail.com> References: <2C743D64-94DB-4D36-BCB9-7D7CCB0E3E04@gmail.com> Message-ID: Hi Jonathan I'm not a developer of petsc4py but have it extensively. I'm not entirely sure what setUp() does only that it has to be called before the matrix can be used. As for the implementation, I'm fairly sure that the x and y vectors need to be petsc4py PETSc Vecs. Mostly likely what you will may want to do something like this: def mult(self, A, X, Y): with X as x: # x is now a numpy array with Y as y: # y is also a numpy array y = do_something(x) Hope that helps. Gaetan On Wed, Jul 30, 2014 at 7:56 PM, Jonathan Tu wrote: > Hi Gaetan, > > Thanks for pointing me to the example. I think I understand more now. It > seems that setPythonContext should point to an object that performs the > multi(x, y) operation, which takes x and modifies y inside the multi() > function. What does setUp() do? > > Also, can I assume that the implementation is Pythonic, such that the > shell can operate on my own custom vector objects, rather than PETSc Vecs? > Defining the multi function to do this would be easy, but I wanted to make > sure that the KSP solver wouldn't complain for some reason. > > > > Thanks, > Jonathan Tu > > > On Jul 30, 2014, at 4:48 PM, Gaetan Kenway wrote: > > Hi Jonathan > > I've successfully used petsc shell matrices from Python. There is an > example of how to do it in demo/poisson2d/poisson2d.py. The example is a > little sparse, but it has all the important information. The most important > is to define the > > def mult(self, mat, x, y): > > routine inside of a Python class that is given in mat.setPythonContext() > command. > > Hope that helps. > > Gaetan > > > On Wed, Jul 30, 2014 at 7:29 PM, Jonathan Tu wrote: > >> Hi, >> >> I would like to solve a large linear system. I have a linear operator >> that operates on a "state" vector. The linear operator does things like >> take the sum of the state elements, take the cross product of the state >> with a known vector, and then multiply the state by a large (distributed) >> matrix. You could imagine it like this: >> >> [sum of elements, cross product with other vector, multiplication with >> large matrix] = A * x >> >> Now, I realize it is possible to implement all of this using a large >> matrix. After all, these are all linear operations. However, I thought it >> might be faster and less error-prone to use some of numpy's built-in >> operations, for the sums and cross products, and only use a distributed >> matrix for the last part of the output. >> >> From the PETSc documentation, it seems like a matrix shell would be >> perfect for this. I could define my own mult operator that does all of >> these things in steps, using built-in numpy functions when appropriate and >> direct matrix multiplication when appropriate. However, I cannot figure >> out how to do this in petsc4py. I found somewhere online that perhaps >> createPython is the way I need to go about this, but I couldn't figure out >> the details. >> >> Any ideas? >> >> >> >> Jonathan Tu >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jul 30 20:27:33 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 30 Jul 2014 20:27:33 -0500 Subject: [petsc-users] Configure with IBM XL toolchain quietly ignores request to build shared libraries In-Reply-To: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> References: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> Message-ID: On Wed, 30 Jul 2014, Alp Dener wrote: > Hi, > > I?m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using the XL toolchain. I need to build the PETSc shared library and to accomplish that, I use the following configure options: > > ./configure \ > --with-shared-libraries=1 \ > --with-debugging=1 \ > --with-fortran-interfaces=1 \ > --with-mpi=/bgsys/drivers/ppcfloor/comm \ > --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \ > --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack" \ > --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ > --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis" \ > --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ > --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lparmetis" \ > --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \ > --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz \ > --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \ > --PETSC_ARCH=xl-dbg \ > FFLAGS="-qextname? > > Despite specifying the ??with-shared-libraries? option, the configuration report shows ?shared libraries : disabled?. The configure log doesn?t appear to contain any warnings or errors about this (or maybe I couldn?t find it). > > Normally, compiling shared objects with XL compilers require a -qmkshrobj flag. I?ve attempted passing this to PETSc using LDFLAGS but this causes the configuration to fail with an error, stating that it cannot run executables created with the C compiler. > > This problem seems to be persistent across a number of PETSc versions. I need to use 3.4.5 for compatibility reasons, but I?ve attempted the same build with 3.5, 3.3 and 3.0 with the exact same results. > > Would really appreciate it if anyone can offer guidance on how to force PETSc to build its shared library. The configure log from the above options is provided in the attachment. Shared libraries with IBM compilers is something I've not tired. If its just a matter of using '-qmkshrobj' - you could try to build it manually - and see if it works. [when I try it I get errors.] Satish ---------- [balay at vestalac1 ~]$ cd petsc/arch-xl-opt-nopkg/lib/ [balay at vestalac1 lib]$ ar x libpetsc.a [balay at vestalac1 lib]$ mpixlc -qmkshrobj -o libpetsc.so *.o [balay at vestalac1 lib]$ file libpetsc.so libpetsc.so: ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, not stripped [balay at vestalac1 lib]$ mv libpetsc.a libpetsc-bkp.a [balay at vestalac1 lib]$ cd $PETSC_DIR [balay at vestalac1 petsc]$ cd src/ksp/ksp/examples/tutorials/ [balay at vestalac1 tutorials]$ make ex2 mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/ppc64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/ld: cannot find -lpetsc make: [ex2] Error 1 (ignored) /bin/rm -f ex2.o [balay at vestalac1 tutorials]$ From balay at mcs.anl.gov Wed Jul 30 20:48:26 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 30 Jul 2014 20:48:26 -0500 Subject: [petsc-users] Configure with IBM XL toolchain quietly ignores request to build shared libraries In-Reply-To: References: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> Message-ID: https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq#how-do-i-compile-and-link-a-shared-library? Looks like I need to specify '-Wl,-dy' for the linker to find the shared library. [balay at vestalac1 tutorials]$ make ex2 CFLAGS="-Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib" mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nop kg/lib -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/pp c64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl /bin/rm -f ex2.o [balay at vestalac1 tutorials]$ ldd ex2 linux-vdso64.so.1 => (0x00000fff8d020000) libpetsc.so => /home/balay/petsc/arch-xl-opt-nopkg/lib/libpetsc.so (0x00000fff8bfb0000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff8bf50000) libmpichf90-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichf90-gcc.so.8 (0x00000fff8be40000) libxlf90_r.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlf90_r.so.1 (0x00000fff8b7a0000) libxlomp_ser.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlomp_ser.so.1 (0x00000fff8b690000) libxlfmath.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlfmath.so.1 (0x00000fff8b580000) libm.so.6 => /lib64/libm.so.6 (0x00000fff8b480000) libmpichcxx-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichcxx-gcc.so.8 (0x00000fff8b370000) libibmc++.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libibmc++.so.1 (0x00000fff8b260000) libdl.so.2 => /lib64/libdl.so.2 (0x00000fff8b230000) libmpich-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpich-gcc.so.8 (0x00000fff8ae90000) libopa-gcc.so.0 => /bgsys/drivers/ppcfloor/comm/lib/libopa-gcc.so.0 (0x00000fff8ad80000) libmpl-gcc.so.1 => /bgsys/drivers/ppcfloor/comm/lib/libmpl-gcc.so.1 (0x00000fff8ac70000) libpami-gcc.so => /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so (0x00000fff8a700000) librt.so.1 => /lib64/librt.so.1 (0x00000fff8a6d0000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000080ea910000) libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00000fff8a6a0000) libnss_dns.so.2 => /lib64/libnss_dns.so.2 (0x00000fff8a670000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00000fff8a630000) libc.so.6 => /lib64/libc.so.6 (0x00000fff8a440000) /lib64/bgq/ld64.so.1 => /lib64/ld64.so.1 (0x00000000264e0000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000080ea720000) libgfortran.so.3 => /bgsys/drivers/toolchain/V1R2M2_base/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so.3 (0x00000fff8a220000) And I could run this binary successfully.. Satish On Wed, 30 Jul 2014, Satish Balay wrote: > On Wed, 30 Jul 2014, Alp Dener wrote: > > > Hi, > > > > I?m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using the XL toolchain. I need to build the PETSc shared library and to accomplish that, I use the following configure options: > > > > ./configure \ > > --with-shared-libraries=1 \ > > --with-debugging=1 \ > > --with-fortran-interfaces=1 \ > > --with-mpi=/bgsys/drivers/ppcfloor/comm \ > > --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \ > > --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack" \ > > --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ > > --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis" \ > > --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ > > --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lparmetis" \ > > --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \ > > --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz \ > > --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \ > > --PETSC_ARCH=xl-dbg \ > > FFLAGS="-qextname? > > > > Despite specifying the ??with-shared-libraries? option, the configuration report shows ?shared libraries : disabled?. The configure log doesn?t appear to contain any warnings or errors about this (or maybe I couldn?t find it). > > > > Normally, compiling shared objects with XL compilers require a -qmkshrobj flag. I?ve attempted passing this to PETSc using LDFLAGS but this causes the configuration to fail with an error, stating that it cannot run executables created with the C compiler. > > > > This problem seems to be persistent across a number of PETSc versions. I need to use 3.4.5 for compatibility reasons, but I?ve attempted the same build with 3.5, 3.3 and 3.0 with the exact same results. > > > > Would really appreciate it if anyone can offer guidance on how to force PETSc to build its shared library. The configure log from the above options is provided in the attachment. > > Shared libraries with IBM compilers is something I've not tired. If its just a matter > of using '-qmkshrobj' - you could try to build it manually - and see if it works. > > [when I try it I get errors.] > > Satish > > ---------- > > [balay at vestalac1 ~]$ cd petsc/arch-xl-opt-nopkg/lib/ > [balay at vestalac1 lib]$ ar x libpetsc.a > [balay at vestalac1 lib]$ mpixlc -qmkshrobj -o libpetsc.so *.o > [balay at vestalac1 lib]$ file libpetsc.so > libpetsc.so: ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, not stripped > [balay at vestalac1 lib]$ mv libpetsc.a libpetsc-bkp.a > [balay at vestalac1 lib]$ cd $PETSC_DIR > [balay at vestalac1 petsc]$ cd src/ksp/ksp/examples/tutorials/ > [balay at vestalac1 tutorials]$ make ex2 > mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/ppc64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c > mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl > /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/ld: cannot find -lpetsc > make: [ex2] Error 1 (ignored) > /bin/rm -f ex2.o > [balay at vestalac1 tutorials]$ From bsmith at mcs.anl.gov Wed Jul 30 20:53:35 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Jul 2014 20:53:35 -0500 Subject: [petsc-users] Configure with IBM XL toolchain quietly ignores request to build shared libraries In-Reply-To: References: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> Message-ID: <22160CB8-2B46-41FE-BDD9-27FB1986C9B4@mcs.anl.gov> Cool, let?s get this into maint On Jul 30, 2014, at 8:48 PM, Satish Balay wrote: > https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq#how-do-i-compile-and-link-a-shared-library? > > Looks like I need to specify '-Wl,-dy' for the linker to find the shared library. > > [balay at vestalac1 tutorials]$ make ex2 CFLAGS="-Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib" > mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nop > kg/lib -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/pp > c64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c > mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl > /bin/rm -f ex2.o > [balay at vestalac1 tutorials]$ ldd ex2 > linux-vdso64.so.1 => (0x00000fff8d020000) > libpetsc.so => /home/balay/petsc/arch-xl-opt-nopkg/lib/libpetsc.so (0x00000fff8bfb0000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff8bf50000) > libmpichf90-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichf90-gcc.so.8 (0x00000fff8be40000) > libxlf90_r.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlf90_r.so.1 (0x00000fff8b7a0000) > libxlomp_ser.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlomp_ser.so.1 (0x00000fff8b690000) > libxlfmath.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlfmath.so.1 (0x00000fff8b580000) > libm.so.6 => /lib64/libm.so.6 (0x00000fff8b480000) > libmpichcxx-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichcxx-gcc.so.8 (0x00000fff8b370000) > libibmc++.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libibmc++.so.1 (0x00000fff8b260000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000fff8b230000) > libmpich-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpich-gcc.so.8 (0x00000fff8ae90000) > libopa-gcc.so.0 => /bgsys/drivers/ppcfloor/comm/lib/libopa-gcc.so.0 (0x00000fff8ad80000) > libmpl-gcc.so.1 => /bgsys/drivers/ppcfloor/comm/lib/libmpl-gcc.so.1 (0x00000fff8ac70000) > libpami-gcc.so => /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so (0x00000fff8a700000) > librt.so.1 => /lib64/librt.so.1 (0x00000fff8a6d0000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000080ea910000) > libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00000fff8a6a0000) > libnss_dns.so.2 => /lib64/libnss_dns.so.2 (0x00000fff8a670000) > libresolv.so.2 => /lib64/libresolv.so.2 (0x00000fff8a630000) > libc.so.6 => /lib64/libc.so.6 (0x00000fff8a440000) > /lib64/bgq/ld64.so.1 => /lib64/ld64.so.1 (0x00000000264e0000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000080ea720000) > libgfortran.so.3 => /bgsys/drivers/toolchain/V1R2M2_base/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so.3 (0x00000fff8a220000) > > And I could run this binary successfully.. > > Satish > > On Wed, 30 Jul 2014, Satish Balay wrote: > >> On Wed, 30 Jul 2014, Alp Dener wrote: >> >>> Hi, >>> >>> I?m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using the XL toolchain. I need to build the PETSc shared library and to accomplish that, I use the following configure options: >>> >>> ./configure \ >>> --with-shared-libraries=1 \ >>> --with-debugging=1 \ >>> --with-fortran-interfaces=1 \ >>> --with-mpi=/bgsys/drivers/ppcfloor/comm \ >>> --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \ >>> --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack" \ >>> --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ >>> --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis" \ >>> --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ >>> --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lparmetis" \ >>> --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \ >>> --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz \ >>> --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \ >>> --PETSC_ARCH=xl-dbg \ >>> FFLAGS="-qextname? >>> >>> Despite specifying the ??with-shared-libraries? option, the configuration report shows ?shared libraries : disabled?. The configure log doesn?t appear to contain any warnings or errors about this (or maybe I couldn?t find it). >>> >>> Normally, compiling shared objects with XL compilers require a -qmkshrobj flag. I?ve attempted passing this to PETSc using LDFLAGS but this causes the configuration to fail with an error, stating that it cannot run executables created with the C compiler. >>> >>> This problem seems to be persistent across a number of PETSc versions. I need to use 3.4.5 for compatibility reasons, but I?ve attempted the same build with 3.5, 3.3 and 3.0 with the exact same results. >>> >>> Would really appreciate it if anyone can offer guidance on how to force PETSc to build its shared library. The configure log from the above options is provided in the attachment. >> >> Shared libraries with IBM compilers is something I've not tired. If its just a matter >> of using '-qmkshrobj' - you could try to build it manually - and see if it works. >> >> [when I try it I get errors.] >> >> Satish >> >> ---------- >> >> [balay at vestalac1 ~]$ cd petsc/arch-xl-opt-nopkg/lib/ >> [balay at vestalac1 lib]$ ar x libpetsc.a >> [balay at vestalac1 lib]$ mpixlc -qmkshrobj -o libpetsc.so *.o >> [balay at vestalac1 lib]$ file libpetsc.so >> libpetsc.so: ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, not stripped >> [balay at vestalac1 lib]$ mv libpetsc.a libpetsc-bkp.a >> [balay at vestalac1 lib]$ cd $PETSC_DIR >> [balay at vestalac1 petsc]$ cd src/ksp/ksp/examples/tutorials/ >> [balay at vestalac1 tutorials]$ make ex2 >> mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/ppc64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c >> mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl >> /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/ld: cannot find -lpetsc >> make: [ex2] Error 1 (ignored) >> /bin/rm -f ex2.o >> [balay at vestalac1 tutorials]$ From alp at dener.me Wed Jul 30 21:30:05 2014 From: alp at dener.me (Alp Dener) Date: Wed, 30 Jul 2014 22:30:05 -0400 Subject: [petsc-users] Configure with IBM XL toolchain quietly ignores request to build shared libraries In-Reply-To: References: <27D6C8EA-EE16-4F7C-BE02-6409751FEB50@dener.me> Message-ID: <86F7986C-086B-466C-98B4-5123850E7011@dener.me> Compiling manually did the trick. Thanks! ?????????????????? Alp Dener > On Jul 30, 2014, at 9:48 PM, Satish Balay wrote: > > https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq#how-do-i-compile-and-link-a-shared-library? > > Looks like I need to specify '-Wl,-dy' for the linker to find the shared library. > > [balay at vestalac1 tutorials]$ make ex2 CFLAGS="-Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib" > mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nop > kg/lib -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/pp > c64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c > mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl > /bin/rm -f ex2.o > [balay at vestalac1 tutorials]$ ldd ex2 > linux-vdso64.so.1 => (0x00000fff8d020000) > libpetsc.so => /home/balay/petsc/arch-xl-opt-nopkg/lib/libpetsc.so (0x00000fff8bfb0000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff8bf50000) > libmpichf90-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichf90-gcc.so.8 (0x00000fff8be40000) > libxlf90_r.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlf90_r.so.1 (0x00000fff8b7a0000) > libxlomp_ser.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlomp_ser.so.1 (0x00000fff8b690000) > libxlfmath.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlfmath.so.1 (0x00000fff8b580000) > libm.so.6 => /lib64/libm.so.6 (0x00000fff8b480000) > libmpichcxx-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpichcxx-gcc.so.8 (0x00000fff8b370000) > libibmc++.so.1 => /soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libibmc++.so.1 (0x00000fff8b260000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000fff8b230000) > libmpich-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpich-gcc.so.8 (0x00000fff8ae90000) > libopa-gcc.so.0 => /bgsys/drivers/ppcfloor/comm/lib/libopa-gcc.so.0 (0x00000fff8ad80000) > libmpl-gcc.so.1 => /bgsys/drivers/ppcfloor/comm/lib/libmpl-gcc.so.1 (0x00000fff8ac70000) > libpami-gcc.so => /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so (0x00000fff8a700000) > librt.so.1 => /lib64/librt.so.1 (0x00000fff8a6d0000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000080ea910000) > libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00000fff8a6a0000) > libnss_dns.so.2 => /lib64/libnss_dns.so.2 (0x00000fff8a670000) > libresolv.so.2 => /lib64/libresolv.so.2 (0x00000fff8a630000) > libc.so.6 => /lib64/libc.so.6 (0x00000fff8a440000) > /lib64/bgq/ld64.so.1 => /lib64/ld64.so.1 (0x00000000264e0000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000080ea720000) > libgfortran.so.3 => /bgsys/drivers/toolchain/V1R2M2_base/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so.3 (0x00000fff8a220000) > > And I could run this binary successfully.. > > Satish > > On Wed, 30 Jul 2014, Satish Balay wrote: > >> On Wed, 30 Jul 2014, Alp Dener wrote: >> >>> Hi, >>> >>> I?m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using the XL toolchain. I need to build the PETSc shared library and to accomplish that, I use the following configure options: >>> >>> ./configure \ >>> --with-shared-libraries=1 \ >>> --with-debugging=1 \ >>> --with-fortran-interfaces=1 \ >>> --with-mpi=/bgsys/drivers/ppcfloor/comm \ >>> --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \ >>> --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack" \ >>> --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ >>> --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis" \ >>> --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \ >>> --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lparmetis" \ >>> --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \ >>> --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz \ >>> --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \ >>> --PETSC_ARCH=xl-dbg \ >>> FFLAGS="-qextname? >>> >>> Despite specifying the ??with-shared-libraries? option, the configuration report shows ?shared libraries : disabled?. The configure log doesn?t appear to contain any warnings or errors about this (or maybe I couldn?t find it). >>> >>> Normally, compiling shared objects with XL compilers require a -qmkshrobj flag. I?ve attempted passing this to PETSc using LDFLAGS but this causes the configuration to fail with an error, stating that it cannot run executables created with the C compiler. >>> >>> This problem seems to be persistent across a number of PETSc versions. I need to use 3.4.5 for compatibility reasons, but I?ve attempted the same build with 3.5, 3.3 and 3.0 with the exact same results. >>> >>> Would really appreciate it if anyone can offer guidance on how to force PETSc to build its shared library. The configure log from the above options is provided in the attachment. >> >> Shared libraries with IBM compilers is something I've not tired. If its just a matter >> of using '-qmkshrobj' - you could try to build it manually - and see if it works. >> >> [when I try it I get errors.] >> >> Satish >> >> ---------- >> >> [balay at vestalac1 ~]$ cd petsc/arch-xl-opt-nopkg/lib/ >> [balay at vestalac1 lib]$ ar x libpetsc.a >> [balay at vestalac1 lib]$ mpixlc -qmkshrobj -o libpetsc.so *.o >> [balay at vestalac1 lib]$ file libpetsc.so >> libpetsc.so: ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, not stripped >> [balay at vestalac1 lib]$ mv libpetsc.a libpetsc-bkp.a >> [balay at vestalac1 lib]$ cd $PETSC_DIR >> [balay at vestalac1 petsc]$ cd src/ksp/ksp/examples/tutorials/ >> [balay at vestalac1 tutorials]$ make ex2 >> mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -I/home/balay/petsc/include -I/home/balay/petsc/arch-xl-opt-nopkg/include -I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/ppc64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64 -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c >> mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread -L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M1/ppc64/spi/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64 -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64 -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64 -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6 -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64 -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl >> /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/ld: cannot find -lpetsc >> make: [ex2] Error 1 (ignored) >> /bin/rm -f ex2.o >> [balay at vestalac1 tutorials]$ From bourdin at lsu.edu Thu Jul 31 09:32:50 2014 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Thu, 31 Jul 2014 14:32:50 +0000 Subject: [petsc-users] TAOGetKSP fails if TAOSolve has not been called. In-Reply-To: <176D72A6-1AD4-4318-BFE1-89B1A41042FB@mcs.anl.gov> References: <936FE6E5-66F7-42A4-9149-396528293CF8@lsu.edu> <8DED8E37-7AE3-4ADC-8B13-229BEF9AB588@lsu.edu> <176D72A6-1AD4-4318-BFE1-89B1A41042FB@mcs.anl.gov> Message-ID: <62236B3D-AB1E-40DC-814D-0EC0AE94C32D@lsu.edu> Duh, I was calling TaoGetKSP _before_ TaoSetFromOptions so that tao_type was still lmvm... Blaise > > Hmm, which solver are you using? I fixed this for one solver already. Most solvers do return the KSP at this stage. For solvers that do not use a KSP I would prefer that it never be created. > > Barry > > On Jul 30, 2014, at 6:19 PM, Blaise A Bourdin wrote: > >> Responding to myself to clarify my question: >> >> This is with petsc-maint (3.5) up to date >> >> bourdin at galerkin:tutorials (maint)$ git show >> commit 587bed1b5ce23d03610a5e586bde2d77cb223937 >> Merge: ac2cab2 4856ba2 >> Author: Satish Balay >> Date: Wed Jul 30 09:53:57 2014 -0500 >> >> Merge branch 'barry/check-mpi-include' into maint >> >> I see in the release notes of 3.5.1 a reference to TAOSetUp()being needed, but calling TaoSetUp after TAOSetFromOptions but before TaoGetKSP does not make any difference. >> >> Breakpoint 1.1, main (argc=3, argv=0x7fff5fbfe9f8) at /opt/HPC/petsc-3.5/src/tao/unconstrained/examples/tutorials/eptorsion1.c:148 >> 148 ierr = TaoSetFromOptions(tao);CHKERRQ(ierr); >> (idb) n >> 150 ierr = TaoSetUp(tao); >> (idb) >> 151 ierr = TaoGetKSP(tao,&ksp); >> (idb) >> 153 ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); >> (idb) print tao >> $1 = (struct _p_Tao *) 0x10885c860 >> (idb) print ksp >> $2 = (struct _p_KSP *) 0x0 >> >> >> On Jul 30, 2014, at 6:09 PM, Blaise A Bourdin wrote: >> >>> Hi, >>> >>> TAOGetKSP returns a null pointer when called before TAOSolve (swap the calls in eptorsion1.c for instance). Is this expected? I want to override the default KSP in my application. >>> >>> Blaise >>> -- >>> Department of Mathematics and Center for Computation & Technology >>> Louisiana State University, Baton Rouge, LA 70803, USA >>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >>> >>> >>> >>> >>> >>> >>> >> >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> > -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From lu_qin_2000 at yahoo.com Thu Jul 31 09:35:09 2014 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Thu, 31 Jul 2014 07:35:09 -0700 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> Message-ID: <1406817309.96050.YahooMailNeo@web160204.mail.bf1.yahoo.com> Do you mean the user has to reorder both rows and columns (unknowns)??Or the renumbering of columns are done by PETSc? Matthew's early response said that only rows need to renumbered by the user so that they are contiguous for each process. In my case, the order of unknowns have been determined before parallel?partitioning,?is this (just renumbering rows) Ok? ? Thanks, Qin???? From: Barry Smith To: Qin Lu Cc: Matthew Knepley ; petsc-users Sent: Wednesday, July 30, 2014 7:37 PM Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix On Jul 30, 2014, at 3:59 PM, Qin Lu wrote: > Renumbering row will make an unknown not associate with its row (equation), ? It is a symmetric reordering. In matrix terms both the rows and columns are reordered the same way. In terms of spaces both the domain and range spaces are reordered the same way. ? Barry > will this affect the solver performance? For example, will the infill pattern change in ILU factorization? >? > Thanks, > Qin > > From: Matthew Knepley > To: Qin Lu > Cc: Barry Smith ; petsc-users > Sent: Wednesday, July 30, 2014 3:50 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > In the context of domain decompostion, if the unknowns are ordered (to reduce the number of infills, for instance) in the way that each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? >? > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain 2 (assigned to process 2) owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make each process own consecutive rows? > > You renumber the rows once they are partitioned. > >? ? Matt > > >? > Thanks, > Qin > > From: Barry Smith > To: Qin Lu > Cc: petsc-users > Sent: Wednesday, July 30, 2014 2:49 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > Hello, > >? > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > >? Yes > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > >? However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > >? Barry > > > >? > > Thanks a lot for your help! > >? > > Regards, > > Qin > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 31 10:46:52 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Jul 2014 10:46:52 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406817309.96050.YahooMailNeo@web160204.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> <1406817309.96050.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: On Jul 31, 2014, at 9:35 AM, Qin Lu wrote: > Do you mean the user has to reorder both rows and columns (unknowns)? Or the renumbering of columns are done by PETSc? NO, never reorder the matrix. The degrees of freedom should be reordered to be contiquous before any matrix is ever created. Then the matrix and everything else is always computed with regard to this new ordering. Barry > Matthew's early response said that only rows need to renumbered by the user so that they are contiguous for each process. In my case, the order of unknowns have been determined before parallel partitioning, is this (just renumbering rows) Ok? > > Thanks, > Qin > > From: Barry Smith > To: Qin Lu > Cc: Matthew Knepley ; petsc-users > Sent: Wednesday, July 30, 2014 7:37 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 3:59 PM, Qin Lu wrote: > > > Renumbering row will make an unknown not associate with its row (equation), > > It is a symmetric reordering. In matrix terms both the rows and columns are reordered the same way. In terms of spaces both the domain and range spaces are reordered the same way. > > Barry > > > > will this affect the solver performance? For example, will the infill pattern change in ILU factorization? > > > > Thanks, > > Qin > > > > From: Matthew Knepley > > To: Qin Lu > > Cc: Barry Smith ; petsc-users > > Sent: Wednesday, July 30, 2014 3:50 PM > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > > In the context of domain decompostion, if the unknowns are ordered (to reduce the number of infills, for instance) in the way that each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? > > > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain 2 (assigned to process 2) owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make each process own consecutive rows? > > > > You renumber the rows once they are partitioned. > > > > Matt > > > > > > > > Thanks, > > Qin > > > > From: Barry Smith > > To: Qin Lu > > Cc: petsc-users > > Sent: Wednesday, July 30, 2014 2:49 PM > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > > > Hello, > > > > > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > > > > Yes > > > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > > > However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > > > > Barry > > > > > > > > > > Thanks a lot for your help! > > > > > > Regards, > > > Qin > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > From lu_qin_2000 at yahoo.com Thu Jul 31 11:04:14 2014 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Thu, 31 Jul 2014 09:04:14 -0700 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> <1406817309.96050.YahooMailNeo@web160204.mail.bf1.yahoo.com> Message-ID: <1406822654.10614.YahooMailNeo@web160203.mail.bf1.yahoo.com> Barry, ? If I understand you correctly, I should number all unknowns?owned by a process contiguously before I construct the matrix, right? ? Thanks, Qin?? ________________________________ From: Barry Smith To: Qin Lu Cc: Matthew Knepley ; petsc-users Sent: Thursday, July 31, 2014 10:46 AM Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix On Jul 31, 2014, at 9:35 AM, Qin Lu wrote: > Do you mean the user has to reorder both rows and columns (unknowns)? Or the renumbering of columns are done by PETSc? ? NO, never reorder the matrix. The degrees of freedom should be reordered to be contiquous before any matrix is ever created. Then the matrix and everything else is always computed with regard to this new ordering. ? Barry > Matthew's early response said that only rows need to renumbered by the user so that they are contiguous for each process. In my case, the order of unknowns have been determined before parallel partitioning, is this (just renumbering rows) Ok? >? > Thanks, > Qin? ? > > From: Barry Smith > To: Qin Lu > Cc: Matthew Knepley ; petsc-users > Sent: Wednesday, July 30, 2014 7:37 PM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 30, 2014, at 3:59 PM, Qin Lu wrote: > > > Renumbering row will make an unknown not associate with its row (equation), > >? It is a symmetric reordering. In matrix terms both the rows and columns are reordered the same way. In terms of spaces both the domain and range spaces are reordered the same way. > >? Barry > > > > will this affect the solver performance? For example, will the infill pattern change in ILU factorization? > >? > > Thanks, > > Qin > > > > From: Matthew Knepley > > To: Qin Lu > > Cc: Barry Smith ; petsc-users > > Sent: Wednesday, July 30, 2014 3:50 PM > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > > In the context of domain decompostion, if the unknowns are ordered (to reduce the number of infills, for instance) in the way that each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? > >? > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain 2 (assigned to process 2) owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make each process own consecutive rows? > > > > You renumber the rows once they are partitioned. > > > >? ? Matt > > > > > >? > > Thanks, > > Qin > > > > From: Barry Smith > > To: Qin Lu > > Cc: petsc-users > > Sent: Wednesday, July 30, 2014 2:49 PM > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > > > Hello, > > >? > > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > > > >? Yes > > > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > > >? However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > > > >? Barry > > > > > > >? > > > Thanks a lot for your help! > > >? > > > Regards, > > > Qin > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 31 11:06:02 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Jul 2014 11:06:02 -0500 Subject: [petsc-users] Partition of parallel AIJ sparce matrix In-Reply-To: <1406822654.10614.YahooMailNeo@web160203.mail.bf1.yahoo.com> References: <1406736500.74252.YahooMailNeo@web160202.mail.bf1.yahoo.com> <7F4963A3-A3B1-4EB1-AE39-EA7216270AA8@mcs.anl.gov> <1406752230.44074.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406753955.11053.YahooMailNeo@web160204.mail.bf1.yahoo.com> <5CE3A328-7CB4-412F-A5F9-4994DCBAF310@mcs.anl.gov> <1406817309.96050.YahooMailNeo@web160204.mail.bf1.yahoo.com> <1406822654.10614.YahooMailNeo@web160203.mail.bf1.yahoo.com> Message-ID: <0BC830D5-4447-48B6-A0E6-F22EE8D2A0F8@mcs.anl.gov> Yes On Jul 31, 2014, at 11:04 AM, Qin Lu wrote: > Barry, > > If I understand you correctly, I should number all unknowns owned by a process contiguously before I construct the matrix, right? > > Thanks, > Qin > > From: Barry Smith > To: Qin Lu > Cc: Matthew Knepley ; petsc-users > Sent: Thursday, July 31, 2014 10:46 AM > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > On Jul 31, 2014, at 9:35 AM, Qin Lu wrote: > > > Do you mean the user has to reorder both rows and columns (unknowns)? Or the renumbering of columns are done by PETSc? > > NO, never reorder the matrix. The degrees of freedom should be reordered to be contiquous before any matrix is ever created. Then the matrix and everything else is always computed with regard to this new ordering. > > Barry > > > > > > Matthew's early response said that only rows need to renumbered by the user so that they are contiguous for each process. In my case, the order of unknowns have been determined before parallel partitioning, is this (just renumbering rows) Ok? > > > > Thanks, > > Qin > > > > From: Barry Smith > > To: Qin Lu > > Cc: Matthew Knepley ; petsc-users > > Sent: Wednesday, July 30, 2014 7:37 PM > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > > > On Jul 30, 2014, at 3:59 PM, Qin Lu wrote: > > > > > Renumbering row will make an unknown not associate with its row (equation), > > > > It is a symmetric reordering. In matrix terms both the rows and columns are reordered the same way. In terms of spaces both the domain and range spaces are reordered the same way. > > > > Barry > > > > > > > will this affect the solver performance? For example, will the infill pattern change in ILU factorization? > > > > > > Thanks, > > > Qin > > > > > > From: Matthew Knepley > > > To: Qin Lu > > > Cc: Barry Smith ; petsc-users > > > Sent: Wednesday, July 30, 2014 3:50 PM > > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > > > On Wed, Jul 30, 2014 at 3:30 PM, Qin Lu wrote: > > > In the context of domain decompostion, if the unknowns are ordered (to reduce the number of infills, for instance) in the way that each subdomain may not own consecutive unknown index, does this mean the partition of the domain will be different from the partition of the matrix? > > > > > > For example, if subdomain 1 (assigned to process 1) owns unknowns 1 and 3 (associated with equation 1 and 3), subdomain 2 (assigned to process 2) owns unknowns 2 and 4 (associated with equation 2 and 4) , how can I make each process own consecutive rows? > > > > > > You renumber the rows once they are partitioned. > > > > > > Matt > > > > > > > > > > > > Thanks, > > > Qin > > > > > > From: Barry Smith > > > To: Qin Lu > > > Cc: petsc-users > > > Sent: Wednesday, July 30, 2014 2:49 PM > > > Subject: Re: [petsc-users] Partition of parallel AIJ sparce matrix > > > > > > > > > On Jul 30, 2014, at 11:08 AM, Qin Lu wrote: > > > > > > > Hello, > > > > > > > > Does a process have to own consecutive rows of the matrix? For example, suppose the global AIJ is 4x4, partitioned by 2 processes. Does process 1 have to own rows 1 and 2, process 2 own rows 3 and 4? > > > > > > Yes > > > > > > > Or process1 may own rows 1 and 3, and process 2 own row 2 and 4? > > > > > > However, the numbering of degrees of freedom is arbitrary. Just renumber you degrees of freedom so the first set is on process 0, the next on process 1 etc. > > > > > > Barry > > > > > > > > > > > > > > Thanks a lot for your help! > > > > > > > > Regards, > > > > Qin > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > From jyang29 at uh.edu Thu Jul 31 15:24:36 2014 From: jyang29 at uh.edu (Brian Yang) Date: Thu, 31 Jul 2014 15:24:36 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec Message-ID: Hi all, Here's an abstract of the problem, I got src and rec, they are 3D images with the same size, say Z, X, Y. We call one (Z, X) is a panel and then there's Y panels for both src and rec. BTW, they hold complex numbers. For example, for the *first* panel (always process the same panel) of src and rec: Take the first panel of src as our A (20x20), take the first column of first panel of rec as our b (20x1), solve the linear system and get x (20x1), go to next column of the first panel of rec until finish this panel, assemble all the solution x column by column (20x20). After finishing the first panel of src and rec, go to next... repeat. Hope I explained well of my problem. I used SeqDense matrix for A and Seq vector for b. Here's the flow, - start - all the nodes will share all the Y panels, each node will get part of them - each node will read in its own part of src and rec images - for each node, take a panel of src and rec *- create Mat and Vec, fill them* *- create KSP and solve by lsqr* *- get the solution* *- destroy all the petsc object, A, b, x (destroying KSP will give me error here!)* - repeat for the next panel Here's the time (seconds) output from node 2 (random choice): *entire time for this panel* *solving time* processing panel 1 *time*= 3.2995000E-02 *solver*= > 3.0995002E-02 > processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 > processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 > processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 > processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 > processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 > processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 > processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 > processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 > processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 > processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 > processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 > processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 > processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 > processing panel 15 time= 0.1009850 solver= 9.8985016E-02 > processing panel 16 time= 0.1119831 solver= 0.1099830 > processing panel 17 time= 0.1269809 solver= 0.1239820 > processing panel 18 time= 0.1469780 solver= 0.1439790 > processing panel 19 time= 0.1709731 solver= 0.1669741 > processing panel 20 time= 0.1909720 solver= 0.1869720 > processing panel 21 time= 0.2019690 solver= 0.1979700 > processing panel 22 time= 0.2239659 solver= 0.2199659 > processing panel 23 time= 0.2369640 solver= 0.2319648 > processing panel 24 time= 0.2499621 solver= 0.2449629 > processing panel 25 time= 0.2709589 solver= 0.2659600 > processing panel 26 time= 0.2869561 solver= 0.2829571 > processing panel 27 time= 0.3129530 solver= 0.3059540 > processing panel 28 time= 0.3389480 solver= 0.3329499 > processing panel 29 time= 0.3719430 solver= 0.3649440 > processing panel 30 time= 0.3949399 solver= 0.3879409 > processing panel 31 time= 0.4249353 solver= 0.4169374 > processing panel 32 time= 0.4549308 solver= 0.4469318 > processing panel 33 time= 0.4859262 solver= 0.4759283 > processing panel 34 time= 0.5119228 solver= 0.5019240 > processing panel 35 time= 0.5449171 solver= 0.5349178 > processing panel 36 time= 0.5689130 solver= 0.5579152 > processing panel 37 time= 0.5959096 solver= 0.5849104 > processing panel 38 time= 0.6199055 solver= 0.6079073 > You could see the time for solving the panels are increasing all the time. The panel number here is the local one. If I start to solve from panel 40 (random choice): processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 > processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 > processing panel 42 time= 0.1309800 solver= 0.1299810 > processing panel 43 time= 0.1719730 solver= 0.1709740 > processing panel 44 time= 0.2119681 solver= 0.2109680 > processing panel 45 time= 0.2529620 solver= 0.2519621 > processing panel 46 time= 0.2919550 solver= 0.2909551 > processing panel 47 time= 0.3319499 solver= 0.3309500 > processing panel 48 time= 0.3719430 solver= 0.3709428 > processing panel 49 time= 0.4129372 solver= 0.4109371 > processing panel 50 time= 0.4529319 solver= 0.4509320 > processing panel 51 time= 0.4929240 solver= 0.4909239 > processing panel 52 time= 0.5339203 solver= 0.5319204 > processing panel 53 time= 0.5779119 solver= 0.5759130 > processing panel 54 time= 0.6199059 solver= 0.6179061 > processing panel 55 time= 0.6648979 solver= 0.6628990 > processing panel 56 time= 0.7248902 solver= 0.7218900 > processing panel 57 time= 0.7938790 solver= 0.7908792 > processing panel 58 time= 0.8728676 solver= 0.8698678 > processing panel 59 time= 0.9778509 solver= 0.9748516 > processing panel 60 time= 1.125830 solver= 1.122829 > processing panel 61 time= 1.273806 solver= 1.268806 > processing panel 62 time= 1.448780 solver= 1.444779 > processing panel 63 time= 1.647749 solver= 1.643749 > processing panel 64 time= 1.901712 solver= 1.896712 > processing panel 65 time= 2.143673 solver= 2.138674 > processing panel 66 time= 2.437630 solver= 2.431629 > processing panel 67 time= 2.744583 solver= 2.736586 > processing panel 68 time= 3.041536 solver= 3.035538 > The trend is the same, the time is increasing and also starts from a very quick one. Since I have thousands of panels for src and rec, the execution time is unbearable as it goes. So I am wondering whether I used the right method? or there's memory issue? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 31 15:35:06 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 31 Jul 2014 15:35:06 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: References: Message-ID: On Thu, Jul 31, 2014 at 3:24 PM, Brian Yang wrote: > Hi all, > > Here's an abstract of the problem, > > I got src and rec, they are 3D images with the same size, say Z, X, Y. > > We call one (Z, X) is a panel and then there's Y panels for both src and > rec. BTW, they hold complex numbers. > > For example, for the *first* panel (always process the same panel) of src > and rec: > > Take the first panel of src as our A (20x20), > take the first column of first panel of rec as our b (20x1), > solve the linear system and get x (20x1), > go to next column of the first panel of rec until finish this panel, > assemble all the solution x column by column (20x20). > This is a fine conceptual explanation of the algorithm, however I do not think you want to implement it this way. Since you are solving all these panels independently, you can just construct the block matrix, with each panel as a block and solve it all at once (they clearly fit into memory). This might not be optimal for multiple rhs. If the matrices really are dense and you have multiple rhs, then you should look at using Elemental. We have an interface to it, although I am not sure we have hooked up the multiple rhs solves. > After finishing the first panel of src and rec, go to next... repeat. > > > Hope I explained well of my problem. I used SeqDense matrix for A and Seq > vector for b. > > Here's the flow, > > - start > - all the nodes will share all the Y panels, each node will get part of > them > - each node will read in its own part of src and rec images > - for each node, take a panel of src and rec > > *- create Mat and Vec, fill them* > > *- create KSP and solve by lsqr* > > *- get the solution* > *- destroy all the petsc object, A, b, x (destroying KSP will give me > error here!)* > - repeat for the next panel > > > Here's the time (seconds) output from node 2 (random choice): > > *entire time for this panel* *solving > time* > > processing panel 1 *time*= 3.2995000E-02 *solver*= >> 3.0995002E-02 >> processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 >> processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 >> processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 >> processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 >> processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 >> processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 >> processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 >> processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 >> processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 >> processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 >> processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 >> processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 >> processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 >> processing panel 15 time= 0.1009850 solver= 9.8985016E-02 >> processing panel 16 time= 0.1119831 solver= 0.1099830 >> processing panel 17 time= 0.1269809 solver= 0.1239820 >> processing panel 18 time= 0.1469780 solver= 0.1439790 >> processing panel 19 time= 0.1709731 solver= 0.1669741 >> processing panel 20 time= 0.1909720 solver= 0.1869720 >> processing panel 21 time= 0.2019690 solver= 0.1979700 >> processing panel 22 time= 0.2239659 solver= 0.2199659 >> processing panel 23 time= 0.2369640 solver= 0.2319648 >> processing panel 24 time= 0.2499621 solver= 0.2449629 >> processing panel 25 time= 0.2709589 solver= 0.2659600 >> processing panel 26 time= 0.2869561 solver= 0.2829571 >> processing panel 27 time= 0.3129530 solver= 0.3059540 >> processing panel 28 time= 0.3389480 solver= 0.3329499 >> processing panel 29 time= 0.3719430 solver= 0.3649440 >> processing panel 30 time= 0.3949399 solver= 0.3879409 >> processing panel 31 time= 0.4249353 solver= 0.4169374 >> processing panel 32 time= 0.4549308 solver= 0.4469318 >> processing panel 33 time= 0.4859262 solver= 0.4759283 >> processing panel 34 time= 0.5119228 solver= 0.5019240 >> processing panel 35 time= 0.5449171 solver= 0.5349178 >> processing panel 36 time= 0.5689130 solver= 0.5579152 >> processing panel 37 time= 0.5959096 solver= 0.5849104 >> processing panel 38 time= 0.6199055 solver= 0.6079073 >> > > You could see the time for solving the panels are increasing all the time. > The panel number here is the local one. If I start to solve from panel 40 > (random choice): > It certainly looks like you have a growing memory footprint. It is likely to have happened when you extracted/replaced parts of the matrix, which I think is unnecessary as I said above. Thanks, Matt > processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 >> processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 >> processing panel 42 time= 0.1309800 solver= 0.1299810 >> processing panel 43 time= 0.1719730 solver= 0.1709740 >> processing panel 44 time= 0.2119681 solver= 0.2109680 >> processing panel 45 time= 0.2529620 solver= 0.2519621 >> processing panel 46 time= 0.2919550 solver= 0.2909551 >> processing panel 47 time= 0.3319499 solver= 0.3309500 >> processing panel 48 time= 0.3719430 solver= 0.3709428 >> processing panel 49 time= 0.4129372 solver= 0.4109371 >> processing panel 50 time= 0.4529319 solver= 0.4509320 >> processing panel 51 time= 0.4929240 solver= 0.4909239 >> processing panel 52 time= 0.5339203 solver= 0.5319204 >> processing panel 53 time= 0.5779119 solver= 0.5759130 >> processing panel 54 time= 0.6199059 solver= 0.6179061 >> processing panel 55 time= 0.6648979 solver= 0.6628990 >> processing panel 56 time= 0.7248902 solver= 0.7218900 >> processing panel 57 time= 0.7938790 solver= 0.7908792 >> processing panel 58 time= 0.8728676 solver= 0.8698678 >> processing panel 59 time= 0.9778509 solver= 0.9748516 >> processing panel 60 time= 1.125830 solver= 1.122829 >> processing panel 61 time= 1.273806 solver= 1.268806 >> processing panel 62 time= 1.448780 solver= 1.444779 >> processing panel 63 time= 1.647749 solver= 1.643749 >> processing panel 64 time= 1.901712 solver= 1.896712 >> processing panel 65 time= 2.143673 solver= 2.138674 >> processing panel 66 time= 2.437630 solver= 2.431629 >> processing panel 67 time= 2.744583 solver= 2.736586 >> processing panel 68 time= 3.041536 solver= 3.035538 >> > > The trend is the same, the time is increasing and also starts from a very > quick one. > > > Since I have thousands of panels for src and rec, the execution time is > unbearable as it goes. > So I am wondering whether I used the right method? or there's memory issue? > > Thanks. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From brianyang1106 at gmail.com Thu Jul 31 15:58:15 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Thu, 31 Jul 2014 15:58:15 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: References: Message-ID: I will check the block matrix, thanks Mat. On Thu, Jul 31, 2014 at 3:35 PM, Matthew Knepley wrote: > On Thu, Jul 31, 2014 at 3:24 PM, Brian Yang wrote: > >> Hi all, >> >> Here's an abstract of the problem, >> >> I got src and rec, they are 3D images with the same size, say Z, X, Y. >> >> We call one (Z, X) is a panel and then there's Y panels for both src and >> rec. BTW, they hold complex numbers. >> >> For example, for the *first* panel (always process the same panel) of >> src and rec: >> >> Take the first panel of src as our A (20x20), >> take the first column of first panel of rec as our b (20x1), >> solve the linear system and get x (20x1), >> go to next column of the first panel of rec until finish this panel, >> assemble all the solution x column by column (20x20). >> > > This is a fine conceptual explanation of the algorithm, however I do not > think you > want to implement it this way. Since you are solving all these panels > independently, > you can just construct the block matrix, with each panel as a block and > solve it all > at once (they clearly fit into memory). This might not be optimal for > multiple rhs. > > If the matrices really are dense and you have multiple rhs, then you > should look at > using Elemental. We have an interface to it, although I am not sure we > have hooked up > the multiple rhs solves. > > >> After finishing the first panel of src and rec, go to next... repeat. >> >> >> Hope I explained well of my problem. I used SeqDense matrix for A and Seq >> vector for b. >> >> Here's the flow, >> >> - start >> - all the nodes will share all the Y panels, each node will get part of >> them >> - each node will read in its own part of src and rec images >> - for each node, take a panel of src and rec >> >> *- create Mat and Vec, fill them* >> >> *- create KSP and solve by lsqr* >> >> *- get the solution* >> *- destroy all the petsc object, A, b, x (destroying KSP will give me >> error here!)* >> - repeat for the next panel >> >> >> Here's the time (seconds) output from node 2 (random choice): >> >> *entire time for this panel* *solving >> time* >> >> processing panel 1 *time*= 3.2995000E-02 *solver*= >>> 3.0995002E-02 >>> processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 >>> processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 >>> processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 >>> processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 >>> processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 >>> processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 >>> processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 >>> processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 >>> processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 >>> processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 >>> processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 >>> processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 >>> processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 >>> processing panel 15 time= 0.1009850 solver= 9.8985016E-02 >>> processing panel 16 time= 0.1119831 solver= 0.1099830 >>> processing panel 17 time= 0.1269809 solver= 0.1239820 >>> processing panel 18 time= 0.1469780 solver= 0.1439790 >>> processing panel 19 time= 0.1709731 solver= 0.1669741 >>> processing panel 20 time= 0.1909720 solver= 0.1869720 >>> processing panel 21 time= 0.2019690 solver= 0.1979700 >>> processing panel 22 time= 0.2239659 solver= 0.2199659 >>> processing panel 23 time= 0.2369640 solver= 0.2319648 >>> processing panel 24 time= 0.2499621 solver= 0.2449629 >>> processing panel 25 time= 0.2709589 solver= 0.2659600 >>> processing panel 26 time= 0.2869561 solver= 0.2829571 >>> processing panel 27 time= 0.3129530 solver= 0.3059540 >>> processing panel 28 time= 0.3389480 solver= 0.3329499 >>> processing panel 29 time= 0.3719430 solver= 0.3649440 >>> processing panel 30 time= 0.3949399 solver= 0.3879409 >>> processing panel 31 time= 0.4249353 solver= 0.4169374 >>> processing panel 32 time= 0.4549308 solver= 0.4469318 >>> processing panel 33 time= 0.4859262 solver= 0.4759283 >>> processing panel 34 time= 0.5119228 solver= 0.5019240 >>> processing panel 35 time= 0.5449171 solver= 0.5349178 >>> processing panel 36 time= 0.5689130 solver= 0.5579152 >>> processing panel 37 time= 0.5959096 solver= 0.5849104 >>> processing panel 38 time= 0.6199055 solver= 0.6079073 >>> >> >> You could see the time for solving the panels are increasing all the >> time. The panel number here is the local one. If I start to solve from >> panel 40 (random choice): >> > > It certainly looks like you have a growing memory footprint. It is likely > to have happened > when you extracted/replaced parts of the matrix, which I think is > unnecessary as I said above. > > Thanks, > > Matt > > >> processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 >>> processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 >>> processing panel 42 time= 0.1309800 solver= 0.1299810 >>> processing panel 43 time= 0.1719730 solver= 0.1709740 >>> processing panel 44 time= 0.2119681 solver= 0.2109680 >>> processing panel 45 time= 0.2529620 solver= 0.2519621 >>> processing panel 46 time= 0.2919550 solver= 0.2909551 >>> processing panel 47 time= 0.3319499 solver= 0.3309500 >>> processing panel 48 time= 0.3719430 solver= 0.3709428 >>> processing panel 49 time= 0.4129372 solver= 0.4109371 >>> processing panel 50 time= 0.4529319 solver= 0.4509320 >>> processing panel 51 time= 0.4929240 solver= 0.4909239 >>> processing panel 52 time= 0.5339203 solver= 0.5319204 >>> processing panel 53 time= 0.5779119 solver= 0.5759130 >>> processing panel 54 time= 0.6199059 solver= 0.6179061 >>> processing panel 55 time= 0.6648979 solver= 0.6628990 >>> processing panel 56 time= 0.7248902 solver= 0.7218900 >>> processing panel 57 time= 0.7938790 solver= 0.7908792 >>> processing panel 58 time= 0.8728676 solver= 0.8698678 >>> processing panel 59 time= 0.9778509 solver= 0.9748516 >>> processing panel 60 time= 1.125830 solver= 1.122829 >>> processing panel 61 time= 1.273806 solver= 1.268806 >>> processing panel 62 time= 1.448780 solver= 1.444779 >>> processing panel 63 time= 1.647749 solver= 1.643749 >>> processing panel 64 time= 1.901712 solver= 1.896712 >>> processing panel 65 time= 2.143673 solver= 2.138674 >>> processing panel 66 time= 2.437630 solver= 2.431629 >>> processing panel 67 time= 2.744583 solver= 2.736586 >>> processing panel 68 time= 3.041536 solver= 3.035538 >>> >> >> The trend is the same, the time is increasing and also starts from a very >> quick one. >> >> >> Since I have thousands of panels for src and rec, the execution time is >> unbearable as it goes. >> So I am wondering whether I used the right method? or there's memory >> issue? >> >> Thanks. >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 31 16:31:02 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Jul 2014 16:31:02 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: References: Message-ID: <93C175C5-E0B3-47C0-8790-5CEF9E50ED51@mcs.anl.gov> On Jul 31, 2014, at 3:24 PM, Brian Yang wrote: > Hi all, > > Here's an abstract of the problem, > > I got src and rec, they are 3D images with the same size, say Z, X, Y. > > We call one (Z, X) is a panel and then there's Y panels for both src and rec. BTW, they hold complex numbers. > > For example, for the first panel (always process the same panel) of src and rec: > > Take the first panel of src as our A (20x20), > take the first column of first panel of rec as our b (20x1), > solve the linear system and get x (20x1), > go to next column of the first panel of rec until finish this panel, > assemble all the solution x column by column (20x20). > > After finishing the first panel of src and rec, go to next... repeat. > > > Hope I explained well of my problem. I used SeqDense matrix for A and Seq vector for b. > > Here's the flow, > > - start > - all the nodes will share all the Y panels, each node will get part of them > - each node will read in its own part of src and rec images > - for each node, take a panel of src and rec > - create Mat and Vec, fill them > - create KSP and solve by lsqr > - get the solution > - destroy all the petsc object, A, b, x (destroying KSP will give me error here!) If you do not destroy the KSP then the other objects won?t get destroyed (since the KSP reference counts them). You need to determine why the KSP cannot be destroyed. What is the error message? Run with valgrind? > - repeat for the next panel > > > Here's the time (seconds) output from node 2 (random choice): > > entire time for this panel solving time > > processing panel 1 time= 3.2995000E-02 solver= 3.0995002E-02 > processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 > processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 > processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 > processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 > processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 > processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 > processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 > processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 > processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 > processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 > processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 > processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 > processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 > processing panel 15 time= 0.1009850 solver= 9.8985016E-02 > processing panel 16 time= 0.1119831 solver= 0.1099830 > processing panel 17 time= 0.1269809 solver= 0.1239820 > processing panel 18 time= 0.1469780 solver= 0.1439790 > processing panel 19 time= 0.1709731 solver= 0.1669741 > processing panel 20 time= 0.1909720 solver= 0.1869720 > processing panel 21 time= 0.2019690 solver= 0.1979700 > processing panel 22 time= 0.2239659 solver= 0.2199659 > processing panel 23 time= 0.2369640 solver= 0.2319648 > processing panel 24 time= 0.2499621 solver= 0.2449629 > processing panel 25 time= 0.2709589 solver= 0.2659600 > processing panel 26 time= 0.2869561 solver= 0.2829571 > processing panel 27 time= 0.3129530 solver= 0.3059540 > processing panel 28 time= 0.3389480 solver= 0.3329499 > processing panel 29 time= 0.3719430 solver= 0.3649440 > processing panel 30 time= 0.3949399 solver= 0.3879409 > processing panel 31 time= 0.4249353 solver= 0.4169374 > processing panel 32 time= 0.4549308 solver= 0.4469318 > processing panel 33 time= 0.4859262 solver= 0.4759283 > processing panel 34 time= 0.5119228 solver= 0.5019240 > processing panel 35 time= 0.5449171 solver= 0.5349178 > processing panel 36 time= 0.5689130 solver= 0.5579152 > processing panel 37 time= 0.5959096 solver= 0.5849104 > processing panel 38 time= 0.6199055 solver= 0.6079073 > > You could see the time for solving the panels are increasing all the time. The panel number here is the local one. If I start to solve from panel 40 (random choice): > > processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 > processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 > processing panel 42 time= 0.1309800 solver= 0.1299810 > processing panel 43 time= 0.1719730 solver= 0.1709740 > processing panel 44 time= 0.2119681 solver= 0.2109680 > processing panel 45 time= 0.2529620 solver= 0.2519621 > processing panel 46 time= 0.2919550 solver= 0.2909551 > processing panel 47 time= 0.3319499 solver= 0.3309500 > processing panel 48 time= 0.3719430 solver= 0.3709428 > processing panel 49 time= 0.4129372 solver= 0.4109371 > processing panel 50 time= 0.4529319 solver= 0.4509320 > processing panel 51 time= 0.4929240 solver= 0.4909239 > processing panel 52 time= 0.5339203 solver= 0.5319204 > processing panel 53 time= 0.5779119 solver= 0.5759130 > processing panel 54 time= 0.6199059 solver= 0.6179061 > processing panel 55 time= 0.6648979 solver= 0.6628990 > processing panel 56 time= 0.7248902 solver= 0.7218900 > processing panel 57 time= 0.7938790 solver= 0.7908792 > processing panel 58 time= 0.8728676 solver= 0.8698678 > processing panel 59 time= 0.9778509 solver= 0.9748516 > processing panel 60 time= 1.125830 solver= 1.122829 > processing panel 61 time= 1.273806 solver= 1.268806 > processing panel 62 time= 1.448780 solver= 1.444779 > processing panel 63 time= 1.647749 solver= 1.643749 > processing panel 64 time= 1.901712 solver= 1.896712 > processing panel 65 time= 2.143673 solver= 2.138674 > processing panel 66 time= 2.437630 solver= 2.431629 > processing panel 67 time= 2.744583 solver= 2.736586 > processing panel 68 time= 3.041536 solver= 3.035538 > > The trend is the same, the time is increasing and also starts from a very quick one. > > > Since I have thousands of panels for src and rec, the execution time is unbearable as it goes. > So I am wondering whether I used the right method? or there's memory issue? > > Thanks. From jyang29 at uh.edu Thu Jul 31 16:36:55 2014 From: jyang29 at uh.edu (Brian Yang) Date: Thu, 31 Jul 2014 16:36:55 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: <93C175C5-E0B3-47C0-8790-5CEF9E50ED51@mcs.anl.gov> References: <93C175C5-E0B3-47C0-8790-5CEF9E50ED51@mcs.anl.gov> Message-ID: Here's the message (all the nodes are the same): [8]PETSC ERROR: ------------------------------------------------------------------------ [8]PETSC ERROR: VecDestroy() line 539 in src/vec/vec/interface/vector.c [8]PETSC ERROR: KSPReset() line 740 in src/ksp/ksp/interface/itfunc.c [8]PETSC ERROR: KSPDestroy() line 783 in src/ksp/ksp/interface/itfunc.c [6]PETSC ERROR: --------------------- Error Message ------------------------------------ [6]PETSC ERROR: Invalid argument! [6]PETSC ERROR: Wrong type of object: Parameter # 1! [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 [6]PETSC ERROR: See docs/changes/index.html for recent updates. [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [6]PETSC ERROR: See docs/index.html for manual pages. [6]PETSC ERROR: ------------------------------------------------------------------------ On Thu, Jul 31, 2014 at 4:31 PM, Barry Smith wrote: > > On Jul 31, 2014, at 3:24 PM, Brian Yang wrote: > > > Hi all, > > > > Here's an abstract of the problem, > > > > I got src and rec, they are 3D images with the same size, say Z, X, Y. > > > > We call one (Z, X) is a panel and then there's Y panels for both src and > rec. BTW, they hold complex numbers. > > > > For example, for the first panel (always process the same panel) of src > and rec: > > > > Take the first panel of src as our A (20x20), > > take the first column of first panel of rec as our b (20x1), > > solve the linear system and get x (20x1), > > go to next column of the first panel of rec until finish this panel, > > assemble all the solution x column by column (20x20). > > > > After finishing the first panel of src and rec, go to next... repeat. > > > > > > Hope I explained well of my problem. I used SeqDense matrix for A and > Seq vector for b. > > > > Here's the flow, > > > > - start > > - all the nodes will share all the Y panels, each node will get part of > them > > - each node will read in its own part of src and rec images > > - for each node, take a panel of src and rec > > - create Mat and Vec, fill them > > - create KSP and solve by lsqr > > - get the solution > > - destroy all the petsc object, A, b, x (destroying KSP will give me > error here!) > > If you do not destroy the KSP then the other objects won't get > destroyed (since the KSP reference counts them). You need to determine why > the KSP cannot be destroyed. What is the error message? Run with valgrind? > > > > - repeat for the next panel > > > > > > Here's the time (seconds) output from node 2 (random choice): > > > > entire time for this panel solving > time > > > > processing panel 1 time= 3.2995000E-02 solver= 3.0995002E-02 > > processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 > > processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 > > processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 > > processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 > > processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 > > processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 > > processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 > > processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 > > processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 > > processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 > > processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 > > processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 > > processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 > > processing panel 15 time= 0.1009850 solver= 9.8985016E-02 > > processing panel 16 time= 0.1119831 solver= 0.1099830 > > processing panel 17 time= 0.1269809 solver= 0.1239820 > > processing panel 18 time= 0.1469780 solver= 0.1439790 > > processing panel 19 time= 0.1709731 solver= 0.1669741 > > processing panel 20 time= 0.1909720 solver= 0.1869720 > > processing panel 21 time= 0.2019690 solver= 0.1979700 > > processing panel 22 time= 0.2239659 solver= 0.2199659 > > processing panel 23 time= 0.2369640 solver= 0.2319648 > > processing panel 24 time= 0.2499621 solver= 0.2449629 > > processing panel 25 time= 0.2709589 solver= 0.2659600 > > processing panel 26 time= 0.2869561 solver= 0.2829571 > > processing panel 27 time= 0.3129530 solver= 0.3059540 > > processing panel 28 time= 0.3389480 solver= 0.3329499 > > processing panel 29 time= 0.3719430 solver= 0.3649440 > > processing panel 30 time= 0.3949399 solver= 0.3879409 > > processing panel 31 time= 0.4249353 solver= 0.4169374 > > processing panel 32 time= 0.4549308 solver= 0.4469318 > > processing panel 33 time= 0.4859262 solver= 0.4759283 > > processing panel 34 time= 0.5119228 solver= 0.5019240 > > processing panel 35 time= 0.5449171 solver= 0.5349178 > > processing panel 36 time= 0.5689130 solver= 0.5579152 > > processing panel 37 time= 0.5959096 solver= 0.5849104 > > processing panel 38 time= 0.6199055 solver= 0.6079073 > > > > You could see the time for solving the panels are increasing all the > time. The panel number here is the local one. If I start to solve from > panel 40 (random choice): > > > > processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 > > processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 > > processing panel 42 time= 0.1309800 solver= 0.1299810 > > processing panel 43 time= 0.1719730 solver= 0.1709740 > > processing panel 44 time= 0.2119681 solver= 0.2109680 > > processing panel 45 time= 0.2529620 solver= 0.2519621 > > processing panel 46 time= 0.2919550 solver= 0.2909551 > > processing panel 47 time= 0.3319499 solver= 0.3309500 > > processing panel 48 time= 0.3719430 solver= 0.3709428 > > processing panel 49 time= 0.4129372 solver= 0.4109371 > > processing panel 50 time= 0.4529319 solver= 0.4509320 > > processing panel 51 time= 0.4929240 solver= 0.4909239 > > processing panel 52 time= 0.5339203 solver= 0.5319204 > > processing panel 53 time= 0.5779119 solver= 0.5759130 > > processing panel 54 time= 0.6199059 solver= 0.6179061 > > processing panel 55 time= 0.6648979 solver= 0.6628990 > > processing panel 56 time= 0.7248902 solver= 0.7218900 > > processing panel 57 time= 0.7938790 solver= 0.7908792 > > processing panel 58 time= 0.8728676 solver= 0.8698678 > > processing panel 59 time= 0.9778509 solver= 0.9748516 > > processing panel 60 time= 1.125830 solver= 1.122829 > > processing panel 61 time= 1.273806 solver= 1.268806 > > processing panel 62 time= 1.448780 solver= 1.444779 > > processing panel 63 time= 1.647749 solver= 1.643749 > > processing panel 64 time= 1.901712 solver= 1.896712 > > processing panel 65 time= 2.143673 solver= 2.138674 > > processing panel 66 time= 2.437630 solver= 2.431629 > > processing panel 67 time= 2.744583 solver= 2.736586 > > processing panel 68 time= 3.041536 solver= 3.035538 > > > > The trend is the same, the time is increasing and also starts from a > very quick one. > > > > > > Since I have thousands of panels for src and rec, the execution time is > unbearable as it goes. > > So I am wondering whether I used the right method? or there's memory > issue? > > > > Thanks. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jyang29 at uh.edu Thu Jul 31 16:43:26 2014 From: jyang29 at uh.edu (Brian Yang) Date: Thu, 31 Jul 2014 16:43:26 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: References: <93C175C5-E0B3-47C0-8790-5CEF9E50ED51@mcs.anl.gov> Message-ID: I found one thing, as long as I don't destroy the vector that holds the solution from KSPGetSolution, I could successfully destroy all other objects (including KSP) for each panel. On Thu, Jul 31, 2014 at 4:36 PM, Brian Yang wrote: > Here's the message (all the nodes are the same): > > [8]PETSC ERROR: > ------------------------------------------------------------------------ > [8]PETSC ERROR: VecDestroy() line 539 in src/vec/vec/interface/vector.c > [8]PETSC ERROR: KSPReset() line 740 in src/ksp/ksp/interface/itfunc.c > [8]PETSC ERROR: KSPDestroy() line 783 in src/ksp/ksp/interface/itfunc.c > [6]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [6]PETSC ERROR: Invalid argument! > [6]PETSC ERROR: Wrong type of object: Parameter # 1! > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: > ------------------------------------------------------------------------ > > > On Thu, Jul 31, 2014 at 4:31 PM, Barry Smith wrote: > >> >> On Jul 31, 2014, at 3:24 PM, Brian Yang wrote: >> >> > Hi all, >> > >> > Here's an abstract of the problem, >> > >> > I got src and rec, they are 3D images with the same size, say Z, X, Y. >> > >> > We call one (Z, X) is a panel and then there's Y panels for both src >> and rec. BTW, they hold complex numbers. >> > >> > For example, for the first panel (always process the same panel) of src >> and rec: >> > >> > Take the first panel of src as our A (20x20), >> > take the first column of first panel of rec as our b (20x1), >> > solve the linear system and get x (20x1), >> > go to next column of the first panel of rec until finish this panel, >> > assemble all the solution x column by column (20x20). >> > >> > After finishing the first panel of src and rec, go to next... repeat. >> > >> > >> > Hope I explained well of my problem. I used SeqDense matrix for A and >> Seq vector for b. >> > >> > Here's the flow, >> > >> > - start >> > - all the nodes will share all the Y panels, each node will get part of >> them >> > - each node will read in its own part of src and rec images >> > - for each node, take a panel of src and rec >> > - create Mat and Vec, fill them >> > - create KSP and solve by lsqr >> > - get the solution >> > - destroy all the petsc object, A, b, x (destroying KSP will give me >> error here!) >> >> If you do not destroy the KSP then the other objects won't get >> destroyed (since the KSP reference counts them). You need to determine why >> the KSP cannot be destroyed. What is the error message? Run with valgrind? >> >> >> > - repeat for the next panel >> > >> > >> > Here's the time (seconds) output from node 2 (random choice): >> > >> > entire time for this panel >> solving time >> > >> > processing panel 1 time= 3.2995000E-02 solver= >> 3.0995002E-02 >> > processing panel 2 time= 3.5994001E-02 solver= >> 3.4995001E-02 >> > processing panel 3 time= 3.9994001E-02 solver= >> 3.8994007E-02 >> > processing panel 4 time= 4.4993997E-02 solver= >> 4.3993995E-02 >> > processing panel 5 time= 4.8991993E-02 solver= >> 4.6992987E-02 >> > processing panel 6 time= 5.4991007E-02 solver= >> 5.3991005E-02 >> > processing panel 7 time= 5.8990985E-02 solver= >> 5.7990998E-02 >> > processing panel 8 time= 6.3990027E-02 solver= >> 6.1990023E-02 >> > processing panel 9 time= 6.8989992E-02 solver= >> 6.6990018E-02 >> > processing panel 10 time= 7.3989004E-02 solver= >> 7.1989000E-02 >> > processing panel 11 time= 7.7987969E-02 solver= >> 7.6987982E-02 >> > processing panel 12 time= 8.1988037E-02 solver= >> 7.9988003E-02 >> > processing panel 13 time= 8.8985980E-02 solver= >> 8.6987019E-02 >> > processing panel 14 time= 9.4985008E-02 solver= >> 9.2984974E-02 >> > processing panel 15 time= 0.1009850 solver= >> 9.8985016E-02 >> > processing panel 16 time= 0.1119831 solver= 0.1099830 >> > processing panel 17 time= 0.1269809 solver= 0.1239820 >> > processing panel 18 time= 0.1469780 solver= 0.1439790 >> > processing panel 19 time= 0.1709731 solver= 0.1669741 >> > processing panel 20 time= 0.1909720 solver= 0.1869720 >> > processing panel 21 time= 0.2019690 solver= 0.1979700 >> > processing panel 22 time= 0.2239659 solver= 0.2199659 >> > processing panel 23 time= 0.2369640 solver= 0.2319648 >> > processing panel 24 time= 0.2499621 solver= 0.2449629 >> > processing panel 25 time= 0.2709589 solver= 0.2659600 >> > processing panel 26 time= 0.2869561 solver= 0.2829571 >> > processing panel 27 time= 0.3129530 solver= 0.3059540 >> > processing panel 28 time= 0.3389480 solver= 0.3329499 >> > processing panel 29 time= 0.3719430 solver= 0.3649440 >> > processing panel 30 time= 0.3949399 solver= 0.3879409 >> > processing panel 31 time= 0.4249353 solver= 0.4169374 >> > processing panel 32 time= 0.4549308 solver= 0.4469318 >> > processing panel 33 time= 0.4859262 solver= 0.4759283 >> > processing panel 34 time= 0.5119228 solver= 0.5019240 >> > processing panel 35 time= 0.5449171 solver= 0.5349178 >> > processing panel 36 time= 0.5689130 solver= 0.5579152 >> > processing panel 37 time= 0.5959096 solver= 0.5849104 >> > processing panel 38 time= 0.6199055 solver= 0.6079073 >> > >> > You could see the time for solving the panels are increasing all the >> time. The panel number here is the local one. If I start to solve from >> panel 40 (random choice): >> > >> > processing panel 40 time= 5.5992007E-02 solver= >> 5.1991999E-02 >> > processing panel 41 time= 9.1986001E-02 solver= >> 9.0986013E-02 >> > processing panel 42 time= 0.1309800 solver= 0.1299810 >> > processing panel 43 time= 0.1719730 solver= 0.1709740 >> > processing panel 44 time= 0.2119681 solver= 0.2109680 >> > processing panel 45 time= 0.2529620 solver= 0.2519621 >> > processing panel 46 time= 0.2919550 solver= 0.2909551 >> > processing panel 47 time= 0.3319499 solver= 0.3309500 >> > processing panel 48 time= 0.3719430 solver= 0.3709428 >> > processing panel 49 time= 0.4129372 solver= 0.4109371 >> > processing panel 50 time= 0.4529319 solver= 0.4509320 >> > processing panel 51 time= 0.4929240 solver= 0.4909239 >> > processing panel 52 time= 0.5339203 solver= 0.5319204 >> > processing panel 53 time= 0.5779119 solver= 0.5759130 >> > processing panel 54 time= 0.6199059 solver= 0.6179061 >> > processing panel 55 time= 0.6648979 solver= 0.6628990 >> > processing panel 56 time= 0.7248902 solver= 0.7218900 >> > processing panel 57 time= 0.7938790 solver= 0.7908792 >> > processing panel 58 time= 0.8728676 solver= 0.8698678 >> > processing panel 59 time= 0.9778509 solver= 0.9748516 >> > processing panel 60 time= 1.125830 solver= 1.122829 >> > processing panel 61 time= 1.273806 solver= 1.268806 >> > processing panel 62 time= 1.448780 solver= 1.444779 >> > processing panel 63 time= 1.647749 solver= 1.643749 >> > processing panel 64 time= 1.901712 solver= 1.896712 >> > processing panel 65 time= 2.143673 solver= 2.138674 >> > processing panel 66 time= 2.437630 solver= 2.431629 >> > processing panel 67 time= 2.744583 solver= 2.736586 >> > processing panel 68 time= 3.041536 solver= 3.035538 >> > >> > The trend is the same, the time is increasing and also starts from a >> very quick one. >> > >> > >> > Since I have thousands of panels for src and rec, the execution time is >> unbearable as it goes. >> > So I am wondering whether I used the right method? or there's memory >> issue? >> > >> > Thanks. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jul 31 16:53:37 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Jul 2014 16:53:37 -0500 Subject: [petsc-users] Performance question using seq Mat and Vec In-Reply-To: References: <93C175C5-E0B3-47C0-8790-5CEF9E50ED51@mcs.anl.gov> Message-ID: On Jul 31, 2014, at 4:43 PM, Brian Yang wrote: > I found one thing, as long as I don't destroy the vector that holds the solution from KSPGetSolution, I could successfully destroy all other objects (including KSP) for each panel. You are not suppose to destroy that one. Since it is obtained with a ?Get? it means you only borrowed it and should not destroy it yourself. So do not destroy this instead destroy the KSP etc. Barry > > > On Thu, Jul 31, 2014 at 4:36 PM, Brian Yang wrote: > Here's the message (all the nodes are the same): > > [8]PETSC ERROR: ------------------------------------------------------------------------ > [8]PETSC ERROR: VecDestroy() line 539 in src/vec/vec/interface/vector.c > [8]PETSC ERROR: KSPReset() line 740 in src/ksp/ksp/interface/itfunc.c > [8]PETSC ERROR: KSPDestroy() line 783 in src/ksp/ksp/interface/itfunc.c > [6]PETSC ERROR: --------------------- Error Message ------------------------------------ > [6]PETSC ERROR: Invalid argument! > [6]PETSC ERROR: Wrong type of object: Parameter # 1! > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: ------------------------------------------------------------------------ > > > On Thu, Jul 31, 2014 at 4:31 PM, Barry Smith wrote: > > On Jul 31, 2014, at 3:24 PM, Brian Yang wrote: > > > Hi all, > > > > Here's an abstract of the problem, > > > > I got src and rec, they are 3D images with the same size, say Z, X, Y. > > > > We call one (Z, X) is a panel and then there's Y panels for both src and rec. BTW, they hold complex numbers. > > > > For example, for the first panel (always process the same panel) of src and rec: > > > > Take the first panel of src as our A (20x20), > > take the first column of first panel of rec as our b (20x1), > > solve the linear system and get x (20x1), > > go to next column of the first panel of rec until finish this panel, > > assemble all the solution x column by column (20x20). > > > > After finishing the first panel of src and rec, go to next... repeat. > > > > > > Hope I explained well of my problem. I used SeqDense matrix for A and Seq vector for b. > > > > Here's the flow, > > > > - start > > - all the nodes will share all the Y panels, each node will get part of them > > - each node will read in its own part of src and rec images > > - for each node, take a panel of src and rec > > - create Mat and Vec, fill them > > - create KSP and solve by lsqr > > - get the solution > > - destroy all the petsc object, A, b, x (destroying KSP will give me error here!) > > If you do not destroy the KSP then the other objects won?t get destroyed (since the KSP reference counts them). You need to determine why the KSP cannot be destroyed. What is the error message? Run with valgrind? > > > > - repeat for the next panel > > > > > > Here's the time (seconds) output from node 2 (random choice): > > > > entire time for this panel solving time > > > > processing panel 1 time= 3.2995000E-02 solver= 3.0995002E-02 > > processing panel 2 time= 3.5994001E-02 solver= 3.4995001E-02 > > processing panel 3 time= 3.9994001E-02 solver= 3.8994007E-02 > > processing panel 4 time= 4.4993997E-02 solver= 4.3993995E-02 > > processing panel 5 time= 4.8991993E-02 solver= 4.6992987E-02 > > processing panel 6 time= 5.4991007E-02 solver= 5.3991005E-02 > > processing panel 7 time= 5.8990985E-02 solver= 5.7990998E-02 > > processing panel 8 time= 6.3990027E-02 solver= 6.1990023E-02 > > processing panel 9 time= 6.8989992E-02 solver= 6.6990018E-02 > > processing panel 10 time= 7.3989004E-02 solver= 7.1989000E-02 > > processing panel 11 time= 7.7987969E-02 solver= 7.6987982E-02 > > processing panel 12 time= 8.1988037E-02 solver= 7.9988003E-02 > > processing panel 13 time= 8.8985980E-02 solver= 8.6987019E-02 > > processing panel 14 time= 9.4985008E-02 solver= 9.2984974E-02 > > processing panel 15 time= 0.1009850 solver= 9.8985016E-02 > > processing panel 16 time= 0.1119831 solver= 0.1099830 > > processing panel 17 time= 0.1269809 solver= 0.1239820 > > processing panel 18 time= 0.1469780 solver= 0.1439790 > > processing panel 19 time= 0.1709731 solver= 0.1669741 > > processing panel 20 time= 0.1909720 solver= 0.1869720 > > processing panel 21 time= 0.2019690 solver= 0.1979700 > > processing panel 22 time= 0.2239659 solver= 0.2199659 > > processing panel 23 time= 0.2369640 solver= 0.2319648 > > processing panel 24 time= 0.2499621 solver= 0.2449629 > > processing panel 25 time= 0.2709589 solver= 0.2659600 > > processing panel 26 time= 0.2869561 solver= 0.2829571 > > processing panel 27 time= 0.3129530 solver= 0.3059540 > > processing panel 28 time= 0.3389480 solver= 0.3329499 > > processing panel 29 time= 0.3719430 solver= 0.3649440 > > processing panel 30 time= 0.3949399 solver= 0.3879409 > > processing panel 31 time= 0.4249353 solver= 0.4169374 > > processing panel 32 time= 0.4549308 solver= 0.4469318 > > processing panel 33 time= 0.4859262 solver= 0.4759283 > > processing panel 34 time= 0.5119228 solver= 0.5019240 > > processing panel 35 time= 0.5449171 solver= 0.5349178 > > processing panel 36 time= 0.5689130 solver= 0.5579152 > > processing panel 37 time= 0.5959096 solver= 0.5849104 > > processing panel 38 time= 0.6199055 solver= 0.6079073 > > > > You could see the time for solving the panels are increasing all the time. The panel number here is the local one. If I start to solve from panel 40 (random choice): > > > > processing panel 40 time= 5.5992007E-02 solver= 5.1991999E-02 > > processing panel 41 time= 9.1986001E-02 solver= 9.0986013E-02 > > processing panel 42 time= 0.1309800 solver= 0.1299810 > > processing panel 43 time= 0.1719730 solver= 0.1709740 > > processing panel 44 time= 0.2119681 solver= 0.2109680 > > processing panel 45 time= 0.2529620 solver= 0.2519621 > > processing panel 46 time= 0.2919550 solver= 0.2909551 > > processing panel 47 time= 0.3319499 solver= 0.3309500 > > processing panel 48 time= 0.3719430 solver= 0.3709428 > > processing panel 49 time= 0.4129372 solver= 0.4109371 > > processing panel 50 time= 0.4529319 solver= 0.4509320 > > processing panel 51 time= 0.4929240 solver= 0.4909239 > > processing panel 52 time= 0.5339203 solver= 0.5319204 > > processing panel 53 time= 0.5779119 solver= 0.5759130 > > processing panel 54 time= 0.6199059 solver= 0.6179061 > > processing panel 55 time= 0.6648979 solver= 0.6628990 > > processing panel 56 time= 0.7248902 solver= 0.7218900 > > processing panel 57 time= 0.7938790 solver= 0.7908792 > > processing panel 58 time= 0.8728676 solver= 0.8698678 > > processing panel 59 time= 0.9778509 solver= 0.9748516 > > processing panel 60 time= 1.125830 solver= 1.122829 > > processing panel 61 time= 1.273806 solver= 1.268806 > > processing panel 62 time= 1.448780 solver= 1.444779 > > processing panel 63 time= 1.647749 solver= 1.643749 > > processing panel 64 time= 1.901712 solver= 1.896712 > > processing panel 65 time= 2.143673 solver= 2.138674 > > processing panel 66 time= 2.437630 solver= 2.431629 > > processing panel 67 time= 2.744583 solver= 2.736586 > > processing panel 68 time= 3.041536 solver= 3.035538 > > > > The trend is the same, the time is increasing and also starts from a very quick one. > > > > > > Since I have thousands of panels for src and rec, the execution time is unbearable as it goes. > > So I am wondering whether I used the right method? or there's memory issue? > > > > Thanks. > > >