From agrayver at gfz-potsdam.de Thu Dec 1 01:50:27 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 01 Dec 2011 08:50:27 +0100 Subject: [petsc-users] MatMumpsSetIcntl from Fortran In-Reply-To: References: Message-ID: <4ED731C3.9010505@gfz-potsdam.de> Hong, This should be enough to reproduce the problem: call KSPCreate(PETSC_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr);CHKERRQ(ierr) call KSPSetType(ksp,KSPPREONLY,ierr);CHKERRQ(ierr) call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) call PCSetType(pc,PCCHOLESKY,ierr);CHKERRQ(ierr) call PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS,ierr);CHKERRQ(ierr) call PCFactorSetUpMatSolverPackage(pc,ierr);CHKERRQ(ierr) call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) call KSPSetFromOptions(ksp,ierr);CHKERRQ(ierr) icntl = 2; ival = 0; call MatMumpsSetIcntl(F,icntl,ival,ierr); Regards, Alexander On 30.11.2011 23:43, Hong Zhang wrote: >> Sorry if I wasn't clear. "c" example works fine, I know, what I meant is >> that if you try to implement lines 150-171 from it on FORTRAN you will see >> the problem I reported. >> If you need particularly my FORTRAN code I can send it tomorrow. > This would save my time :-) > Appreciate. > > Hong >> Regards, >> Alexander >> >> >> ----- Reply message ----- >> From: "Hong Zhang" >> To: "PETSc users list" >> Subject: [petsc-users] MatMumpsSetIcntl from Fortran >> Date: Wed, Nov 30, 2011 9:37 pm >> >> >> Alexander : >> >>> I just rewrote code concerning mumps from this example (lines 150-170): >>> >>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c.html >> Where is your Fortran code? ex52.c works fine. >> Hong >> >>> On 30.11.2011 17:40, Hong Zhang wrote: >>>> Alexander: >>>> >>>>> Has anybody tried to use MatMumpsSetIcntl from Fortran? >>>> We are not aware of it. >>>> >>>>> Because when I try to call it I fall into infinite recursion in >>>>> function: >>>> Can you give me a short Fortran code that repeats this error for >>>> investigating? >>>> Meanwhile, you can use runtime option '-mat_mumps_icntl_xxx<>' to get >>>> your code run. >>>> >>>> Hong >>>>> PetscErrorCode MatMumpsSetIcntl(Mat F,PetscInt icntl,PetscInt ival) >>>>> { >>>>> PetscErrorCode ierr; >>>>> >>>>> PetscFunctionBegin; >>>>> PetscValidLogicalCollectiveInt(F,icntl,2); >>>>> PetscValidLogicalCollectiveInt(F,ival,3); >>>>> ierr = >>>>> >>>>> >>>>> PetscTryMethod(F,"MatMumpsSetIcntl_C",(Mat,PetscInt,PetscInt),(F,icntl,ival));CHKERRQ(ierr); >>>>> PetscFunctionReturn(0); >>>>> } >>>>> >>>>> At the moment when program crashes call stack looks like: >>>>> >>>>> __libc_memalign, >>>>> FP=7fff342ca010 >>>>> PetscMallocAlign, FP=7fff342ca080 >>>>> PetscTrMallocDefault, FP=7fff342ca180 >>>>> PetscStrallocpy, FP=7fff342ca230 >>>>> PetscFListGetPathAndFunction, FP=7fff342cb2e0 >>>>> PetscFListFind, FP=7fff342cb520 >>>>> PetscObjectQueryFunction_Petsc, FP=7fff342cb590 >>>>> PetscObjectQueryFunction, FP=7fff342cb620 >>>>> MatMumpsSetIcntl, FP=7fff342cb720 >>>>> MatMumpsSetIcntl, FP=7fff342cb820 >>>>> MatMumpsSetIcntl, FP=7fff342cb920 >>>>> MatMumpsSetIcntl, FP=7fff342cba20 >>>>> MatMumpsSetIcntl, FP=7fff342cbb20 >>>>> MatMumpsSetIcntl, FP=7fff342cbc20 >>>>> MatMumpsSetIcntl, FP=7fff342cbd20 >>>>> MatMumpsSetIcntl, FP=7fff342cbe20 >>>>> MatMumpsSetIcntl, FP=7fff342cbf20 >>>>> ... (Hundreds of MatMumpsSetIcntl) ... >>>>> >>>>> What can I do about that? >>>>> >>>>> Regards, >>>>> Alexander >>> From Andrew.Parker2 at baesystems.com Thu Dec 1 04:28:48 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Thu, 1 Dec 2011 10:28:48 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 Message-ID: Hi, I've been using PETSc for a conventional full parallel implicit solver, currently in MPIBAIJ format. However, I'd like to understand the performance verse memory and speed trade-offs of just having a block diagonal system. In this case I do not scatter to the off diagonal blocks. Each block is 7x7 in size and fully dense. In my view this should be a point implicit solution of the system. I am looking for the best way to solve this in PETSc, clearly because of the lack of coupling between neighbouring cells by only having a block diagonal it's actually a 7x7 dense linear solve, but to test my theory at present I'd prefer to just not scatter to off diagonals, and use: MATBDIAG with my current ksp solver and take it from there. BDIAG seems to have been removed as off 3.0: http://www.mcs.anl.gov/petsc/documentation/changes/300.html Can anybody help me with this, your thoughts on the above and what I should use to solve this reduced system would be appreciated?? Cheers, Andy ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredva at ifi.uio.no Thu Dec 1 05:39:12 2011 From: fredva at ifi.uio.no (Fredrik Heffer Valdmanis) Date: Thu, 1 Dec 2011 12:39:12 +0100 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: 2011/11/29 Matthew Knepley > On Tue, Nov 29, 2011 at 10:37 AM, Fredrik Heffer Valdmanis < > fredva at ifi.uio.no> wrote: > >> 2011/11/29 Matthew Knepley >> >>> On Tue, Nov 29, 2011 at 2:38 AM, Fredrik Heffer Valdmanis < >>> fredva at ifi.uio.no> wrote: >>> >>>> 2011/10/28 Matthew Knepley >>>> >>>>> On Fri, Oct 28, 2011 at 10:24 AM, Fredrik Heffer Valdmanis < >>>>> fredva at ifi.uio.no> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am working on integrating the new GPU based vectors and matrices >>>>>> into FEniCS. Now, I'm looking at the possibility for getting some speedup >>>>>> during finite element assembly, specifically when inserting the local >>>>>> element matrix into the global element matrix. In that regard, I have a few >>>>>> questions I hope you can help me out with: >>>>>> >>>>>> - When calling MatSetValues with a MATSEQAIJCUSP matrix as parameter, >>>>>> what exactly is it that happens? As far as I can see, MatSetValues is not >>>>>> implemented for GPU based matrices, neither is the mat->ops->setvalues set >>>>>> to point at any function for this Mat type. >>>>>> >>>>> >>>>> Yes, MatSetValues always operates on the CPU side. It would not make >>>>> sense to do individual operations on the GPU. >>>>> >>>>> I have written batched of assembly for element matrices that are all >>>>> the same size: >>>>> >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBatch.html >>>>> >>>>> >>>>>> - Is it such that matrices are assembled in their entirety on the >>>>>> CPU, and then copied over to the GPU (after calling MatAssemblyBegin)? Or >>>>>> are values copied over to the GPU each time you call MatSetValues? >>>>>> >>>>> >>>>> That function assembles the matrix on the GPU and then copies to the >>>>> CPU. The only time you do not want this copy is when >>>>> you are running in serial and never touch the matrix afterwards, so I >>>>> left it in. >>>>> >>>>> >>>>>> - Can we expect to see any speedup from using MatSetValuesBatch over >>>>>> MatSetValues, or is the batch version simply a utility function? This >>>>>> question goes for both CPU- and GPU-based matrices. >>>>>> >>>>> >>>>> CPU: no >>>>> >>>>> GPU: yes, I see about the memory bandwidth ratio >>>>> >>>>> >>>>> Hi, >>>> >>>> I have now integrated MatSetValuesBatch in our existing PETSc wrapper >>>> layer. I have tested matrix assembly with Poisson's equation on different >>>> meshes with elements of varying order. I have timed the single call to >>>> MatSetValuesBatch and compared that to the total time consumed by the >>>> repeated calls to MatSetValues in the old implementation. I have the >>>> following results: >>>> >>>> Poisson on 1000x1000 unit square, 1st order Lagrange elements: >>>> MatSetValuesBatch: 0.88576 s >>>> repeated calls to MatSetValues: 0.76654 s >>>> >>>> Poisson on 500x500 unit square, 2nd order Lagrange elements: >>>> MatSetValuesBatch: 0.9324 s >>>> repeated calls to MatSetValues: 0.81644 s >>>> >>>> Poisson on 300x300 unit square, 3rd order Lagrange elements: >>>> MatSetValuesBatch: 0.93988 s >>>> repeated calls to MatSetValues: 1.03884 s >>>> >>>> As you can see, the two methods take almost the same amount of time. >>>> What behavior and performance should we expect? Is there any way to >>>> optimize the performance of batched assembly? >>>> >>> >>> Almost certainly it is not dispatching to the CUDA version. The regular >>> version just calls MatSetValues() in a loop. Are you >>> using a SEQAIJCUSP matrix? >>> >> Yes. The same matrices yields a speedup of 4-6x when solving the system >> on the GPU. >> > > Please confirm that the correct routine by running wth -info and sending > the output. > > Please send the output of -log_summary so I can confirm the results. > > You can run KSP ex4 and reproduce my results where I see a 5.5x speedup on > the GTX285 > > I am not sure what to look for in those outputs. I have uploaded the output of running my assembly program with -info and -log_summary, and the output of running ex4 with -log_summary. See http://folk.uio.no/fredva/assembly_info.txt http://folk.uio.no/fredva/assembly_log_summary.txt http://folk.uio.no/fredva/ex4_log_summary.txt Trying this on a different machine now, I actually see some speedup. 3rd order Poisson on 300x300 assembles in 0.211 sec on GPU and 0.4232 sec on CPU. For 1st order and 1000x1000 mesh, I go from 0.31 sec to 0.205 sec. I have tried to increase the mesh size to see if the speedup increases, but I hit the bad_alloc error pretty quick. For a problem of that size, should I expect even more speedup? Please let me know if you need any more output from test runs on my machine. -- Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 1 07:00:16 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Dec 2011 07:00:16 -0600 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: On Thu, Dec 1, 2011 at 5:39 AM, Fredrik Heffer Valdmanis wrote: > > 2011/11/29 Matthew Knepley > >> On Tue, Nov 29, 2011 at 10:37 AM, Fredrik Heffer Valdmanis < >> fredva at ifi.uio.no> wrote: >> >>> 2011/11/29 Matthew Knepley >>> >>>> On Tue, Nov 29, 2011 at 2:38 AM, Fredrik Heffer Valdmanis < >>>> fredva at ifi.uio.no> wrote: >>>> >>>>> 2011/10/28 Matthew Knepley >>>>> >>>>>> On Fri, Oct 28, 2011 at 10:24 AM, Fredrik Heffer Valdmanis < >>>>>> fredva at ifi.uio.no> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am working on integrating the new GPU based vectors and matrices >>>>>>> into FEniCS. Now, I'm looking at the possibility for getting some speedup >>>>>>> during finite element assembly, specifically when inserting the local >>>>>>> element matrix into the global element matrix. In that regard, I have a few >>>>>>> questions I hope you can help me out with: >>>>>>> >>>>>>> - When calling MatSetValues with a MATSEQAIJCUSP matrix as >>>>>>> parameter, what exactly is it that happens? As far as I can see, >>>>>>> MatSetValues is not implemented for GPU based matrices, neither is >>>>>>> the mat->ops->setvalues set to point at any function for this Mat type. >>>>>>> >>>>>> >>>>>> Yes, MatSetValues always operates on the CPU side. It would not make >>>>>> sense to do individual operations on the GPU. >>>>>> >>>>>> I have written batched of assembly for element matrices that are all >>>>>> the same size: >>>>>> >>>>>> >>>>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBatch.html >>>>>> >>>>>> >>>>>>> - Is it such that matrices are assembled in their entirety on the >>>>>>> CPU, and then copied over to the GPU (after calling MatAssemblyBegin)? Or >>>>>>> are values copied over to the GPU each time you call MatSetValues? >>>>>>> >>>>>> >>>>>> That function assembles the matrix on the GPU and then copies to the >>>>>> CPU. The only time you do not want this copy is when >>>>>> you are running in serial and never touch the matrix afterwards, so I >>>>>> left it in. >>>>>> >>>>>> >>>>>>> - Can we expect to see any speedup from using MatSetValuesBatch over >>>>>>> MatSetValues, or is the batch version simply a utility function? This >>>>>>> question goes for both CPU- and GPU-based matrices. >>>>>>> >>>>>> >>>>>> CPU: no >>>>>> >>>>>> GPU: yes, I see about the memory bandwidth ratio >>>>>> >>>>>> >>>>>> Hi, >>>>> >>>>> I have now integrated MatSetValuesBatch in our existing PETSc wrapper >>>>> layer. I have tested matrix assembly with Poisson's equation on different >>>>> meshes with elements of varying order. I have timed the single call to >>>>> MatSetValuesBatch and compared that to the total time consumed by the >>>>> repeated calls to MatSetValues in the old implementation. I have the >>>>> following results: >>>>> >>>>> Poisson on 1000x1000 unit square, 1st order Lagrange elements: >>>>> MatSetValuesBatch: 0.88576 s >>>>> repeated calls to MatSetValues: 0.76654 s >>>>> >>>>> Poisson on 500x500 unit square, 2nd order Lagrange elements: >>>>> MatSetValuesBatch: 0.9324 s >>>>> repeated calls to MatSetValues: 0.81644 s >>>>> >>>>> Poisson on 300x300 unit square, 3rd order Lagrange elements: >>>>> MatSetValuesBatch: 0.93988 s >>>>> repeated calls to MatSetValues: 1.03884 s >>>>> >>>>> As you can see, the two methods take almost the same amount of time. >>>>> What behavior and performance should we expect? Is there any way to >>>>> optimize the performance of batched assembly? >>>>> >>>> >>>> Almost certainly it is not dispatching to the CUDA version. The regular >>>> version just calls MatSetValues() in a loop. Are you >>>> using a SEQAIJCUSP matrix? >>>> >>> Yes. The same matrices yields a speedup of 4-6x when solving the system >>> on the GPU. >>> >> >> Please confirm that the correct routine by running wth -info and sending >> the output. >> >> Please send the output of -log_summary so I can confirm the results. >> >> You can run KSP ex4 and reproduce my results where I see a 5.5x speedup >> on the GTX285 >> >> I am not sure what to look for in those outputs. I have uploaded the > output of running my assembly program with -info and -log_summary, and the > output of running ex4 with -log_summary. See > > http://folk.uio.no/fredva/assembly_info.txt > http://folk.uio.no/fredva/assembly_log_summary.txt > http://folk.uio.no/fredva/ex4_log_summary.txt > > Trying this on a different machine now, I actually see some speedup. 3rd > order Poisson on 300x300 assembles in 0.211 sec on GPU and 0.4232 sec on > CPU. For 1st order and 1000x1000 mesh, I go from 0.31 sec to 0.205 sec. > I have tried to increase the mesh size to see if the speedup increases, > but I hit the bad_alloc error pretty quick. > > For a problem of that size, should I expect even more speedup? Please let > me know if you need any more output from test runs on my machine. > Here are my results for nxn grids where n = range(150, 1350, 100). This is using a GTX 285. What card are you using? Matt > -- > Fredrik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: AssemblyResults.pdf Type: application/pdf Size: 63138 bytes Desc: not available URL: From knepley at gmail.com Thu Dec 1 07:05:20 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Dec 2011 07:05:20 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: Message-ID: You can get this effect using PCBJACOBI, setting the number of local blocks, and using -sub_pc_type lu -sub_ksp_type preonly. Matt On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > Hi,**** > > ** ** > > I?ve been using PETSc for a conventional full parallel implicit solver, > currently in MPIBAIJ format. However, I?d like to understand the > performance verse memory and speed trade-offs of just having a block > diagonal system. In this case I do not scatter to the off diagonal > blocks. Each block is 7x7 in size and fully dense. In my view this should > be a point implicit solution of the system. I am looking for the best way > to solve this in PETSc, clearly because of the lack of coupling between > neighbouring cells by only having a block diagonal it?s actually a 7x7 > dense linear solve, but to test my theory at present I?d prefer to just not > scatter to off diagonals, and use:**** > > ** ** > > MATBDIAG with my current ksp solver and take it from there. BDIAG seems > to have been removed as off 3.0:**** > > ** ** > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html**** > > ** ** > > Can anybody help me with this, your thoughts on the above and what I > should use to solve this reduced system would be appreciated??**** > > > Cheers,**** > > Andy**** > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 1 08:08:51 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 08:08:51 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: Message-ID: On Dec 1, 2011, at 7:05 AM, Matthew Knepley wrote: > You can get this effect using PCBJACOBI, setting the number of local blocks, and using -sub_pc_type lu -sub_ksp_type preonly. Do not do this. It will give bad performance since it explicitly pulls out each block with MatGetSubMatrices() and solves each small matrix as a separate PETSc matrix. Instead use the BAIJ matrix format with a block size of 7 and then use a PCPBJACOBI (note the P in the name for point block, each point block is that little 7 by 7 block that it solves efficiently using a tuned kernel) with -ksp_type preonly. This should be very fast and does not do any any neccessary scatters or communication. Barry > > Matt > > On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) wrote: > Hi, > > > > I?ve been using PETSc for a conventional full parallel implicit solver, currently in MPIBAIJ format. However, I?d like to understand the performance verse memory and speed trade-offs of just having a block diagonal system. In this case I do not scatter to the off diagonal blocks. Each block is 7x7 in size and fully dense. In my view this should be a point implicit solution of the system. I am looking for the best way to solve this in PETSc, clearly because of the lack of coupling between neighbouring cells by only having a block diagonal it?s actually a 7x7 dense linear solve, but to test my theory at present I?d prefer to just not scatter to off diagonals, and use: > > > > MATBDIAG with my current ksp solver and take it from there. BDIAG seems to have been removed as off 3.0: > > > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html > > > > Can anybody help me with this, your thoughts on the above and what I should use to solve this reduced system would be appreciated?? > > > Cheers, > > Andy > > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Andrew.Parker2 at baesystems.com Thu Dec 1 08:44:50 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Thu, 1 Dec 2011 14:44:50 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: Message-ID: I have tried to use the following, this was before I had both you're emails (Barry, Matthew) ;-) 1) First I set this up PetscInt size = 7; MatCreate(MPI_COMM_SELF,&LHS); MatSetSizes(LHS,size,size,size,size); MatSetType(LHS,MATSEQDENSE); //**************NOTE THIS LINE MatAssemblyBegin(LHS,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(LHS,MAT_FINAL_ASSEMBLY); VecCreate(PETSC_COMM_SELF,&RHS); VecSetSizes(RHS,PETSC_DECIDE, size); VecSetFromOptions(RHS); VecAssemblyBegin(RHS); VecAssemblyEnd(RHS); VecCreate(PETSC_COMM_SELF,&Updates); VecSetSizes(RHS,PETSC_DECIDE, size); VecSetFromOptions(Updates); VecAssemblyBegin(Updates); VecAssemblyEnd(Updates); KSPCreate(PETSC_COMM_SELF,&Solver); KSPSetOperators(Solver, LHS, LHS, SAME_NONZERO_PATTERN); KSPSetType(Solver, KSPPREONLY); //**************NOTE THIS LINE KSPGetPC(Solver, &prec); PCSetType(prec, PCLU); //**************NOTE THIS LINE KSPSetFromOptions(Solver); KSPSetUp(Solver); For each cell I then do: 2) I copy out the appropriate data from global structures into LHS and RHS 3) KSPSolve(Solver, RHS, Updates); 4) Stick the local Updates back into my global data structure Could I have your thoughts on whether the above is ok? In terms of efficiency and the correct thing to do? Would I be correct in assuming that Barry's method and mine are identical with the exception that I've got some copies going on to set up these small 7x7 and 1x7 matrices and vectors, but Barry's method does this inline by doing a preonly, lu solve DIRECT solve on each 7x7?? Many thanks, Andy -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: 01 December 2011 14:09 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Dec 1, 2011, at 7:05 AM, Matthew Knepley wrote: > You can get this effect using PCBJACOBI, setting the number of local blocks, and using -sub_pc_type lu -sub_ksp_type preonly. Do not do this. It will give bad performance since it explicitly pulls out each block with MatGetSubMatrices() and solves each small matrix as a separate PETSc matrix. Instead use the BAIJ matrix format with a block size of 7 and then use a PCPBJACOBI (note the P in the name for point block, each point block is that little 7 by 7 block that it solves efficiently using a tuned kernel) with -ksp_type preonly. This should be very fast and does not do any any neccessary scatters or communication. Barry > > Matt > > On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) wrote: > Hi, > > > > I've been using PETSc for a conventional full parallel implicit solver, currently in MPIBAIJ format. However, I'd like to understand the performance verse memory and speed trade-offs of just having a block diagonal system. In this case I do not scatter to the off diagonal blocks. Each block is 7x7 in size and fully dense. In my view this should be a point implicit solution of the system. I am looking for the best way to solve this in PETSc, clearly because of the lack of coupling between neighbouring cells by only having a block diagonal it's actually a 7x7 dense linear solve, but to test my theory at present I'd prefer to just not scatter to off diagonals, and use: > > > > MATBDIAG with my current ksp solver and take it from there. BDIAG seems to have been removed as off 3.0: > > > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html > > > > Can anybody help me with this, your thoughts on the above and what I should use to solve this reduced system would be appreciated?? > > > Cheers, > > Andy > > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Thu Dec 1 08:50:18 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Dec 2011 08:50:18 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: Message-ID: On Thu, Dec 1, 2011 at 8:44 AM, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > I have tried to use the following, this was before I had both you're > emails (Barry, Matthew) ;-) > > 1) First I set this up > > PetscInt size = 7; > MatCreate(MPI_COMM_SELF,&LHS); > MatSetSizes(LHS,size,size,size,size); > MatSetType(LHS,MATSEQDENSE); //**************NOTE THIS LINE > MatAssemblyBegin(LHS,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(LHS,MAT_FINAL_ASSEMBLY); > > VecCreate(PETSC_COMM_SELF,&RHS); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(RHS); > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > VecCreate(PETSC_COMM_SELF,&Updates); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(Updates); > VecAssemblyBegin(Updates); > VecAssemblyEnd(Updates); > > KSPCreate(PETSC_COMM_SELF,&Solver); > KSPSetOperators(Solver, LHS, LHS, SAME_NONZERO_PATTERN); > KSPSetType(Solver, KSPPREONLY); //**************NOTE THIS LINE > KSPGetPC(Solver, &prec); > PCSetType(prec, PCLU); //**************NOTE THIS LINE > KSPSetFromOptions(Solver); > KSPSetUp(Solver); > > For each cell I then do: > > 2) I copy out the appropriate data from global structures into LHS and > RHS > > 3) KSPSolve(Solver, RHS, Updates); > > 4) Stick the local Updates back into my global data structure > > Could I have your thoughts on whether the above is ok? In terms of > efficiency and the correct thing to do? Would I be correct in assuming > that Barry's method and mine are identical with the exception that I've > got some copies going on to set up these small 7x7 and 1x7 matrices and > vectors, but Barry's method does this inline by doing a preonly, lu > solve DIRECT solve on each 7x7?? > With Barry's method you can do this from the command line so you can easily compare with other solvers without changing your code. Matt > Many thanks, > Andy > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: 01 December 2011 14:09 > To: PETSc users list > Subject: Re: [petsc-users] MATBDIAG in > 3.1 > > > *** WARNING *** > > This message has originated outside your organisation, > either from an external partner or the Global Internet. > Keep this in mind if you answer this message. > > > On Dec 1, 2011, at 7:05 AM, Matthew Knepley wrote: > > > You can get this effect using PCBJACOBI, setting the number of local > blocks, and using -sub_pc_type lu -sub_ksp_type preonly. > > Do not do this. It will give bad performance since it explicitly > pulls out each block with MatGetSubMatrices() and solves each small > matrix as a separate PETSc matrix. > > Instead use the BAIJ matrix format with a block size of 7 and then > use a PCPBJACOBI (note the P in the name for point block, each point > block is that little 7 by 7 block that it solves efficiently using a > tuned kernel) with -ksp_type preonly. This should be very fast and does > not do any any neccessary scatters or communication. > > Barry > > > > > Matt > > > > On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) > wrote: > > Hi, > > > > > > > > I've been using PETSc for a conventional full parallel implicit > solver, currently in MPIBAIJ format. However, I'd like to understand > the performance verse memory and speed trade-offs of just having a block > diagonal system. In this case I do not scatter to the off diagonal > blocks. Each block is 7x7 in size and fully dense. In my view this > should be a point implicit solution of the system. I am looking for the > best way to solve this in PETSc, clearly because of the lack of coupling > between neighbouring cells by only having a block diagonal it's actually > a 7x7 dense linear solve, but to test my theory at present I'd prefer to > just not scatter to off diagonals, and use: > > > > > > > > MATBDIAG with my current ksp solver and take it from there. BDIAG > seems to have been removed as off 3.0: > > > > > > > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html > > > > > > > > Can anybody help me with this, your thoughts on the above and what I > should use to solve this reduced system would be appreciated?? > > > > > > Cheers, > > > > Andy > > > > > > ******************************************************************** > > This email and any attachments are confidential to the intended > > recipient and may also be privileged. If you are not the intended > > recipient please delete it from your system and notify the sender. > > You should not copy it or use it for any purpose nor disclose or > > distribute its contents to any other person. > > ******************************************************************** > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Thu Dec 1 11:06:29 2011 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Thu, 1 Dec 2011 11:06:29 -0600 Subject: [petsc-users] A postdoctoral position at Argonne involving PETSc Message-ID: Dear All, For your information: there is a postdoctoral position opening at Argonne that might be of interest to you or to one of your colleagues: http://www.anl.gov/jobsearch/detail.jsp?userreqid=318546+MSD&lsBrowse=POSTDOC This projects involves development of PETSc-based application codes, NOT PETSc development. Specifically, the types of problems involved are elasticity with highly variable elastic properties. A working knowledge of PETSc is highly desired. Please, feel free to forward this to others that might be interested. Regards, Dmitry ------------------------------------------------------------- Dmitry Karpeev, Ph.D. Assistant Computational Mathematician Mathematics and Computer Science Argonne National Laboratory Argonne, Illinois, USA and Fellow Computation Institute University of Chicago 5735 S. Ellis Avenue Chicago, IL 60637 ----------------------- Phone: 630-252-1229 Fax: 630-252-5986 -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Thu Dec 1 12:14:55 2011 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Thu, 1 Dec 2011 12:14:55 -0600 Subject: [petsc-users] A postdoctoral position at Argonne involving PETSc In-Reply-To: References: Message-ID: Sorry for (potential) multiple postings. Just trying the make sure the word gets out: Dear All, For your information: there is a postdoctoral position opening at Argonne that might be of interest to you or to one of your colleagues: http://www.anl.gov/jobsearch/detail.jsp?userreqid=318546+MSD&lsBrowse=POSTDOC This projects involves development of PETSc-based application codes, NOT PETSc development. Specifically, the types of problems involved are elasticity with highly variable elastic properties. A working knowledge of PETSc is highly desired. Please, feel free to forward this to others that might be interested. Regards, Dmitry ------------------------------------------------------------- Dmitry Karpeev, Ph.D. Assistant Computational Mathematician Mathematics and Computer Science Argonne National Laboratory Argonne, Illinois, USA and Fellow Computation Institute University of Chicago 5735 S. Ellis Avenue Chicago, IL 60637 ----------------------- Phone: 630-252-1229 Fax: 630-252-5986 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 1 13:57:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 13:57:01 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: Message-ID: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> My approach will be a million times faster. Barry On Dec 1, 2011, at 8:50 AM, Matthew Knepley wrote: > On Thu, Dec 1, 2011 at 8:44 AM, Parker, Andrew (UK Filton) wrote: > I have tried to use the following, this was before I had both you're > emails (Barry, Matthew) ;-) > > 1) First I set this up > > PetscInt size = 7; > MatCreate(MPI_COMM_SELF,&LHS); > MatSetSizes(LHS,size,size,size,size); > MatSetType(LHS,MATSEQDENSE); //**************NOTE THIS LINE > MatAssemblyBegin(LHS,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(LHS,MAT_FINAL_ASSEMBLY); > > VecCreate(PETSC_COMM_SELF,&RHS); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(RHS); > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > VecCreate(PETSC_COMM_SELF,&Updates); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(Updates); > VecAssemblyBegin(Updates); > VecAssemblyEnd(Updates); > > KSPCreate(PETSC_COMM_SELF,&Solver); > KSPSetOperators(Solver, LHS, LHS, SAME_NONZERO_PATTERN); > KSPSetType(Solver, KSPPREONLY); //**************NOTE THIS LINE > KSPGetPC(Solver, &prec); > PCSetType(prec, PCLU); //**************NOTE THIS LINE > KSPSetFromOptions(Solver); > KSPSetUp(Solver); > > For each cell I then do: > > 2) I copy out the appropriate data from global structures into LHS and > RHS > > 3) KSPSolve(Solver, RHS, Updates); > > 4) Stick the local Updates back into my global data structure > > Could I have your thoughts on whether the above is ok? In terms of > efficiency and the correct thing to do? Would I be correct in assuming > that Barry's method and mine are identical with the exception that I've > got some copies going on to set up these small 7x7 and 1x7 matrices and > vectors, but Barry's method does this inline by doing a preonly, lu > solve DIRECT solve on each 7x7?? > > With Barry's method you can do this from the command line so you can easily compare > with other solvers without changing your code. > > Matt > > Many thanks, > Andy > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: 01 December 2011 14:09 > To: PETSc users list > Subject: Re: [petsc-users] MATBDIAG in > 3.1 > > > *** WARNING *** > > This message has originated outside your organisation, > either from an external partner or the Global Internet. > Keep this in mind if you answer this message. > > > On Dec 1, 2011, at 7:05 AM, Matthew Knepley wrote: > > > You can get this effect using PCBJACOBI, setting the number of local > blocks, and using -sub_pc_type lu -sub_ksp_type preonly. > > Do not do this. It will give bad performance since it explicitly > pulls out each block with MatGetSubMatrices() and solves each small > matrix as a separate PETSc matrix. > > Instead use the BAIJ matrix format with a block size of 7 and then > use a PCPBJACOBI (note the P in the name for point block, each point > block is that little 7 by 7 block that it solves efficiently using a > tuned kernel) with -ksp_type preonly. This should be very fast and does > not do any any neccessary scatters or communication. > > Barry > > > > > Matt > > > > On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) > wrote: > > Hi, > > > > > > > > I've been using PETSc for a conventional full parallel implicit > solver, currently in MPIBAIJ format. However, I'd like to understand > the performance verse memory and speed trade-offs of just having a block > diagonal system. In this case I do not scatter to the off diagonal > blocks. Each block is 7x7 in size and fully dense. In my view this > should be a point implicit solution of the system. I am looking for the > best way to solve this in PETSc, clearly because of the lack of coupling > between neighbouring cells by only having a block diagonal it's actually > a 7x7 dense linear solve, but to test my theory at present I'd prefer to > just not scatter to off diagonals, and use: > > > > > > > > MATBDIAG with my current ksp solver and take it from there. BDIAG > seems to have been removed as off 3.0: > > > > > > > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html > > > > > > > > Can anybody help me with this, your thoughts on the above and what I > should use to solve this reduced system would be appreciated?? > > > > > > Cheers, > > > > Andy > > > > > > ******************************************************************** > > This email and any attachments are confidential to the intended > > recipient and may also be privileged. If you are not the intended > > recipient please delete it from your system and notify the sender. > > You should not copy it or use it for any purpose nor disclose or > > distribute its contents to any other person. > > ******************************************************************** > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From mirzadeh at gmail.com Thu Dec 1 16:48:28 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 1 Dec 2011 14:48:28 -0800 Subject: [petsc-users] VecGet/RestoreArray and related functions Message-ID: Hi guys, I am having a conceptual problem trying to design my vector wrappers for my application. The problem is some of vector functionality in PETSc require extra function calls which makes it hard to design a good wrapper. One example that I encountered was when trying to write the destructor for my vector object. You see, for example, the following code is not good enough Vector::~Vector(){ VecDestroy(v_local); } This, in itself wont work since all calls to VecDestroy() should be made before PetscFinalize() and this wont happen if you define a Vector in your main function since the destructor is called upon exit ... So, I had to fix this by actually having a destructor function memer that I could call explicitly before PetscFinalize() or create the Vector object on the heap so that I can manually invoke the destructor via delete operator! Another example I'm struggling with is the following: I am trying to write accessors so that I can do the following: Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of global length 10 using my own wrappers for (int i=0; i From knepley at gmail.com Thu Dec 1 16:54:35 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Dec 2011 16:54:35 -0600 Subject: [petsc-users] VecGet/RestoreArray and related functions In-Reply-To: References: Message-ID: On Thu, Dec 1, 2011 at 4:48 PM, Mohammad Mirzadeh wrote: > Hi guys, > > I am having a conceptual problem trying to design my vector wrappers for > my application. The problem is some of vector functionality in PETSc > require extra function calls which makes it hard to design a good wrapper. > One example that I encountered was when trying to write the destructor for > my vector object. You see, for example, the following code is not good > enough > 1) We asked before, what could you possibly gain by writing a wrapper? 2) If you insist on doing this, register PetscFinalize() with atexit(). Matt > Vector::~Vector(){ > VecDestroy(v_local); > } > > This, in itself wont work since all calls to VecDestroy() should be made > before PetscFinalize() and this wont happen if you define a Vector in your > main function since the destructor is called upon exit ... So, I had to fix > this by actually having a destructor function memer that I could call > explicitly before PetscFinalize() or create the Vector object on the heap > so that I can manually invoke the destructor via delete operator! > > Another example I'm struggling with is the following: I am trying to write > accessors so that I can do the following: > > Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of > global length 10 using my own wrappers > for (int i=0; i a(i) = i; > } > > For this to work, I am using the VecGetArray() and VecRestoreArray() > functions. The problem is you cannot easily use them inside the accessor > which basically looks something like: > > inline double& Vector::operator()(int i){ > return ptr_to_dataArray[i]; > } > > since you need to call to VecRestoreArray after the assignment which means > after return!. The simple fix to this is to call the VecGetArray() and > VecRestoreArray() in the main function where you are putting in the values > but this does not look very neat to me ... so I ended up putting the > VecGetArray() inside the constructor and VecRestoreArray() inside the > destructor to automate the process. > > Now I have the following two questions. 1) Is this safe? As far as I could > understand from the manual, VecGetArray() "returns" a pointer to internal > data stored on this processor. Then I guess VecRestoreArray() is there to > somehow nullify the pointer to prevent accidental use of pointer? Since all > pointers all private members of my class I assume this is safe then unless > these two functions are actually doing some extra work? > > 2) Is there any better way of writing these wrappers and get around the > problem of having to call "supplemental" functions safely? > > Sorry if my lack of knowledge is causing all these confusion. > > Thanks, > Mohammad > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Dec 1 17:02:19 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 1 Dec 2011 15:02:19 -0800 Subject: [petsc-users] VecGet/RestoreArray and related functions In-Reply-To: References: Message-ID: Well the reason is merely to keep the "look" and "feel" of the code similar to the sequential library that has been developed over the years in the group such that other folks can jump right into it. On Thu, Dec 1, 2011 at 2:54 PM, Matthew Knepley wrote: > On Thu, Dec 1, 2011 at 4:48 PM, Mohammad Mirzadeh wrote: > >> Hi guys, >> >> I am having a conceptual problem trying to design my vector wrappers for >> my application. The problem is some of vector functionality in PETSc >> require extra function calls which makes it hard to design a good wrapper. >> One example that I encountered was when trying to write the destructor for >> my vector object. You see, for example, the following code is not good >> enough >> > > 1) We asked before, what could you possibly gain by writing a wrapper? > > 2) If you insist on doing this, register PetscFinalize() with atexit(). > > Matt > > >> Vector::~Vector(){ >> VecDestroy(v_local); >> } >> >> This, in itself wont work since all calls to VecDestroy() should be made >> before PetscFinalize() and this wont happen if you define a Vector in your >> main function since the destructor is called upon exit ... So, I had to fix >> this by actually having a destructor function memer that I could call >> explicitly before PetscFinalize() or create the Vector object on the heap >> so that I can manually invoke the destructor via delete operator! >> >> Another example I'm struggling with is the following: I am trying to >> write accessors so that I can do the following: >> >> Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of >> global length 10 using my own wrappers >> for (int i=0; i> a(i) = i; >> } >> >> For this to work, I am using the VecGetArray() and VecRestoreArray() >> functions. The problem is you cannot easily use them inside the accessor >> which basically looks something like: >> >> inline double& Vector::operator()(int i){ >> return ptr_to_dataArray[i]; >> } >> >> since you need to call to VecRestoreArray after the assignment which >> means after return!. The simple fix to this is to call the VecGetArray() >> and VecRestoreArray() in the main function where you are putting in the >> values but this does not look very neat to me ... so I ended up putting the >> VecGetArray() inside the constructor and VecRestoreArray() inside the >> destructor to automate the process. >> >> Now I have the following two questions. 1) Is this safe? As far as I >> could understand from the manual, VecGetArray() "returns" a pointer to >> internal data stored on this processor. Then I guess VecRestoreArray() is >> there to somehow nullify the pointer to prevent accidental use of pointer? >> Since all pointers all private members of my class I assume this is safe >> then unless these two functions are actually doing some extra work? >> >> 2) Is there any better way of writing these wrappers and get around the >> problem of having to call "supplemental" functions safely? >> >> Sorry if my lack of knowledge is causing all these confusion. >> >> Thanks, >> Mohammad >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetan.jhurani at gmail.com Thu Dec 1 17:06:23 2011 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Thu, 1 Dec 2011 18:06:23 -0500 Subject: [petsc-users] VecGet/RestoreArray and related functions In-Reply-To: References: Message-ID: <4ed80867.491be00a.37cd.3306@mx.google.com> You may want to write a class, say petsc_init_uninit, whose constructor initializes petsc and whose destructor finalizes it. Then declare an object of class petsc_init_uninit in main. This will solve the first issue. http://www.parashift.com/c++-faq-lite/dtors.html#faq-11.2 Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Mohammad Mirzadeh Sent: Thursday, December 01, 2011 5:48 PM To: PETSc users list Subject: [petsc-users] VecGet/RestoreArray and related functions Hi guys, I am having a conceptual problem trying to design my vector wrappers for my application. The problem is some of vector functionality in PETSc require extra function calls which makes it hard to design a good wrapper. One example that I encountered was when trying to write the destructor for my vector object. You see, for example, the following code is not good enough Vector::~Vector(){ VecDestroy(v_local); } This, in itself wont work since all calls to VecDestroy() should be made before PetscFinalize() and this wont happen if you define a Vector in your main function since the destructor is called upon exit ... So, I had to fix this by actually having a destructor function memer that I could call explicitly before PetscFinalize() or create the Vector object on the heap so that I can manually invoke the destructor via delete operator! Another example I'm struggling with is the following: I am trying to write accessors so that I can do the following: Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of global length 10 using my own wrappers for (int i=0; i From bsmith at mcs.anl.gov Thu Dec 1 17:10:03 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 17:10:03 -0600 Subject: [petsc-users] VecGet/RestoreArray and related functions In-Reply-To: References: Message-ID: <72720551-0CA6-4597-A429-B513FB87983E@mcs.anl.gov> On Dec 1, 2011, at 4:48 PM, Mohammad Mirzadeh wrote: > Hi guys, > > I am having a conceptual problem trying to design my vector wrappers for my application. The problem is some of vector functionality in PETSc require extra function calls which makes it hard to design a good wrapper. One example that I encountered was when trying to write the destructor for my vector object. You see, for example, the following code is not good enough > > Vector::~Vector(){ > VecDestroy(v_local); > } > > This, in itself wont work since all calls to VecDestroy() should be made before PetscFinalize() and this wont happen if you define a Vector in your main function since the destructor is called upon exit ... So, I had to fix this by actually having a destructor function memer that I could call explicitly before PetscFinalize() or create the Vector object on the heap so that I can manually invoke the destructor via delete operator! > > Another example I'm struggling with is the following: I am trying to write accessors so that I can do the following: > > Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of global length 10 using my own wrappers > for (int i=0; i a(i) = i; > } > > For this to work, I am using the VecGetArray() and VecRestoreArray() functions. The problem is you cannot easily use them inside the accessor which basically looks something like: > > inline double& Vector::operator()(int i){ > return ptr_to_dataArray[i]; > } You are really missing the purpose of the entire get/restore (and assembly) paradigm in PETSc. The idea is that the user cannot and should not be able to willy-nilly access/change data that is inside a PETSc object. They can/should only do it in a well controlled way (time and place) and that is between the get/restore calls. Yes you can "cheat" by "secretly keeping access to the underlying pointer the entire time but that is dangerous and could result in unexplained crashes or wrong answers when someone changes things at a point in the code they shouldn't. You could overload the square brackets in a[i] to get the array, do what is needed and then restore the array all in the one overload call. That is the only save way to do it. Barry > > since you need to call to VecRestoreArray after the assignment which means after return!. The simple fix to this is to call the VecGetArray() and VecRestoreArray() in the main function where you are putting in the values but this does not look very neat to me ... so I ended up putting the VecGetArray() inside the constructor and VecRestoreArray() inside the destructor to automate the process. > > Now I have the following two questions. 1) Is this safe? As far as I could understand from the manual, VecGetArray() "returns" a pointer to internal data stored on this processor. Then I guess VecRestoreArray() is there to somehow nullify the pointer to prevent accidental use of pointer? Since all pointers all private members of my class I assume this is safe then unless these two functions are actually doing some extra work? > > 2) Is there any better way of writing these wrappers and get around the problem of having to call "supplemental" functions safely? > > Sorry if my lack of knowledge is causing all these confusion. > > Thanks, > Mohammad > > From mirzadeh at gmail.com Thu Dec 1 17:22:29 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 1 Dec 2011 15:22:29 -0800 Subject: [petsc-users] VecGet/RestoreArray and related functions In-Reply-To: <72720551-0CA6-4597-A429-B513FB87983E@mcs.anl.gov> References: <72720551-0CA6-4597-A429-B513FB87983E@mcs.anl.gov> Message-ID: Thank you Barry. I agree that what I did disregards the safety measures in PETSc but I figured that's the only way I could make it look more like normal std::vector. Chetan: That's an interesting trick! Thanks. Thanks again guys On Thu, Dec 1, 2011 at 3:10 PM, Barry Smith wrote: > > On Dec 1, 2011, at 4:48 PM, Mohammad Mirzadeh wrote: > > > Hi guys, > > > > I am having a conceptual problem trying to design my vector wrappers for > my application. The problem is some of vector functionality in PETSc > require extra function calls which makes it hard to design a good wrapper. > One example that I encountered was when trying to write the destructor for > my vector object. You see, for example, the following code is not good > enough > > > > Vector::~Vector(){ > > VecDestroy(v_local); > > } > > > > This, in itself wont work since all calls to VecDestroy() should be made > before PetscFinalize() and this wont happen if you define a Vector in your > main function since the destructor is called upon exit ... So, I had to fix > this by actually having a destructor function memer that I could call > explicitly before PetscFinalize() or create the Vector object on the heap > so that I can manually invoke the destructor via delete operator! > > > > Another example I'm struggling with is the following: I am trying to > write accessors so that I can do the following: > > > > Vector a (PETSC_COMM_WORLD, PETSC_DECIDE,10); // make a petsc vector of > global length 10 using my own wrappers > > for (int i=0; i > a(i) = i; > > } > > > > For this to work, I am using the VecGetArray() and VecRestoreArray() > functions. The problem is you cannot easily use them inside the accessor > which basically looks something like: > > > > inline double& Vector::operator()(int i){ > > return ptr_to_dataArray[i]; > > } > > You are really missing the purpose of the entire get/restore (and > assembly) paradigm in PETSc. The idea is that the user cannot and should > not be able to willy-nilly access/change data that is inside a PETSc > object. They can/should only do it in a well controlled way (time and > place) and that is between the get/restore calls. Yes you can "cheat" by > "secretly keeping access to the underlying pointer the entire time but that > is dangerous and could result in unexplained crashes or wrong answers when > someone changes things at a point in the code they shouldn't. > > You could overload the square brackets in a[i] to get the array, do > what is needed and then restore the array all in the one overload call. > That is the only save way to do it. > > Barry > > > > > > > since you need to call to VecRestoreArray after the assignment which > means after return!. The simple fix to this is to call the VecGetArray() > and VecRestoreArray() in the main function where you are putting in the > values but this does not look very neat to me ... so I ended up putting the > VecGetArray() inside the constructor and VecRestoreArray() inside the > destructor to automate the process. > > > > Now I have the following two questions. 1) Is this safe? As far as I > could understand from the manual, VecGetArray() "returns" a pointer to > internal data stored on this processor. Then I guess VecRestoreArray() is > there to somehow nullify the pointer to prevent accidental use of pointer? > Since all pointers all private members of my class I assume this is safe > then unless these two functions are actually doing some extra work? > > > > 2) Is there any better way of writing these wrappers and get around the > problem of having to call "supplemental" functions safely? > > > > Sorry if my lack of knowledge is causing all these confusion. > > > > Thanks, > > Mohammad > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Thu Dec 1 18:44:21 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Thu, 1 Dec 2011 19:44:21 -0500 Subject: [petsc-users] zeros in sparse aij Message-ID: Hello everyone, I have a question about whether zeros will change the sparse pattern. Suppose I am generating two sparse matrices A and B. A is generated with exactly number of nonzeros, say 10 nnz per row. When I generate matrix B, I specify 12 nonzeros, but two of them are given zero values by MatSetValue. Will A and B have same sparsity patterns and performance ? In other words, will PETSc simply ignore these zeros entries? I output these A and B in matlab, and it seems that these zeros entries are ignored. I just want to double check on this. Is the option MAT_IGNORE_ZERO_ENTRIES on by default? Thanks. Best, Xiangdong From bsmith at mcs.anl.gov Thu Dec 1 19:00:37 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 19:00:37 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: Message-ID: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: > Hello everyone, > > I have a question about whether zeros will change the sparse pattern. > Suppose I am generating two sparse matrices A and B. A is generated > with exactly number of nonzeros, say 10 nnz per row. When I generate > matrix B, I specify 12 nonzeros, but two of them are given zero values > by MatSetValue. Will A and B have same sparsity patterns and > performance ? In other words, will PETSc simply ignore these zeros > entries? I output these A and B in matlab, and it seems that these > zeros entries are ignored. I just want to double check on this. > > Is the option MAT_IGNORE_ZERO_ENTRIES on by default? No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. Barry > Thanks. > > Best, > Xiangdong From mmnasr at gmail.com Thu Dec 1 20:42:08 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Thu, 1 Dec 2011 18:42:08 -0800 Subject: [petsc-users] zeros in sparse aij In-Reply-To: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> Message-ID: I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. Best, Mohamad On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: > > On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: > > > Hello everyone, > > > > I have a question about whether zeros will change the sparse pattern. > > Suppose I am generating two sparse matrices A and B. A is generated > > with exactly number of nonzeros, say 10 nnz per row. When I generate > > matrix B, I specify 12 nonzeros, but two of them are given zero values > > by MatSetValue. Will A and B have same sparsity patterns and > > performance ? In other words, will PETSc simply ignore these zeros > > entries? I output these A and B in matlab, and it seems that these > > zeros entries are ignored. I just want to double check on this. > > > > Is the option MAT_IGNORE_ZERO_ENTRIES on by default? > > No, by default PETSc will insert the zero values thus increasing the > nonzero pattern. If you set that option to the matrix then inserting values > it will not introduce an increase in the pattern. > > > BTW: if you then load the matrix via a binary file to MATLAB it may be > that MATLAB silently removes those locations so you don't see them. > Barry > > > Thanks. > > > > Best, > > Xiangdong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Thu Dec 1 20:46:29 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Thu, 1 Dec 2011 21:46:29 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> Message-ID: Thanks. If this is the case, will the option MAT_IGNORE_ZERO_ENTRIES be helpful? Xiangdong On Thu, Dec 1, 2011 at 8:00 PM, Barry Smith wrote: > > On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: > >> Hello everyone, >> >> I have a question about whether zeros will change the sparse pattern. >> Suppose I am generating two sparse matrices A and B. A is generated >> with exactly number of nonzeros, say 10 nnz per row. When I generate >> matrix B, I specify 12 nonzeros, but two of them are given zero values >> by MatSetValue. Will A and B have same sparsity patterns and >> performance ? In other words, will PETSc simply ignore these zeros >> entries? ?I output these A and B in matlab, and it seems that these >> zeros entries are ignored. I just want to double check on this. >> >> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? > > ? No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. > > > ? ?BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. > ? Barry > >> Thanks. >> >> Best, >> Xiangdong > > From bsmith at mcs.anl.gov Thu Dec 1 20:59:00 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 20:59:00 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> Message-ID: On Dec 1, 2011, at 8:46 PM, Xiangdong Liang wrote: > Thanks. If this is the case, will the option MAT_IGNORE_ZERO_ENTRIES be helpful? What do you mean by "be helpful"? Sometimes you do want those "zero" locations represented and sometimes you don't, so sometimes you would use the option and sometimes not. For example, if you are going to refill the matrix a number of times with the union of the nonzero patterns then you don't want it to ignore the zero locations the first time. Otherwise the second time through when you put a true nonzero at that location it will be incredibly expensive. Barry > > Xiangdong > > On Thu, Dec 1, 2011 at 8:00 PM, Barry Smith wrote: >> >> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >> >>> Hello everyone, >>> >>> I have a question about whether zeros will change the sparse pattern. >>> Suppose I am generating two sparse matrices A and B. A is generated >>> with exactly number of nonzeros, say 10 nnz per row. When I generate >>> matrix B, I specify 12 nonzeros, but two of them are given zero values >>> by MatSetValue. Will A and B have same sparsity patterns and >>> performance ? In other words, will PETSc simply ignore these zeros >>> entries? I output these A and B in matlab, and it seems that these >>> zeros entries are ignored. I just want to double check on this. >>> >>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >> >> No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >> >> >> BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >> Barry >> >>> Thanks. >>> >>> Best, >>> Xiangdong >> >> From bsmith at mcs.anl.gov Thu Dec 1 21:01:57 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Dec 2011 21:01:57 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> Message-ID: <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> The checks in the code are lines like if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; hence changing the source code to ignore small values is not difficult. But I think this is a terribly cheesy hack and would hope people would not do this. Barry On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: > I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? > I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. > > Best, > Mohamad > > > > > > On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: > > On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: > > > Hello everyone, > > > > I have a question about whether zeros will change the sparse pattern. > > Suppose I am generating two sparse matrices A and B. A is generated > > with exactly number of nonzeros, say 10 nnz per row. When I generate > > matrix B, I specify 12 nonzeros, but two of them are given zero values > > by MatSetValue. Will A and B have same sparsity patterns and > > performance ? In other words, will PETSc simply ignore these zeros > > entries? I output these A and B in matlab, and it seems that these > > zeros entries are ignored. I just want to double check on this. > > > > Is the option MAT_IGNORE_ZERO_ENTRIES on by default? > > No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. > > > BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. > Barry > > > Thanks. > > > > Best, > > Xiangdong > > From hzhang at mcs.anl.gov Thu Dec 1 22:20:43 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 1 Dec 2011 22:20:43 -0600 Subject: [petsc-users] MatMumpsSetIcntl from Fortran In-Reply-To: <4ED731C3.9010505@gfz-potsdam.de> References: <4ED731C3.9010505@gfz-potsdam.de> Message-ID: Alexander Using blocks below and petsc-dev/src/ksp/ksp/examples/tutorials/ex2f.F, I wrote a Fortran test petsc-dev/src/ksp/ksp/examples/tutorials/ex57f.F (attached). I noticed some sloppiness in our interface that fails to take the procedural options, but I did not see crash as you experienced. I updated petsc-mumps interface in petsc-dev. Can you switch to petsc-dev which interfaces with MUMPS_4.10.0 and see if it still crash in Fortran? See http://www.mcs.anl.gov/petsc/developers/index.html on how to get petsc-dev. Let us know if you still get problems, Hong > > This should be enough to reproduce the problem: > > ?call KSPCreate(PETSC_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) > ?call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr);CHKERRQ(ierr) > ?call KSPSetType(ksp,KSPPREONLY,ierr);CHKERRQ(ierr) > ?call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) > ?call PCSetType(pc,PCCHOLESKY,ierr);CHKERRQ(ierr) > ?call PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS,ierr);CHKERRQ(ierr) > ?call PCFactorSetUpMatSolverPackage(pc,ierr);CHKERRQ(ierr) > ?call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) > ?call KSPSetFromOptions(ksp,ierr);CHKERRQ(ierr) > ?icntl = 2; ival = 0; > ?call MatMumpsSetIcntl(F,icntl,ival,ierr); > > Regards, > Alexander > > > On 30.11.2011 23:43, Hong Zhang wrote: >>> >>> Sorry if I wasn't clear. ?"c" example works fine, I know, what I meant is >>> that if you try to implement lines 150-171 from it on FORTRAN you will >>> see >>> the problem I reported. >>> If you need particularly my FORTRAN code I can send it tomorrow. >> >> This would save my time :-) >> Appreciate. >> >> Hong >>> >>> Regards, >>> Alexander >>> >>> >>> ----- Reply message ----- >>> From: "Hong Zhang" >>> To: "PETSc users list" >>> Subject: [petsc-users] MatMumpsSetIcntl from Fortran >>> Date: Wed, Nov 30, 2011 9:37 pm >>> >>> >>> Alexander : >>> >>>> I just rewrote code concerning mumps from this example (lines 150-170): >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c.html >>> >>> Where is your Fortran code? ex52.c works fine. >>> Hong >>> >>>> On 30.11.2011 17:40, Hong Zhang wrote: >>>>> >>>>> Alexander: >>>>> >>>>>> Has anybody tried to use MatMumpsSetIcntl from Fortran? >>>>> >>>>> We are not aware of it. >>>>> >>>>>> Because when I try to call it I fall into infinite recursion in >>>>>> function: >>>>> >>>>> Can you give me a short Fortran code that repeats this error for >>>>> investigating? >>>>> Meanwhile, you can use runtime option '-mat_mumps_icntl_xxx<>' to get >>>>> your code run. >>>>> >>>>> Hong >>>>>> >>>>>> ?PetscErrorCode MatMumpsSetIcntl(Mat F,PetscInt icntl,PetscInt ival) >>>>>> ?{ >>>>>> ? PetscErrorCode ierr; >>>>>> >>>>>> ? PetscFunctionBegin; >>>>>> ? PetscValidLogicalCollectiveInt(F,icntl,2); >>>>>> ? PetscValidLogicalCollectiveInt(F,ival,3); >>>>>> ? ierr = >>>>>> >>>>>> >>>>>> >>>>>> PetscTryMethod(F,"MatMumpsSetIcntl_C",(Mat,PetscInt,PetscInt),(F,icntl,ival));CHKERRQ(ierr); >>>>>> ? PetscFunctionReturn(0); >>>>>> ?} >>>>>> >>>>>> At the moment when program crashes call stack looks like: >>>>>> >>>>>> ? ? __libc_memalign, >>>>>> FP=7fff342ca010 >>>>>> PetscMallocAlign, >>>>>> ?FP=7fff342ca080 >>>>>> PetscTrMallocDefault, >>>>>> ?FP=7fff342ca180 >>>>>> PetscStrallocpy, >>>>>> FP=7fff342ca230 >>>>>> PetscFListGetPathAndFunction, >>>>>> ?FP=7fff342cb2e0 >>>>>> PetscFListFind, >>>>>> ?FP=7fff342cb520 >>>>>> PetscObjectQueryFunction_Petsc, >>>>>> ?FP=7fff342cb590 >>>>>> PetscObjectQueryFunction, >>>>>> ?FP=7fff342cb620 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb720 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb820 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb920 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cba20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbb20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbc20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbd20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbe20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbf20 >>>>>> ... (Hundreds of MatMumpsSetIcntl) ... >>>>>> >>>>>> What can I do about that? >>>>>> >>>>>> Regards, >>>>>> Alexander >>>> >>>> > -------------- next part -------------- A non-text attachment was scrubbed... Name: ex57f.F Type: application/octet-stream Size: 13471 bytes Desc: not available URL: From frtr at risoe.dtu.dk Fri Dec 2 03:32:17 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Fri, 2 Dec 2011 10:32:17 +0100 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Wednesday, November 30, 2011 5:05 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Wed, Nov 30, 2011 at 9:59 AM, Treue, Frederik > wrote: Hi everyone, Caveat: I have just started using petsc, so the answer to my question may very well be fairly trivial. See SNES ex5 for the right way to interact with the DMDA. We will preallocate the matrix for you and allow you to set values using a stencil. Matt OK, but that example seems to assume that you wish to connect only one matrix (the Jacobian) to a DA - I wish to specify many and I think I found this done in ksp ex39, is that example doing anything deprecated or will that work for me, e.g. with the various basic mat routines (matmult, matAXPY etc.) in a multiprocessor setup? I'm trying to run the following bits of code: DMDACreate2d(PETSC_COMM_WORLD, DMDA_BOUNDARY_GHOSTED, DMDA_BOUNDARY_GHOSTED, DMDA_STENCIL_BOX,10,10,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,&da); [snip] MatCreate(PETSC_COMM_WORLD,&((*FD).ddx)); MatSetSizes((*FD).ddx,PETSC_DECIDE,PETSC_DECIDE,100,100); MatSetFromOptions((*FD).ddx); for (i=0;i<10;i++) { col[0]=i*10;col[1]=i*10+1; row[0]=i*10; val[0]=1;val[1]=1; MatSetValues((*FD).ddx,1,row,2,col,val,INSERT_VALUES); for (j=1;j<10-1;j++) { col[0]=i*10+j-1;col[1]=i*10+j+1; row[0]=i*10+j; val[0]=-1;val[1]=1; MatSetValues((*FD).ddx,1,row,2,col,val,INSERT_VALUES); } col[0]=i*10+10-2;col[1]=i*10+10-1; row[0]=i*10+10-1; val[0]=-1;val[1]=-1; MatSetValues((*FD).ddx,1,row,2,col,val,INSERT_VALUES); } MatAssemblyBegin((*FD).ddx,MAT_FINAL_ASSEMBLY); MatAssemblyEnd((*FD).ddx,MAT_FINAL_ASSEMBLY); MatScale((*FD).ddx,1/(2*(1/9))); [snip] DMCreateGlobalVector(da,&tmpvec2); VecSet(tmpvec2,1.0); VecAssemblyBegin(tmpvec2); VecAssemblyEnd(tmpvec2); DMCreateGlobalVector(da,&tmpvec3); VecSet(tmpvec3,1.0); VecAssemblyBegin(tmpvec3); VecAssemblyEnd(tmpvec3); MatView((*FD).ddx,PETSC_VIEWER_STDOUT_WORLD); VecView(tmpvec2,PETSC_VIEWER_STDOUT_WORLD); MatMult((*FD).ddx,tmpvec2,tmpvec3); VecView(tmpvec3,PETSC_VIEWER_STDOUT_WORLD); int tid,first,last; MPI_Comm_rank(PETSC_COMM_WORLD, &tid); sleep(1); MatGetOwnershipRange((*FD).ddx,&first,&last); printf("rank: %d,first: %d,last: %d\n",tid,first,last); When running it on a single processor, everything works as expected, see attached file seqRes However when running with 4 processors (mpirun -np 4 ./progname) I get the output in mpiRes. Notice that there really is a difference, its not just a surprising division of points between the processes - I checked this with PETSC_VIEWER_DRAW_WORLD. How come? I notice that although in the end each process postulates that it has 25 rows, the result of matview is Matrix Object: 1 MPI processes type: mpiaij Is this OK? And if not, what am I doing wrong, presumably in the matrix allocation code? --- yours sincerily Frederik Treue -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredva at ifi.uio.no Fri Dec 2 03:41:52 2011 From: fredva at ifi.uio.no (Fredrik Heffer Valdmanis) Date: Fri, 2 Dec 2011 10:41:52 +0100 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: 2011/12/1 Matthew Knepley > On Thu, Dec 1, 2011 at 5:39 AM, Fredrik Heffer Valdmanis < > fredva at ifi.uio.no> wrote: > >> >> 2011/11/29 Matthew Knepley >> >>> On Tue, Nov 29, 2011 at 10:37 AM, Fredrik Heffer Valdmanis < >>> fredva at ifi.uio.no> wrote: >>> >>>> 2011/11/29 Matthew Knepley >>>> >>>>> On Tue, Nov 29, 2011 at 2:38 AM, Fredrik Heffer Valdmanis < >>>>> fredva at ifi.uio.no> wrote: >>>>> >>>>>> 2011/10/28 Matthew Knepley >>>>>> >>>>>>> On Fri, Oct 28, 2011 at 10:24 AM, Fredrik Heffer Valdmanis < >>>>>>> fredva at ifi.uio.no> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am working on integrating the new GPU based vectors and matrices >>>>>>>> into FEniCS. Now, I'm looking at the possibility for getting some speedup >>>>>>>> during finite element assembly, specifically when inserting the local >>>>>>>> element matrix into the global element matrix. In that regard, I have a few >>>>>>>> questions I hope you can help me out with: >>>>>>>> >>>>>>>> - When calling MatSetValues with a MATSEQAIJCUSP matrix as >>>>>>>> parameter, what exactly is it that happens? As far as I can see, >>>>>>>> MatSetValues is not implemented for GPU based matrices, neither is >>>>>>>> the mat->ops->setvalues set to point at any function for this Mat type. >>>>>>>> >>>>>>> >>>>>>> Yes, MatSetValues always operates on the CPU side. It would not make >>>>>>> sense to do individual operations on the GPU. >>>>>>> >>>>>>> I have written batched of assembly for element matrices that are all >>>>>>> the same size: >>>>>>> >>>>>>> >>>>>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBatch.html >>>>>>> >>>>>>> >>>>>>>> - Is it such that matrices are assembled in their entirety on the >>>>>>>> CPU, and then copied over to the GPU (after calling MatAssemblyBegin)? Or >>>>>>>> are values copied over to the GPU each time you call MatSetValues? >>>>>>>> >>>>>>> >>>>>>> That function assembles the matrix on the GPU and then copies to the >>>>>>> CPU. The only time you do not want this copy is when >>>>>>> you are running in serial and never touch the matrix afterwards, so >>>>>>> I left it in. >>>>>>> >>>>>>> >>>>>>>> - Can we expect to see any speedup from using MatSetValuesBatch >>>>>>>> over MatSetValues, or is the batch version simply a utility function? This >>>>>>>> question goes for both CPU- and GPU-based matrices. >>>>>>>> >>>>>>> >>>>>>> CPU: no >>>>>>> >>>>>>> GPU: yes, I see about the memory bandwidth ratio >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>> >>>>>> I have now integrated MatSetValuesBatch in our existing PETSc wrapper >>>>>> layer. I have tested matrix assembly with Poisson's equation on different >>>>>> meshes with elements of varying order. I have timed the single call to >>>>>> MatSetValuesBatch and compared that to the total time consumed by the >>>>>> repeated calls to MatSetValues in the old implementation. I have the >>>>>> following results: >>>>>> >>>>>> Poisson on 1000x1000 unit square, 1st order Lagrange elements: >>>>>> MatSetValuesBatch: 0.88576 s >>>>>> repeated calls to MatSetValues: 0.76654 s >>>>>> >>>>>> Poisson on 500x500 unit square, 2nd order Lagrange elements: >>>>>> MatSetValuesBatch: 0.9324 s >>>>>> repeated calls to MatSetValues: 0.81644 s >>>>>> >>>>>> Poisson on 300x300 unit square, 3rd order Lagrange elements: >>>>>> MatSetValuesBatch: 0.93988 s >>>>>> repeated calls to MatSetValues: 1.03884 s >>>>>> >>>>>> As you can see, the two methods take almost the same amount of time. >>>>>> What behavior and performance should we expect? Is there any way to >>>>>> optimize the performance of batched assembly? >>>>>> >>>>> >>>>> Almost certainly it is not dispatching to the CUDA version. The >>>>> regular version just calls MatSetValues() in a loop. Are you >>>>> using a SEQAIJCUSP matrix? >>>>> >>>> Yes. The same matrices yields a speedup of 4-6x when solving the >>>> system on the GPU. >>>> >>> >>> Please confirm that the correct routine by running wth -info and sending >>> the output. >>> >>> Please send the output of -log_summary so I can confirm the results. >>> >>> You can run KSP ex4 and reproduce my results where I see a 5.5x speedup >>> on the GTX285 >>> >>> I am not sure what to look for in those outputs. I have uploaded the >> output of running my assembly program with -info and -log_summary, and the >> output of running ex4 with -log_summary. See >> >> http://folk.uio.no/fredva/assembly_info.txt >> http://folk.uio.no/fredva/assembly_log_summary.txt >> http://folk.uio.no/fredva/ex4_log_summary.txt >> >> Trying this on a different machine now, I actually see some speedup. 3rd >> order Poisson on 300x300 assembles in 0.211 sec on GPU and 0.4232 sec on >> CPU. For 1st order and 1000x1000 mesh, I go from 0.31 sec to 0.205 sec. >> I have tried to increase the mesh size to see if the speedup increases, >> but I hit the bad_alloc error pretty quick. >> >> For a problem of that size, should I expect even more speedup? Please let >> me know if you need any more output from test runs on my machine. >> > > Here are my results for nxn grids where n = range(150, 1350, 100). This is > using a GTX 285. What card are you using? > > I realize now that I was including the time it takes to construct the large flattended array of values that is sent to MatSetValuesBatch. I assume of course that you only time MatSetValues/MatSetValuesBatch completely isolated. If I do this, I get significant speedup as well. Sorry for the confusion here. Still, this construction has to be done somehow in order to have meaningful data to pass to MatSetValuesBatch. The way I do this is apparently almost as costly as calling MatSetValues for each local matrix. Have you got any ideas on how to speed up the construction of the values array? This has to be done very efficiently in order for batch assembly to yield any speedup overall. -- Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Fri Dec 2 03:47:23 2011 From: gdiso at ustc.edu (Gong Ding) Date: Fri, 2 Dec 2011 17:47:23 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 Message-ID: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> petsc-3.1-p8 & mumps 4.9 works well. However, petsc-3.2 p3-p5 & mumps 4.10 seems have memory problem. The code occasionally crash on Linux/X64 and alwasy crash on AIX/PPC. This problem may caused by mumps4.10 or parmetis, but I havn't test. The valgrind report: ==5139== Invalid read of size 8 ==5139== at 0x57BE749: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5139== by 0x2269296: __GrowBisection (initpart.c:200) ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) ==5139== Address 0x8718408 is 440 bytes inside a block of size 444 alloc'd ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) ==5139== by 0x226E6A2: __GKmalloc (util.c:111) ==5139== by 0x226E6E4: __idxmalloc (util.c:60) ==5139== by 0x2269014: __GrowBisection (initpart.c:101) ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) ==5139== ==5139== Invalid read of size 8 ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5139== by 0x73F8CCCCC: ??? ==5139== by 0x300000001F: ??? ==5139== by 0x7FEFF8B4F: ??? ==5139== by 0x7FEFF8C37: ??? ==5139== by 0xE326EEF: ??? ==5139== by 0x216F: ??? ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) ==5139== Address 0x9288b38 is 0 bytes after a block of size 15,224 alloc'd ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) ==5139== by 0x226E6A2: __GKmalloc (util.c:111) ==5139== by 0x226E6E4: __idxmalloc (util.c:60) ==5139== by 0x223DB60: __MlevelNodeBisectionMultiple (ometis.c:402) ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) ==5139== ==5139== Invalid read of size 8 ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5139== by 0x21913DD: dmumps_557_ (dmumps_part2.F:2147) ==5139== by 0x218E243: dmumps_195_ (dmumps_part2.F:1535) ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) ==5139== Address 0xf24f6e8 is 3,163,816 bytes inside a block of size 3,163,820 alloc'd ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) ==5139== by 0x5CB3C43: for_allocate (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) ==5139== by 0x5CB3B50: for_alloc_allocatable (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) ==5139== by 0x218CD64: dmumps_195_ (dmumps_part2.F:1072) ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) From gdiso at ustc.edu Fri Dec 2 03:52:06 2011 From: gdiso at ustc.edu (Gong Ding) Date: Fri, 2 Dec 2011 17:52:06 +0800 (CST) Subject: [petsc-users] many valgrind error of petsc string functions Message-ID: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> Valgrind reported many errors about string processing functionss of petsc. These error seems harmless, but really ugly. Some are ==5137== Invalid read of size 8 ==5137== at 0x171A9EA: PetscStrlen (str.c:149) ==5137== by 0x171B272: PetscStrallocpy (str.c:185) ==5137== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5137== by 0x16C833A: PetscFListFind (reg.c:355) ==5137== by 0x1871568: MatSetType (matreg.c:62) ==5137== by 0x197B2D8: MatGetFactor_aij_mumps (mumps.c:1310) ==5137== by 0x1845DD3: MatGetFactor (matrix.c:3950) ==5137== by 0x1C9949F: PCSetUp_LU (lu.c:133) ==5137== by 0x1F5B26B: PCSetUp (precon.c:819) ==5137== by 0x1CEAD36: KSPSetUp (itfunc.c:260) ==5137== by 0x1CE67C3: KSPSolve (itfunc.c:379) ==5137== by 0x1DA4F08: SNES_KSPSolve (snes.c:3396) ==5137== Address 0xcc09e10 is 0 bytes inside a block of size 7 alloc'd ==5137== at 0x4A06936: memalign (vg_replace_malloc.c:694) ==5137== by 0x16D6EFA: PetscMallocAlign (mal.c:30) ==5137== by 0x171B2C0: PetscStrallocpy (str.c:186) ==5137== by 0x16DCD3A: PetscObjectChangeTypeName (pname.c:134) ==5137== by 0x1943A53: MatCreate_MPIAIJ (mpiaij.c:5528) ==5137== by 0x18715C9: MatSetType (matreg.c:73) ==5137== by 0x124790C: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:214) ==5137== by 0x1143BF0: DDMSolverBase::create_solver() (ddm_solver.cc:66) ==5137== by 0xFD7442: DDM1Solver::create_solver() (ddm1.cc:46) ==5137== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) ==5137== by 0xEB9688: SolverControl::mainloop() (control.cc:200) ==5137== by 0x141B97A: main (main.cc:225) ==5137== ==5139== Invalid read of size 8 ==5139== at 0x171A9EA: PetscStrlen (str.c:149) ==5139== by 0x171B272: PetscStrallocpy (str.c:185) ==5139== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5139== by 0x16C833A: PetscFListFind (reg.c:355) ==5139== by 0x1871568: MatSetType (matreg.c:62) ==5139== by 0x197B2D8: MatGetFactor_aij_mumps (mumps.c:1310) ==5139== by 0x1845DD3: MatGetFactor (matrix.c:3950) ==5139== by 0x1C9949F: PCSetUp_LU (lu.c:133) ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) ==5139== by 0x1CEAD36: KSPSetUp (itfunc.c:260) ==5139== by 0x1CE67C3: KSPSolve (itfunc.c:379) ==5139== by 0x1DA4F08: SNES_KSPSolve (snes.c:3396) ==5139== Address 0xb1225e0 is 0 bytes inside a block of size 7 alloc'd ==5139== at 0x4A06936: memalign (vg_replace_malloc.c:694) ==5139== by 0x16D6EFA: PetscMallocAlign (mal.c:30) ==5139== by 0x171B2C0: PetscStrallocpy (str.c:186) ==5139== by 0x16DCD3A: PetscObjectChangeTypeName (pname.c:134) ==5139== by 0x1943A53: MatCreate_MPIAIJ (mpiaij.c:5528) ==5139== by 0x18715C9: MatSetType (matreg.c:73) ==5139== by 0x124790C: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:214) ==5139== by 0x1143BF0: DDMSolverBase::create_solver() (ddm_solver.cc:66) ==5139== by 0xFD7442: DDM1Solver::create_solver() (ddm1.cc:46) ==5139== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) ==5139== by 0xEB9688: SolverControl::mainloop() (control.cc:200) ==5139== by 0x141B97A: main (main.cc:225) ==5139== ==5136== Invalid read of size 8 ==5136== at 0x171A9EA: PetscStrlen (str.c:149) ==5136== by 0x171B272: PetscStrallocpy (str.c:185) ==5136== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5136== by 0x16C833A: PetscFListFind (reg.c:355) ==5136== by 0x1871568: MatSetType (matreg.c:62) ==5136== by 0x197B2D8: MatGetFactor_aij_mumps (mumps.c:1310) ==5136== by 0x1845DD3: MatGetFactor (matrix.c:3950) ==5136== by 0x1C9949F: PCSetUp_LU (lu.c:133) ==5136== by 0x1F5B26B: PCSetUp (precon.c:819) ==5136== by 0x1CEAD36: KSPSetUp (itfunc.c:260) ==5136== by 0x1CE67C3: KSPSolve (itfunc.c:379) ==5136== by 0x1DA4F08: SNES_KSPSolve (snes.c:3396) ==5136== Address 0xabde120 is 0 bytes inside a block of size 7 alloc'd ==5136== at 0x4A06936: memalign (vg_replace_malloc.c:694) ==5136== by 0x16D6EFA: PetscMallocAlign (mal.c:30) ==5136== by 0x171B2C0: PetscStrallocpy (str.c:186) ==5136== by 0x16DCD3A: PetscObjectChangeTypeName (pname.c:134) ==5136== by 0x1943A53: MatCreate_MPIAIJ (mpiaij.c:5528) ==5136== by 0x18715C9: MatSetType (matreg.c:73) ==5136== by 0x124790C: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:214) ==5136== by 0x1143BF0: DDMSolverBase::create_solver() (ddm_solver.cc:66) ==5136== by 0xFD7442: DDM1Solver::create_solver() (ddm1.cc:46) ==5136== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) ==5136== by 0xEB9688: SolverControl::mainloop() (control.cc:200) ==5136== by 0x141B97A: main (main.cc:225) ==5136== ==5138== Invalid read of size 8 ==5138== at 0x171A9EA: PetscStrlen (str.c:149) ==5138== by 0x171B272: PetscStrallocpy (str.c:185) ==5138== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5138== by 0x16C833A: PetscFListFind (reg.c:355) ==5138== by 0x1871568: MatSetType (matreg.c:62) ==5138== by 0x197B2D8: MatGetFactor_aij_mumps (mumps.c:1310) ==5138== by 0x1845DD3: MatGetFactor (matrix.c:3950) ==5138== by 0x1C9949F: PCSetUp_LU (lu.c:133) ==5138== by 0x1F5B26B: PCSetUp (precon.c:819) ==5138== by 0x1CEAD36: KSPSetUp (itfunc.c:260) ==5138== by 0x1CE67C3: KSPSolve (itfunc.c:379) ==5138== by 0x1DA4F08: SNES_KSPSolve (snes.c:3396) ==5138== Address 0xe316cb0 is 0 bytes inside a block of size 7 alloc'd ==5138== at 0x4A06936: memalign (vg_replace_malloc.c:694) ==5138== by 0x16D6EFA: PetscMallocAlign (mal.c:30) ==5138== by 0x171B2C0: PetscStrallocpy (str.c:186) ==5138== by 0x16DCD3A: PetscObjectChangeTypeName (pname.c:134) ==5138== by 0x1943A53: MatCreate_MPIAIJ (mpiaij.c:5528) ==5138== by 0x18715C9: MatSetType (matreg.c:73) ==5138== by 0x124790C: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:214) ==5138== by 0x1143BF0: DDMSolverBase::create_solver() (ddm_solver.cc:66) ==5138== by 0xFD7442: DDM1Solver::create_solver() (ddm1.cc:46) ==5138== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) ==5138== by 0xEB9688: SolverControl::mainloop() (control.cc:200) ==5138== by 0x141B97A: main (main.cc:225) and more ==5138== Conditional jump or move depends on uninitialised value(s) ==5138== at 0x171A9FA: PetscStrlen (str.c:149) ==5138== by 0x171B272: PetscStrallocpy (str.c:185) ==5138== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5138== by 0x16CA932: PetscFListAdd (reg.c:204) ==5138== by 0x1DA2D21: SNESRegister (snes.c:3059) ==5138== by 0x1D8AD1C: SNESRegisterAll (snesregi.c:48) ==5138== by 0x1DAF496: SNESInitializePackage (dlregissnes.c:53) ==5138== by 0x1DA6FE9: SNESCreate (snes.c:1051) ==5138== by 0x1246CD6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:158) ==5138== by 0x12500F6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:0) ==5138== by 0x11439E2: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:36) ==5138== by 0x114C0F8: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:0) ==5138== ==5138== Conditional jump or move depends on uninitialised value(s) ==5138== at 0x171B284: PetscStrallocpy (str.c:186) ==5138== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5138== by 0x16CA932: PetscFListAdd (reg.c:204) ==5138== by 0x1DA2D21: SNESRegister (snes.c:3059) ==5138== by 0x1D8AD1C: SNESRegisterAll (snesregi.c:48) ==5138== by 0x1DAF496: SNESInitializePackage (dlregissnes.c:53) ==5138== by 0x1DA6FE9: SNESCreate (snes.c:1051) ==5138== by 0x1246CD6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:158) ==5138== by 0x12500F6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:0) ==5138== by 0x11439E2: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:36) ==5138== by 0x114C0F8: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:0) ==5137== Conditional jump or move depends on uninitialised value(s) ==5137== at 0x171A9FA: PetscStrlen (str.c:149) ==5137== by 0x171B272: PetscStrallocpy (str.c:185) ==5137== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5137== by 0x16CA932: PetscFListAdd (reg.c:204) ==5137== by 0x1DA2D21: SNESRegister (snes.c:3059) ==5137== by 0x1D8AD1C: SNESRegisterAll (snesregi.c:48) ==5137== by 0x1DAF496: SNESInitializePackage (dlregissnes.c:53) ==5137== by 0x1DA6FE9: SNESCreate (snes.c:1051) ==5138== by 0xEB6E86: DDM1Solver::DDM1Solver(SimulationSystem&) (ddm1.h:37) ==5137== by 0x1246CD6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:158) ==5137== by 0x12500F6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:0) ==5137== by 0x11439E2: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:36) ==5137== by 0x114C0F8: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:0) ==5137== ==5137== Conditional jump or move depends on uninitialised value(s) ==5137== at 0x171B284: PetscStrallocpy (str.c:186) ==5137== by 0x16CAD73: PetscFListGetPathAndFunction (reg.c:24) ==5137== by 0x16CA932: PetscFListAdd (reg.c:204) ==5137== by 0x1DA2D21: SNESRegister (snes.c:3059) ==5137== by 0x1D8AD1C: SNESRegisterAll (snesregi.c:48) ==5137== by 0x1DAF496: SNESInitializePackage (dlregissnes.c:53) ==5137== by 0x1DA6FE9: SNESCreate (snes.c:1051) ==5137== by 0x1246CD6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:158) ==5137== by==5138== 0x12500F6: FVM_NonlinearSolver::FVM_NonlinearSolver(SimulationSystem&) (fvm_nonlinear_solver.cc:0) ==5137== by 0x11439E2: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:36) ==5137== by 0x114C0F8: DDMSolverBase::DDMSolverBase(SimulationSystem&) (ddm_solver.cc:0) ==5137== by 0xEB6E86: DDM1Solver::DDM1Solver(SimulationSystem&) (ddm1.h:37) From fredva at ifi.uio.no Fri Dec 2 04:54:20 2011 From: fredva at ifi.uio.no (Fredrik Heffer Valdmanis) Date: Fri, 2 Dec 2011 11:54:20 +0100 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: 2011/12/2 Fredrik Heffer Valdmanis > 2011/12/1 Matthew Knepley > >> On Thu, Dec 1, 2011 at 5:39 AM, Fredrik Heffer Valdmanis < >> fredva at ifi.uio.no> wrote: >> >>> >>> 2011/11/29 Matthew Knepley >>> >>>> On Tue, Nov 29, 2011 at 10:37 AM, Fredrik Heffer Valdmanis < >>>> fredva at ifi.uio.no> wrote: >>>> >>>>> 2011/11/29 Matthew Knepley >>>>> >>>>>> On Tue, Nov 29, 2011 at 2:38 AM, Fredrik Heffer Valdmanis < >>>>>> fredva at ifi.uio.no> wrote: >>>>>> >>>>>>> 2011/10/28 Matthew Knepley >>>>>>> >>>>>>>> On Fri, Oct 28, 2011 at 10:24 AM, Fredrik Heffer Valdmanis < >>>>>>>> fredva at ifi.uio.no> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am working on integrating the new GPU based vectors and matrices >>>>>>>>> into FEniCS. Now, I'm looking at the possibility for getting some speedup >>>>>>>>> during finite element assembly, specifically when inserting the local >>>>>>>>> element matrix into the global element matrix. In that regard, I have a few >>>>>>>>> questions I hope you can help me out with: >>>>>>>>> >>>>>>>>> - When calling MatSetValues with a MATSEQAIJCUSP matrix as >>>>>>>>> parameter, what exactly is it that happens? As far as I can see, >>>>>>>>> MatSetValues is not implemented for GPU based matrices, neither is >>>>>>>>> the mat->ops->setvalues set to point at any function for this Mat type. >>>>>>>>> >>>>>>>> >>>>>>>> Yes, MatSetValues always operates on the CPU side. It would not >>>>>>>> make sense to do individual operations on the GPU. >>>>>>>> >>>>>>>> I have written batched of assembly for element matrices that are >>>>>>>> all the same size: >>>>>>>> >>>>>>>> >>>>>>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBatch.html >>>>>>>> >>>>>>>> >>>>>>>>> - Is it such that matrices are assembled in their entirety on the >>>>>>>>> CPU, and then copied over to the GPU (after calling MatAssemblyBegin)? Or >>>>>>>>> are values copied over to the GPU each time you call MatSetValues? >>>>>>>>> >>>>>>>> >>>>>>>> That function assembles the matrix on the GPU and then copies to >>>>>>>> the CPU. The only time you do not want this copy is when >>>>>>>> you are running in serial and never touch the matrix afterwards, so >>>>>>>> I left it in. >>>>>>>> >>>>>>>> >>>>>>>>> - Can we expect to see any speedup from using MatSetValuesBatch >>>>>>>>> over MatSetValues, or is the batch version simply a utility function? This >>>>>>>>> question goes for both CPU- and GPU-based matrices. >>>>>>>>> >>>>>>>> >>>>>>>> CPU: no >>>>>>>> >>>>>>>> GPU: yes, I see about the memory bandwidth ratio >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>> >>>>>>> I have now integrated MatSetValuesBatch in our existing PETSc >>>>>>> wrapper layer. I have tested matrix assembly with Poisson's equation on >>>>>>> different meshes with elements of varying order. I have timed the single >>>>>>> call to MatSetValuesBatch and compared that to the total time consumed by >>>>>>> the repeated calls to MatSetValues in the old implementation. I have the >>>>>>> following results: >>>>>>> >>>>>>> Poisson on 1000x1000 unit square, 1st order Lagrange elements: >>>>>>> MatSetValuesBatch: 0.88576 s >>>>>>> repeated calls to MatSetValues: 0.76654 s >>>>>>> >>>>>>> Poisson on 500x500 unit square, 2nd order Lagrange elements: >>>>>>> MatSetValuesBatch: 0.9324 s >>>>>>> repeated calls to MatSetValues: 0.81644 s >>>>>>> >>>>>>> Poisson on 300x300 unit square, 3rd order Lagrange elements: >>>>>>> MatSetValuesBatch: 0.93988 s >>>>>>> repeated calls to MatSetValues: 1.03884 s >>>>>>> >>>>>>> As you can see, the two methods take almost the same amount of time. >>>>>>> What behavior and performance should we expect? Is there any way to >>>>>>> optimize the performance of batched assembly? >>>>>>> >>>>>> >>>>>> Almost certainly it is not dispatching to the CUDA version. The >>>>>> regular version just calls MatSetValues() in a loop. Are you >>>>>> using a SEQAIJCUSP matrix? >>>>>> >>>>> Yes. The same matrices yields a speedup of 4-6x when solving the >>>>> system on the GPU. >>>>> >>>> >>>> Please confirm that the correct routine by running wth -info and >>>> sending the output. >>>> >>>> Please send the output of -log_summary so I can confirm the results. >>>> >>>> You can run KSP ex4 and reproduce my results where I see a 5.5x speedup >>>> on the GTX285 >>>> >>>> I am not sure what to look for in those outputs. I have uploaded the >>> output of running my assembly program with -info and -log_summary, and the >>> output of running ex4 with -log_summary. See >>> >>> http://folk.uio.no/fredva/assembly_info.txt >>> http://folk.uio.no/fredva/assembly_log_summary.txt >>> http://folk.uio.no/fredva/ex4_log_summary.txt >>> >>> Trying this on a different machine now, I actually see some speedup. 3rd >>> order Poisson on 300x300 assembles in 0.211 sec on GPU and 0.4232 sec on >>> CPU. For 1st order and 1000x1000 mesh, I go from 0.31 sec to 0.205 sec. >>> I have tried to increase the mesh size to see if the speedup increases, >>> but I hit the bad_alloc error pretty quick. >>> >>> For a problem of that size, should I expect even more speedup? Please >>> let me know if you need any more output from test runs on my machine. >>> >> >> Here are my results for nxn grids where n = range(150, 1350, 100). This >> is using a GTX 285. What card are you using? >> >> I realize now that I was including the time it takes to construct the > large flattended array of values that is sent to MatSetValuesBatch. I > assume of course that you only time MatSetValues/MatSetValuesBatch > completely isolated. If I do this, I get significant speedup as well. Sorry > for the confusion here. > > Still, this construction has to be done somehow in order to have > meaningful data to pass to MatSetValuesBatch. The way I do this is > apparently almost as costly as calling MatSetValues for each local matrix. > > Have you got any ideas on how to speed up the construction of the values > array? This has to be done very efficiently in order for batch assembly to > yield any speedup overall. > > Arg, disregard last transmission! I was confusing myself with timings from several runs, and the "significant speedup" I referred to was seen when I timed things very badly. The numbers from yesterdays mail are correct, those were obtained using a GTX 280. That is, 30%-50% speedup on Poisson 2D on different meshes. The question from my previous email remains though, we need to speed up the construction of the values array to get good speedup overall. Sorry for the spamming, Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From logg at simula.no Fri Dec 2 05:19:24 2011 From: logg at simula.no (Anders Logg) Date: Fri, 2 Dec 2011 12:19:24 +0100 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: <20111202111922.GV4889@smaug> On Fri, Dec 02, 2011 at 11:54:20AM +0100, Fredrik Heffer Valdmanis wrote: > 2011/12/2 Fredrik Heffer Valdmanis > 2011/12/1 Matthew Knepley > On Thu, Dec 1, 2011 at 5:39 AM, Fredrik Heffer Valdmanis > wrote: > > 2011/11/29 Matthew Knepley > On Tue, Nov 29, 2011 at 10:37 AM, > Fredrik Heffer Valdmanis > wrote: > 2011/11/29 Matthew Knepley > > On Tue, Nov 29, > 2011 at 2:38 AM, > Fredrik Heffer > Valdmanis > > wrote: > 2011/10/28 > Matthew > Knepley > > On > Fri, > Oct > 28, > 2011 > at > 10:24 > AM, > Fredrik > Heffer > Valdmanis > > wrote: > Hi, > > I > am > working > on > integrating > the > new > GPU > based > vectors > and > matrices > into > FEniCS. > Now, > I'm > looking > at > the > possibility > for > getting > some > speedup > during > finite > element > assembly, > specifically > when > inserting > the > local > element > matrix > into > the > global > element > matrix. > In > that > regard, > I > have > a > few > questions > I > hope > you > can > help > me > out > with: > > - > When > calling > MatSetValues > with > a > MATSEQAIJCUSP > matrix > as > parameter, > what > exactly > is > it > that > happens? > As > far > as > I > can > see, > MatSetValues > is > not > implemented > for > GPU > based > matrices, > neither > is > the?mat- > >ops- > >setvalues > set > to > point > at > any > function > for > this > Mat > type.? > > Yes, > MatSetValues > always > operates > on > the > CPU > side. > It > would > not > make > sense > to do > individual > operations > on > the > GPU. > > I > have > written > batched > of > assembly > for > element > matrices > that > are > all > the > same > size: > > ??http: > // > www.mcs.anl.gov/ > petsc/ > petsc- > as/ > snapshots/ > petsc- > current/ > docs/ > manualpages/ > Mat/ > MatSetValuesBatch.html > ? > - > Is > it > such > that > matrices > are > assembled > in > their > entirety > on > the > CPU, > and > then > copied > over > to > the > GPU > (after > calling > MatAssemblyBegin)? > Or > are > values > copied > over > to > the > GPU > each > time > you > call > MatSetValues? > > That > function > assembles > the > matrix > on > the > GPU > and > then > copies > to > the > CPU. > The > only > time > you > do > not > want > this > copy > is > when > you > are > running > in > serial > and > never > touch > the > matrix > afterwards, > so I > left > it > in. > ? > - > Can > we > expect > to > see > any > speedup > from > using > MatSetValuesBatch > over > MatSetValues, > or > is > the > batch > version > simply > a > utility > function? > This > question > goes > for > both > CPU- > and > GPU- > based > matrices. > > CPU: > no > > GPU: > yes, > I see > about > the > memory > bandwidth > ratio > > > Hi, > > I have now > integrated > MatSetValuesBatch > in our > existing > PETSc > wrapper > layer. I > have > tested > matrix > assembly > with > Poisson's > equation > on > different > meshes > with > elements > of varying > order. I > have timed > the single > call to > MatSetValuesBatch > and > compared > that to > the total > time > consumed > by the > repeated > calls to > MatSetValues > in the old > implementation. > I have the > following > results: > > Poisson on > 1000x1000 > unit > square, > 1st order > Lagrange > elements: > MatSetValuesBatch: > 0.88576 s > repeated > calls to > MatSetValues: > 0.76654 s > > Poisson on > 500x500 > unit > square, > 2nd?order > Lagrange > elements: > MatSetValuesBatch: > 0.9324 s > repeated > calls to > MatSetValues: > 0.81644 s > > Poisson on > 300x300 > unit > square, > 3rd?order > Lagrange > elements: > MatSetValuesBatch: > 0.93988 s > repeated > calls to > MatSetValues: > 1.03884 s > > As you can > see, the > two > methods > take > almost the > same > amount of > time. > What?behavior?and > performance > should we > expect? Is > there any > way to > optimize > the > performance > of batched > assembly? > > Almost certainly it > is not dispatching > to the CUDA > version. The > regular version > just calls > MatSetValues() in a > loop. Are you > using a SEQAIJCUSP > matrix? > Yes. The same matrices yields > a speedup of 4-6x when > solving the system on the > GPU.? > > Please confirm that the correct routine > by running wth -info and sending the > output. > > Please send the output of -log_summary > so I can confirm the results. > > You can run KSP ex4 and reproduce my > results where I see a 5.5x speedup on > the GTX285 > > I am not sure what to look for in those outputs. > I have uploaded the output of running my assembly > program with -info and -log_summary, and the > output of running ex4 with -log_summary. See > > http://folk.uio.no/fredva/assembly_info.txt > http://folk.uio.no/fredva/ > assembly_log_summary.txt > http://folk.uio.no/fredva/ex4_log_summary.txt > > Trying this on a different machine now, I > actually see some speedup. 3rd order Poisson on > 300x300 assembles in 0.211 sec on GPU and 0.4232 > sec on CPU. For 1st order and 1000x1000 mesh, I > go from 0.31 sec to 0.205 sec.? > I have tried to increase the mesh size to see if > the speedup increases, but I hit the bad_alloc > error pretty quick.? > > For a problem of that size, should I expect even > more speedup? Please let me know if you need any > more output from test runs on my machine.? > > Here are my results for nxn grids where n = range(150, > 1350, 100). This is using a GTX 285. What card are you > using? > > I realize now that I was including the time it takes to construct the > large flattended array of values that is sent to MatSetValuesBatch. I > assume of course that you only time MatSetValues/MatSetValuesBatch > completely isolated. If I do this, I get significant speedup as well. > Sorry for the confusion here.? > > Still, this construction has to be done somehow in order to have > meaningful data to pass to MatSetValuesBatch. The way I do this is > apparently almost as costly as calling MatSetValues for each local > matrix. > > Have you got any ideas on how to speed up the construction of the > values array? This has to be done very efficiently in order for batch > assembly to yield any speedup overall.? > > Arg, disregard last transmission! I was confusing myself with timings from > several runs, and the "significant speedup" I referred to was seen > when I timed things very badly. The numbers from yesterdays mail are correct, > those were obtained using a GTX 280. That is, 30%-50% speedup on Poisson 2D on > different meshes.? > > The question from my previous email remains though, we need to speed up the > construction of the values array to get good speedup overall.? > > Sorry for the spamming, Off-topic: I find this thread extremely hard to follow. Is Gmail required to read this list? The html-formatting with indentation (and no ">") makes it really hard to read in my email-client (mutt). -- Anders From jedbrown at mcs.anl.gov Fri Dec 2 06:13:58 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 06:13:58 -0600 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: <20111202111922.GV4889@smaug> References: <20111202111922.GV4889@smaug> Message-ID: On Fri, Dec 2, 2011 at 05:19, Anders Logg wrote: > Off-topic: I find this thread extremely hard to follow. Is Gmail > required to read this list? The html-formatting with indentation (and > no ">") makes it really hard to read in my email-client (mutt). > Gmail does clip adequately in the default view, and it manages levels of indentation if you expand, but I still find it annoying when people don't clip to only the relevant part of what they are replying to. If mutt is worse than this, you should be able to change its configuration to make it tolerable: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-December/011287.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 2 06:28:02 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 06:28:02 -0600 Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/12/2 Gong Ding > Valgrind reported many errors about string processing functionss of petsc. > These error seems harmless, but really ugly. > Agreed, I think you have a really long path (but still much shorter than the system path length). If this is the problem, then the fix I just pushed to petsc-3.2 will fix it. http://petsc.cs.iit.edu/petsc/releases/petsc-3.2/rev/36bd87d582e9 You can pull this with mercurial or apply manually. Let me know if this works. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 2 06:32:24 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 06:32:24 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 03:32, Treue, Frederik wrote: > OK, but that example seems to assume that you wish to connect only one > matrix (the Jacobian) to a DA ? I wish to specify many and I think I found > this done in ksp ex39, is that example doing anything deprecated or will > that work for me, e.g. with the various basic mat routines (matmult, > matAXPY etc.) in a multiprocessor setup? What do you mean by wanting many matrices? How do you want to use them? There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can use as many times as you want. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Parker2 at baesystems.com Fri Dec 2 06:40:31 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Fri, 2 Dec 2011 12:40:31 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: Ok, So I've tried what you suggested, I get this: [0]PETSC ERROR: MatInvertBlockDiagonal_SeqBAIJ() line 129 in src/mat/impls/baij/seq/baij.c not supported for block size 7 I'm using these settings: -ksp_type preonly -pc_type pbjacobi I have created a SeqBAIJ mat Any thoughts, must be missing something.... Many thanks, Andy -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: 01 December 2011 19:57 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. My approach will be a million times faster. Barry On Dec 1, 2011, at 8:50 AM, Matthew Knepley wrote: > On Thu, Dec 1, 2011 at 8:44 AM, Parker, Andrew (UK Filton) wrote: > I have tried to use the following, this was before I had both you're > emails (Barry, Matthew) ;-) > > 1) First I set this up > > PetscInt size = 7; > MatCreate(MPI_COMM_SELF,&LHS); > MatSetSizes(LHS,size,size,size,size); > MatSetType(LHS,MATSEQDENSE); //**************NOTE THIS LINE > MatAssemblyBegin(LHS,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(LHS,MAT_FINAL_ASSEMBLY); > > VecCreate(PETSC_COMM_SELF,&RHS); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(RHS); > VecAssemblyBegin(RHS); > VecAssemblyEnd(RHS); > > VecCreate(PETSC_COMM_SELF,&Updates); > VecSetSizes(RHS,PETSC_DECIDE, size); > VecSetFromOptions(Updates); > VecAssemblyBegin(Updates); > VecAssemblyEnd(Updates); > > KSPCreate(PETSC_COMM_SELF,&Solver); > KSPSetOperators(Solver, LHS, LHS, SAME_NONZERO_PATTERN); > KSPSetType(Solver, KSPPREONLY); //**************NOTE THIS LINE > KSPGetPC(Solver, &prec); > PCSetType(prec, PCLU); //**************NOTE THIS LINE > KSPSetFromOptions(Solver); > KSPSetUp(Solver); > > For each cell I then do: > > 2) I copy out the appropriate data from global structures into LHS and > RHS > > 3) KSPSolve(Solver, RHS, Updates); > > 4) Stick the local Updates back into my global data structure > > Could I have your thoughts on whether the above is ok? In terms of > efficiency and the correct thing to do? Would I be correct in assuming > that Barry's method and mine are identical with the exception that I've > got some copies going on to set up these small 7x7 and 1x7 matrices and > vectors, but Barry's method does this inline by doing a preonly, lu > solve DIRECT solve on each 7x7?? > > With Barry's method you can do this from the command line so you can easily compare > with other solvers without changing your code. > > Matt > > Many thanks, > Andy > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: 01 December 2011 14:09 > To: PETSc users list > Subject: Re: [petsc-users] MATBDIAG in > 3.1 > > > *** WARNING *** > > This message has originated outside your organisation, > either from an external partner or the Global Internet. > Keep this in mind if you answer this message. > > > On Dec 1, 2011, at 7:05 AM, Matthew Knepley wrote: > > > You can get this effect using PCBJACOBI, setting the number of local > blocks, and using -sub_pc_type lu -sub_ksp_type preonly. > > Do not do this. It will give bad performance since it explicitly > pulls out each block with MatGetSubMatrices() and solves each small > matrix as a separate PETSc matrix. > > Instead use the BAIJ matrix format with a block size of 7 and then > use a PCPBJACOBI (note the P in the name for point block, each point > block is that little 7 by 7 block that it solves efficiently using a > tuned kernel) with -ksp_type preonly. This should be very fast and does > not do any any neccessary scatters or communication. > > Barry > > > > > Matt > > > > On Thu, Dec 1, 2011 at 4:28 AM, Parker, Andrew (UK Filton) > wrote: > > Hi, > > > > > > > > I've been using PETSc for a conventional full parallel implicit > solver, currently in MPIBAIJ format. However, I'd like to understand > the performance verse memory and speed trade-offs of just having a block > diagonal system. In this case I do not scatter to the off diagonal > blocks. Each block is 7x7 in size and fully dense. In my view this > should be a point implicit solution of the system. I am looking for the > best way to solve this in PETSc, clearly because of the lack of coupling > between neighbouring cells by only having a block diagonal it's actually > a 7x7 dense linear solve, but to test my theory at present I'd prefer to > just not scatter to off diagonals, and use: > > > > > > > > MATBDIAG with my current ksp solver and take it from there. BDIAG > seems to have been removed as off 3.0: > > > > > > > > http://www.mcs.anl.gov/petsc/documentation/changes/300.html > > > > > > > > Can anybody help me with this, your thoughts on the above and what I > should use to solve this reduced system would be appreciated?? > > > > > > Cheers, > > > > Andy > > > > > > ******************************************************************** > > This email and any attachments are confidential to the intended > > recipient and may also be privileged. If you are not the intended > > recipient please delete it from your system and notify the sender. > > You should not copy it or use it for any purpose nor disclose or > > distribute its contents to any other person. > > ******************************************************************** > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jedbrown at mcs.anl.gov Fri Dec 2 06:42:15 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 06:42:15 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: On Fri, Dec 2, 2011 at 06:40, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > [0]PETSC ERROR: MatInvertBlockDiagonal_SeqBAIJ() line 129 in > src/mat/impls/baij/seq/baij.c not supported for block size 7 > This was fixed in petsc-3.2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 07:17:38 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 07:17:38 -0600 Subject: [petsc-users] Questions about setting values for GPU based matrices In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 4:54 AM, Fredrik Heffer Valdmanis wrote: > Arg, disregard last transmission! I was confusing myself with timings from > several runs, and the "significant speedup" I referred to was seen when I > timed things very badly. The numbers from yesterdays mail are correct, > those were obtained using a GTX 280. That is, 30%-50% speedup on Poisson 2D > on different meshes. > > The question from my previous email remains though, we need to speed up > the construction of the values array to get good speedup overall. > > Sorry for the spamming, > 0) zthis discussion has become too long and belongs on petsc-maint at mcs.anl.gov 1) I believe timings from -log_summary and no where else. Please send that output for any timing discussion 2) I sent instructions for running ex4 and data. Can you reproduce that? 3) I do not need to do any data conversion when calling the batch routine. You should not need to either. See ex4 Matt > Fredrik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Fri Dec 2 08:04:59 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Fri, 2 Dec 2011 22:04:59 +0800 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file Message-ID: <201112022204567009476@gmail.com> Dear all, I intend to postprocess the solution of a PDE (stored in a parallel PETSc vector ) using tecplot. Thus I want to output the values of the parallel vector as well as other data (such as the grid information) to a ASCII file. When I deal this with VecGetArray(U, &u); for (i=0; i From knepley at gmail.com Fri Dec 2 08:07:25 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 08:07:25 -0600 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file In-Reply-To: <201112022204567009476@gmail.com> References: <201112022204567009476@gmail.com> Message-ID: On Fri, Dec 2, 2011 at 8:04 AM, Feng-Chao Wang wrote: > ** > Dear all, > > I intend to postprocess the solution of a PDE (stored in a > parallel PETSc vector ) using tecplot. Thus I want to output the values of > the parallel vector as well as other data (such as the grid information) to > a ASCII file. > > When I deal this with > > > VecGetArray(U, &u); > > for (i=0; i { > PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); > } > > If more than 1 processor are used, I found that only part of the value of > u was output correctly (that stored on the first processor). > I found in the manual page of "PetscViewerASCIIPrintf" that "Not > Collective, but only first processor in set has any effect" > > Could anyone give me some suggestions on this issue? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecView.html Matt > Thank you very much in advance! > > Best Regards > > > > > > 2011-12-02 > ------------------------------ > Feng-Chao Wang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 2 08:07:54 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 08:07:54 -0600 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file In-Reply-To: <201112022204567009476@gmail.com> References: <201112022204567009476@gmail.com> Message-ID: On Fri, Dec 2, 2011 at 08:04, Feng-Chao Wang wrote: > ** > Dear all, > > I intend to postprocess the solution of a PDE (stored in a > parallel PETSc vector ) using tecplot. Thus I want to output the values of > the parallel vector as well as other data (such as the grid information) to > a ASCII file. > > When I deal this with > > > VecGetArray(U, &u); > > for (i=0; i { > PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); > } > > If more than 1 processor are used, I found that only part of the value of > u was output correctly (that stored on the first processor). > I found in the manual page of "PetscViewerASCIIPrintf" that "Not > Collective, but only first processor in set has any effect" > If you can afford the memory: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html Slower: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscSynchronizedPrintf.html Binary output can be written in parallel using http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecView.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 08:11:25 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 08:11:25 -0600 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file In-Reply-To: References: <201112022204567009476@gmail.com> Message-ID: On Fri, Dec 2, 2011 at 8:07 AM, Jed Brown wrote: > > > On Fri, Dec 2, 2011 at 08:04, Feng-Chao Wang wrote: > >> ** >> Dear all, >> >> I intend to postprocess the solution of a PDE (stored in a >> parallel PETSc vector ) using tecplot. Thus I want to output the values of >> the parallel vector as well as other data (such as the grid information) to >> a ASCII file. >> >> When I deal this with >> >> >> VecGetArray(U, &u); >> >> for (i=0; i> { >> PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); >> } >> >> If more than 1 processor are used, I found that only part of the value of >> u was output correctly (that stored on the first processor). >> I found in the manual page of "PetscViewerASCIIPrintf" that "Not >> Collective, but only first processor in set has any effect" >> > > If you can afford the memory: > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html > > Slower: > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscSynchronizedPrintf.html > > Binary output can be written in parallel using > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecView.html > Note that VecView also does ASCII output Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at risoe.dtu.dk Fri Dec 2 08:58:58 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Fri, 2 Dec 2011 15:58:58 +0100 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, December 02, 2011 1:32 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 03:32, Treue, Frederik > wrote: OK, but that example seems to assume that you wish to connect only one matrix (the Jacobian) to a DA ? I wish to specify many and I think I found this done in ksp ex39, is that example doing anything deprecated or will that work for me, e.g. with the various basic mat routines (matmult, matAXPY etc.) in a multiprocessor setup? What do you mean by wanting many matrices? How do you want to use them? There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can use as many times as you want.` And this was the one I needed. However I have another question: What does DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From experience I now know that the PERIODIC option automagically does the right thing when I?m defining matrices so I can simply specify the same stencil at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? And if so, how is it controlled, ie. How do I specify if I?ve got Neumann or Dirichlet conditions, and what order extrapolation you want, and so forth? And if not, does it then ONLY make a difference if I?m working with more than on processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE equivalent? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 09:01:23 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 09:01:23 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik wrote: > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Friday, December 02, 2011 1:32 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] newbie question on the parallel allocation > of matrices**** > > ** ** > > On Fri, Dec 2, 2011 at 03:32, Treue, Frederik wrote:** > ** > > OK, but that example seems to assume that you wish to connect only one > matrix (the Jacobian) to a DA ? I wish to specify many and I think I found > this done in ksp ex39, is that example doing anything deprecated or will > that work for me, e.g. with the various basic mat routines (matmult, > matAXPY etc.) in a multiprocessor setup?**** > > ** ** > > What do you mean by wanting many matrices? How do you want to use them? There > is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can > use as many times as you want.`**** > > ** ** > > And this was the one I needed. However I have another question: What does > DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From > experience I now know that the PERIODIC option automagically does the right > thing when I?m defining matrices so I can simply specify the same stencil > at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? And if so, > how is it controlled, ie. How do I specify if I?ve got Neumann or Dirichlet > conditions, and what order extrapolation you want, and so forth? And if > not, does it then ONLY make a difference if I?m working with more than on > processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and > DMDA_BOUNDARY_NONE equivalent?**** > GHOSTED adds extra space at the boundary so you can always use the same stencil, but you decide what goes in there. PERIODIC fills that extra space with the values from the connected side. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at risoe.dtu.dk Fri Dec 2 09:25:57 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Fri, 2 Dec 2011 16:25:57 +0100 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, December 02, 2011 4:01 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik > wrote: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, December 02, 2011 1:32 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 03:32, Treue, Frederik > wrote: OK, but that example seems to assume that you wish to connect only one matrix (the Jacobian) to a DA - I wish to specify many and I think I found this done in ksp ex39, is that example doing anything deprecated or will that work for me, e.g. with the various basic mat routines (matmult, matAXPY etc.) in a multiprocessor setup? What do you mean by wanting many matrices? How do you want to use them? There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can use as many times as you want.` And this was the one I needed. However I have another question: What does DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From experience I now know that the PERIODIC option automagically does the right thing when I'm defining matrices so I can simply specify the same stencil at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? And if so, how is it controlled, ie. How do I specify if I've got Neumann or Dirichlet conditions, and what order extrapolation you want, and so forth? And if not, does it then ONLY make a difference if I'm working with more than on processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE equivalent? GHOSTED adds extra space at the boundary so you can always use the same stencil, but you decide what goes in there. Does this apply to both matrices and vectors, ie. Will the ghost points be considered part of my computational domain or not? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 09:32:00 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 09:32:00 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 9:25 AM, Treue, Frederik wrote: > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, December 02, 2011 4:01 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] newbie question on the parallel allocation > of matrices**** > > ** ** > > On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik wrote: > **** > > **** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Friday, December 02, 2011 1:32 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] newbie question on the parallel allocation > of matrices**** > > **** > > On Fri, Dec 2, 2011 at 03:32, Treue, Frederik wrote:** > ** > > OK, but that example seems to assume that you wish to connect only one > matrix (the Jacobian) to a DA ? I wish to specify many and I think I found > this done in ksp ex39, is that example doing anything deprecated or will > that work for me, e.g. with the various basic mat routines (matmult, > matAXPY etc.) in a multiprocessor setup?**** > > **** > > What do you mean by wanting many matrices? How do you want to use them? There > is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can > use as many times as you want.`**** > > **** > > And this was the one I needed. However I have another question: What does > DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From > experience I now know that the PERIODIC option automagically does the right > thing when I?m defining matrices so I can simply specify the same stencil > at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? And if so, > how is it controlled, ie. How do I specify if I?ve got Neumann or Dirichlet > conditions, and what order extrapolation you want, and so forth? And if > not, does it then ONLY make a difference if I?m working with more than on > processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and > DMDA_BOUNDARY_NONE equivalent?**** > > > GHOSTED adds extra space at the boundary so you can always use the same > stencil, but you decide what goes in there. **** > > ** ** > > Does this apply to both matrices and vectors, ie. Will the ghost points be > considered part of my computational domain or not? > The ghost nodes only exist in local vectors, not the global vectors for the solver. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Parker2 at baesystems.com Fri Dec 2 09:31:58 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Fri, 2 Dec 2011 15:31:58 -0000 Subject: [petsc-users] Bug with 4.6.2 Message-ID: Hi, I appreciate that this tool may not be supported yet, but I get a compile error with gcc 4.6.2 petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: 'VecType' does not name a type petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: ISO C++ forbids declaration of 'parameter' with no type [-fpermissive] Any thoughts on how I fix this locally? But more generally needs to be fixed. Cheers, Andy ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 09:34:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 09:34:50 -0600 Subject: [petsc-users] Bug with 4.6.2 In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 9:31 AM, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > Hi,**** > > ** ** > > I appreciate that this tool may not be supported yet, but I get a compile > error with gcc 4.6.2**** > > ** ** > > petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: 'VecType' does not > name a type**** > > petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: ISO C++ forbids > declaration of 'parameter' with no type [-fpermissive]**** > > ** ** > > Any thoughts on how I fix this locally? But more generally needs to be > fixed. > You have to send the entire message. I cannot even see which file is being compiled here. Also, is the GPU stuff turned on? Matt > > > ** > > Cheers,**** > > Andy**** > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Parker2 at baesystems.com Fri Dec 2 09:39:09 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Fri, 2 Dec 2011 15:39:09 -0000 Subject: [petsc-users] Bug with 4.6.2 In-Reply-To: References: Message-ID: Not sure what you mean, it's been including form a file I've used somewhere, but the c++ type is undefined in that file.... It doesn't know what vectype is... From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: 02 December 2011 15:35 To: PETSc users list Subject: Re: [petsc-users] Bug with 4.6.2 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Fri, Dec 2, 2011 at 9:31 AM, Parker, Andrew (UK Filton) wrote: Hi, I appreciate that this tool may not be supported yet, but I get a compile error with gcc 4.6.2 petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: 'VecType' does not name a type petsc/petsc-3.2-p5/include/petscdm.h:138:47: error: ISO C++ forbids declaration of 'parameter' with no type [-fpermissive] Any thoughts on how I fix this locally? But more generally needs to be fixed. You have to send the entire message. I cannot even see which file is being compiled here. Also, is the GPU stuff turned on? Matt Cheers, Andy ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 2 09:46:59 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Dec 2011 09:46:59 -0600 Subject: [petsc-users] Bug with 4.6.2 In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 09:39, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > Not sure what you mean, it?s been including form a file I?ve used > somewhere, but the c++ type is undefined in that file?.**** > > ** ** > > It doesn?t know what vectype is? > It is defined in petscvec.h, which is included by petscmat.h which is included by petscdm.h. You have to send the entire error message for us to speculate. -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at risoe.dtu.dk Fri Dec 2 09:52:02 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Fri, 2 Dec 2011 16:52:02 +0100 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, December 02, 2011 4:32 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 9:25 AM, Treue, Frederik > wrote: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, December 02, 2011 4:01 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik > wrote: From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, December 02, 2011 1:32 PM To: PETSc users list Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices On Fri, Dec 2, 2011 at 03:32, Treue, Frederik > wrote: OK, but that example seems to assume that you wish to connect only one matrix (the Jacobian) to a DA - I wish to specify many and I think I found this done in ksp ex39, is that example doing anything deprecated or will that work for me, e.g. with the various basic mat routines (matmult, matAXPY etc.) in a multiprocessor setup? What do you mean by wanting many matrices? How do you want to use them? There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can use as many times as you want.` And this was the one I needed. However I have another question: What does DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From experience I now know that the PERIODIC option automagically does the right thing when I'm defining matrices so I can simply specify the same stencil at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? And if so, how is it controlled, ie. How do I specify if I've got Neumann or Dirichlet conditions, and what order extrapolation you want, and so forth? And if not, does it then ONLY make a difference if I'm working with more than on processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE equivalent? GHOSTED adds extra space at the boundary so you can always use the same stencil, but you decide what goes in there. Does this apply to both matrices and vectors, ie. Will the ghost points be considered part of my computational domain or not? The ghost nodes only exist in local vectors, not the global vectors for the solver. OK? So how does one implement boundary conditions? Normally I would include (say) one extra point over the edge of the domain (the ghost point) and then implement the equation (if I start out with Ax=b, A and b known, x desired, and dirichlet boundary conditions) x_G+x_1=2*b_B, where x_G is the unknown at the ghost point, x_1 is the unknown at the first "real" point, and b_B is my dirichlet boundary condition. Thus, I need to a special stencil in the first and last row (ie. The -1 and nx row, with nx internal points) of my matrix, but this leads to memory errors. Is this possible while using GHOSTED? As I understand it, GHOSTED also deals with the MPI communication, so I'd like to retain it, instead of working with NONE. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 2 09:53:45 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 09:53:45 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> Message-ID: This is a problem in Metis/Parmetis. PETSc 3.2 MUST be used with parmetis 3.2.0 Are you perhaps using Metis/Parmetis metis-5.0.2 and parmetis-4.0.2? Did you use --download-parmetis or install it yourself? You should use the --download to the the right one Barry On Dec 2, 2011, at 3:47 AM, Gong Ding wrote: > petsc-3.1-p8 & mumps 4.9 works well. > However, petsc-3.2 p3-p5 & mumps 4.10 seems have memory problem. > > The code occasionally crash on Linux/X64 and alwasy crash on AIX/PPC. > This problem may caused by mumps4.10 or parmetis, but I havn't test. > > The valgrind report: > > ==5139== Invalid read of size 8 > ==5139== at 0x57BE749: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x2269296: __GrowBisection (initpart.c:200) > ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== Address 0x8718408 is 440 bytes inside a block of size 444 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > ==5139== by 0x2269014: __GrowBisection (initpart.c:101) > ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== > ==5139== Invalid read of size 8 > ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x73F8CCCCC: ??? > ==5139== by 0x300000001F: ??? > ==5139== by 0x7FEFF8B4F: ??? > ==5139== by 0x7FEFF8C37: ??? > ==5139== by 0xE326EEF: ??? > ==5139== by 0x216F: ??? > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== Address 0x9288b38 is 0 bytes after a block of size 15,224 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > ==5139== by 0x223DB60: __MlevelNodeBisectionMultiple (ometis.c:402) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== > ==5139== Invalid read of size 8 > ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x21913DD: dmumps_557_ (dmumps_part2.F:2147) > ==5139== by 0x218E243: dmumps_195_ (dmumps_part2.F:1535) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > ==5139== Address 0xf24f6e8 is 3,163,816 bytes inside a block of size 3,163,820 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x5CB3C43: for_allocate (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > ==5139== by 0x5CB3B50: for_alloc_allocatable (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > ==5139== by 0x218CD64: dmumps_195_ (dmumps_part2.F:1072) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > From bsmith at mcs.anl.gov Fri Dec 2 09:59:06 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 09:59:06 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Dec 2, 2011, at 9:52 AM, Treue, Frederik wrote: > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, December 02, 2011 4:32 PM > To: PETSc users list > Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices > > On Fri, Dec 2, 2011 at 9:25 AM, Treue, Frederik wrote: > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, December 02, 2011 4:01 PM > To: PETSc users list > Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices > > On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik wrote: > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown > Sent: Friday, December 02, 2011 1:32 PM > To: PETSc users list > Subject: Re: [petsc-users] newbie question on the parallel allocation of matrices > > On Fri, Dec 2, 2011 at 03:32, Treue, Frederik wrote: > OK, but that example seems to assume that you wish to connect only one matrix (the Jacobian) to a DA ? I wish to specify many and I think I found this done in ksp ex39, is that example doing anything deprecated or will that work for me, e.g. with the various basic mat routines (matmult, matAXPY etc.) in a multiprocessor setup? > > What do you mean by wanting many matrices? How do you want to use them? There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you can use as many times as you want.` > > And this was the one I needed. However I have another question: What does DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From experience I now know that the PERIODIC option automagically does the right thing when I?m defining matrices so I can simply specify the same stencil at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? No, nothing to do with matrices because that extra point is a fixed (Dirichlet) value and the so the derivative of the contribution for that value is zero. > And if so, how is it controlled, ie. How do I specify if I?ve got Neumann or Dirichlet conditions, and what order extrapolation you want, and so forth? And if not, does it then ONLY make a difference if I?m working with more than on processor, ie. If everything is sequential, is DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE equivalent? No, the are not. This option has nothing to do with parallelism. > > GHOSTED adds extra space at the boundary so you can always use the same stencil, but you decide what goes in there. > > Does this apply to both matrices and vectors, ie. Will the ghost points be considered part of my computational domain or not? > > The ghost nodes only exist in local vectors, not the global vectors for the solver. > > OK? So how does one implement boundary conditions? Normally I would include (say) one extra point over the edge of the domain (the ghost point) and then implement the equation (if I start out with Ax=b, A and b known, x desired, and dirichlet boundary conditions) > x_G+x_1=2*b_B, where x_G is the unknown at the ghost point, x_1 is the unknown at the first ?real? point, and b_B is my dirichlet boundary condition. > Thus, I need to a special stencil in the first and last row (ie. The -1 and nx row, with nx internal points) of my matrix, but this leads to memory errors. Is this possible while using GHOSTED? As I understand it, GHOSTED also deals with the MPI communication, so I?d like to retain it, instead of working with NONE. Ghosted has nothing to do with MPI communication! Ghosted vs None is only about the physical boundary, nothing to do with the boundary between domains. Ghosted is only for the (nonlinear) function evaluation. It is not for storage of the Jacobian/matrix. I suggest just using NONE and make your life and understanding easier. You can do Neuman and Dirichlet boundary conditions with none just fine. Barry From knepley at gmail.com Fri Dec 2 10:03:17 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 10:03:17 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Fri, Dec 2, 2011 at 9:59 AM, Barry Smith wrote: > > On Dec 2, 2011, at 9:52 AM, Treue, Frederik wrote: > > > > > > > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > > Sent: Friday, December 02, 2011 4:32 PM > > To: PETSc users list > > Subject: Re: [petsc-users] newbie question on the parallel allocation of > matrices > > > > On Fri, Dec 2, 2011 at 9:25 AM, Treue, Frederik > wrote: > > > > > > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > > Sent: Friday, December 02, 2011 4:01 PM > > To: PETSc users list > > Subject: Re: [petsc-users] newbie question on the parallel allocation of > matrices > > > > On Fri, Dec 2, 2011 at 8:58 AM, Treue, Frederik > wrote: > > > > > > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown > > Sent: Friday, December 02, 2011 1:32 PM > > To: PETSc users list > > Subject: Re: [petsc-users] newbie question on the parallel allocation of > matrices > > > > On Fri, Dec 2, 2011 at 03:32, Treue, Frederik wrote: > > OK, but that example seems to assume that you wish to connect only one > matrix (the Jacobian) to a DA ? I wish to specify many and I think I found > this done in ksp ex39, is that example doing anything deprecated or will > that work for me, e.g. with the various basic mat routines (matmult, > matAXPY etc.) in a multiprocessor setup? > > > > What do you mean by wanting many matrices? How do you want to use them? > There is DMCreateMatrix() (misnamed DMGetMatrix() in petsc-3.2), which you > can use as many times as you want.` > > > > And this was the one I needed. However I have another question: What > does DMDA_BOUNDARY_GHOSTED do, compared to DMDA_BOUNDARY_PERIODIC? From > experience I now know that the PERIODIC option automagically does the right > thing when I?m defining matrices so I can simply specify the same stencil > at all points. Does DMDA_BOUNDARY_GHOSTED do something similar? > > No, nothing to do with matrices because that extra point is a fixed > (Dirichlet) value and the so the derivative of the contribution for that > value is zero. > > > And if so, how is it controlled, ie. How do I specify if I?ve got > Neumann or Dirichlet conditions, and what order extrapolation you want, and > so forth? And if not, does it then ONLY make a difference if I?m working > with more than on processor, ie. If everything is sequential, is > DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE equivalent? > > No, the are not. This option has nothing to do with parallelism. > > > > GHOSTED adds extra space at the boundary so you can always use the same > stencil, but you decide what goes in there. > > > > Does this apply to both matrices and vectors, ie. Will the ghost points > be considered part of my computational domain or not? > > > > The ghost nodes only exist in local vectors, not the global vectors for > the solver. > > > > OK? So how does one implement boundary conditions? Normally I would > include (say) one extra point over the edge of the domain (the ghost point) > and then implement the equation (if I start out with Ax=b, A and b known, x > desired, and dirichlet boundary conditions) > > x_G+x_1=2*b_B, where x_G is the unknown at the ghost point, x_1 is the > unknown at the first ?real? point, and b_B is my dirichlet boundary > condition. > > Thus, I need to a special stencil in the first and last row (ie. The -1 > and nx row, with nx internal points) of my matrix, but this leads to memory > errors. Is this possible while using GHOSTED? As I understand it, GHOSTED > also deals with the MPI communication, so I?d like to retain it, instead of > working with NONE. > > Ghosted has nothing to do with MPI communication! Ghosted vs None is > only about the physical boundary, nothing to do with the boundary between > domains. > > Ghosted is only for the (nonlinear) function evaluation. It is not for > storage of the Jacobian/matrix. > > I suggest just using NONE and make your life and understanding easier. > You can do Neuman and Dirichlet boundary conditions with none just fine. You can see us apply Dirichlet conditions with NONE in SNES ex5 Matt > > Barry > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Dec 2 10:05:00 2011 From: agrayver at gfz-potsdam.de (=?utf-8?B?YWdyYXl2ZXJAZ2Z6LXBvdHNkYW0uZGU=?=) Date: Fri, 02 Dec 2011 17:05:00 +0100 Subject: [petsc-users] =?utf-8?q?MatMumpsSetIcntl_from_Fortran?= Message-ID: Hong, Thanks for testing. Next days I will be away, but next week will report you on problem. Am I right that mumps in dev version uses latest parmetis (which means it was patched)? Regards, Alexander ----- Reply message ----- From: "Hong Zhang" To: "PETSc users list" Subject: [petsc-users] MatMumpsSetIcntl from Fortran Date: Fri, Dec 2, 2011 5:20 am Alexander Using blocks below and petsc-dev/src/ksp/ksp/examples/tutorials/ex2f.F, I wrote a Fortran test petsc-dev/src/ksp/ksp/examples/tutorials/ex57f.F (attached). I noticed some sloppiness in our interface that fails to take the procedural options, but I did not see crash as you experienced. I updated petsc-mumps interface in petsc-dev. Can you switch to petsc-dev which interfaces with MUMPS_4.10.0 and see if it still crash in Fortran? See http://www.mcs.anl.gov/petsc/developers/index.html on how to get petsc-dev. Let us know if you still get problems, Hong > > This should be enough to reproduce the problem: > > ?call KSPCreate(PETSC_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) > ?call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr);CHKERRQ(ierr) > ?call KSPSetType(ksp,KSPPREONLY,ierr);CHKERRQ(ierr) > ?call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) > ?call PCSetType(pc,PCCHOLESKY,ierr);CHKERRQ(ierr) > ?call PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS,ierr);CHKERRQ(ierr) > ?call PCFactorSetUpMatSolverPackage(pc,ierr);CHKERRQ(ierr) > ?call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) > ?call KSPSetFromOptions(ksp,ierr);CHKERRQ(ierr) > ?icntl = 2; ival = 0; > ?call MatMumpsSetIcntl(F,icntl,ival,ierr); > > Regards, > Alexander > > > On 30.11.2011 23:43, Hong Zhang wrote: >>> >>> Sorry if I wasn't clear. ?"c" example works fine, I know, what I meant is >>> that if you try to implement lines 150-171 from it on FORTRAN you will >>> see >>> the problem I reported. >>> If you need particularly my FORTRAN code I can send it tomorrow. >> >> This would save my time :-) >> Appreciate. >> >> Hong >>> >>> Regards, >>> Alexander >>> >>> >>> ----- Reply message ----- >>> From: "Hong Zhang" >>> To: "PETSc users list" >>> Subject: [petsc-users] MatMumpsSetIcntl from Fortran >>> Date: Wed, Nov 30, 2011 9:37 pm >>> >>> >>> Alexander : >>> >>>> I just rewrote code concerning mumps from this example (lines 150-170): >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c.html >>> >>> Where is your Fortran code? ex52.c works fine. >>> Hong >>> >>>> On 30.11.2011 17:40, Hong Zhang wrote: >>>>> >>>>> Alexander: >>>>> >>>>>> Has anybody tried to use MatMumpsSetIcntl from Fortran? >>>>> >>>>> We are not aware of it. >>>>> >>>>>> Because when I try to call it I fall into infinite recursion in >>>>>> function: >>>>> >>>>> Can you give me a short Fortran code that repeats this error for >>>>> investigating? >>>>> Meanwhile, you can use runtime option '-mat_mumps_icntl_xxx<>' to get >>>>> your code run. >>>>> >>>>> Hong >>>>>> >>>>>> ?PetscErrorCode MatMumpsSetIcntl(Mat F,PetscInt icntl,PetscInt ival) >>>>>> ?{ >>>>>> ? PetscErrorCode ierr; >>>>>> >>>>>> ? PetscFunctionBegin; >>>>>> ? PetscValidLogicalCollectiveInt(F,icntl,2); >>>>>> ? PetscValidLogicalCollectiveInt(F,ival,3); >>>>>> ? ierr = >>>>>> >>>>>> >>>>>> >>>>>> PetscTryMethod(F,"MatMumpsSetIcntl_C",(Mat,PetscInt,PetscInt),(F,icntl,ival));CHKERRQ(ierr); >>>>>> ? PetscFunctionReturn(0); >>>>>> ?} >>>>>> >>>>>> At the moment when program crashes call stack looks like: >>>>>> >>>>>> ? ? __libc_memalign, >>>>>> FP=7fff342ca010 >>>>>> PetscMallocAlign, >>>>>> ?FP=7fff342ca080 >>>>>> PetscTrMallocDefault, >>>>>> ?FP=7fff342ca180 >>>>>> PetscStrallocpy, >>>>>> FP=7fff342ca230 >>>>>> PetscFListGetPathAndFunction, >>>>>> ?FP=7fff342cb2e0 >>>>>> PetscFListFind, >>>>>> ?FP=7fff342cb520 >>>>>> PetscObjectQueryFunction_Petsc, >>>>>> ?FP=7fff342cb590 >>>>>> PetscObjectQueryFunction, >>>>>> ?FP=7fff342cb620 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb720 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb820 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cb920 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cba20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbb20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbc20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbd20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbe20 >>>>>> MatMumpsSetIcntl, >>>>>> ?FP=7fff342cbf20 >>>>>> ... (Hundreds of MatMumpsSetIcntl) ... >>>>>> >>>>>> What can I do about that? >>>>>> >>>>>> Regards, >>>>>> Alexander >>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Dec 2 12:30:22 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 2 Dec 2011 12:30:22 -0600 Subject: [petsc-users] MatMumpsSetIcntl from Fortran In-Reply-To: References: Message-ID: > Am I right that mumps in dev version uses latest parmetis (which means it > was patched)? Yes. Hong > > > Regards, > Alexander > > ----- Reply message ----- > From: "Hong Zhang" > To: "PETSc users list" > Subject: [petsc-users] MatMumpsSetIcntl from Fortran > Date: Fri, Dec 2, 2011 5:20 am > > > Alexander > Using blocks below and > petsc-dev/src/ksp/ksp/examples/tutorials/ex2f.F, I wrote a Fortran test > petsc-dev/src/ksp/ksp/examples/tutorials/ex57f.F (attached). > > I noticed some sloppiness in our interface that fails to take the > procedural options, > but I did not see crash as you experienced. I updated petsc-mumps > interface in petsc-dev. > Can you switch to petsc-dev which interfaces with MUMPS_4.10.0 and see > if it still crash in Fortran? > See http://www.mcs.anl.gov/petsc/developers/index.html on how to get > petsc-dev. > > Let us know if you still get problems, > > Hong > >> >> This should be enough to reproduce the problem: >> >> ?call KSPCreate(PETSC_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >> ?call >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr);CHKERRQ(ierr) >> ?call KSPSetType(ksp,KSPPREONLY,ierr);CHKERRQ(ierr) >> ?call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >> ?call PCSetType(pc,PCCHOLESKY,ierr);CHKERRQ(ierr) >> ?call PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS,ierr);CHKERRQ(ierr) >> ?call PCFactorSetUpMatSolverPackage(pc,ierr);CHKERRQ(ierr) >> ?call PCFactorGetMatrix(pc,F,ierr);CHKERRQ(ierr) >> ?call KSPSetFromOptions(ksp,ierr);CHKERRQ(ierr) >> ?icntl = 2; ival = 0; >> ?call MatMumpsSetIcntl(F,icntl,ival,ierr); >> >> Regards, >> Alexander >> >> >> On 30.11.2011 23:43, Hong Zhang wrote: >>>> >>>> Sorry if I wasn't clear. ?"c" example works fine, I know, what I meant >>>> is >>>> that if you try to implement lines 150-171 from it on FORTRAN you will >>>> see >>>> the problem I reported. >>>> If you need particularly my FORTRAN code I can send it tomorrow. >>> >>> This would save my time :-) >>> Appreciate. >>> >>> Hong >>>> >>>> Regards, >>>> Alexander >>>> >>>> >>>> ----- Reply message ----- >>>> From: "Hong Zhang" >>>> To: "PETSc users list" >>>> Subject: [petsc-users] MatMumpsSetIcntl from Fortran >>>> Date: Wed, Nov 30, 2011 9:37 pm >>>> >>>> >>>> Alexander : >>>> >>>>> I just rewrote code concerning mumps from this example (lines 150-170): >>>>> >>>>> >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c.html >>>> >>>> Where is your Fortran code? ex52.c works fine. >>>> Hong >>>> >>>>> On 30.11.2011 17:40, Hong Zhang wrote: >>>>>> >>>>>> Alexander: >>>>>> >>>>>>> Has anybody tried to use MatMumpsSetIcntl from Fortran? >>>>>> >>>>>> We are not aware of it. >>>>>> >>>>>>> Because when I try to call it I fall into infinite recursion in >>>>>>> function: >>>>>> >>>>>> Can you give me a short Fortran code that repeats this error for >>>>>> investigating? >>>>>> Meanwhile, you can use runtime option '-mat_mumps_icntl_xxx<>' to get >>>>>> your code run. >>>>>> >>>>>> Hong >>>>>>> >>>>>>> ?PetscErrorCode MatMumpsSetIcntl(Mat F,PetscInt icntl,PetscInt ival) >>>>>>> ?{ >>>>>>> ? PetscErrorCode ierr; >>>>>>> >>>>>>> ? PetscFunctionBegin; >>>>>>> ? PetscValidLogicalCollectiveInt(F,icntl,2); >>>>>>> ? PetscValidLogicalCollectiveInt(F,ival,3); >>>>>>> ? ierr = >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> PetscTryMethod(F,"MatMumpsSetIcntl_C",(Mat,PetscInt,PetscInt),(F,icntl,ival));CHKERRQ(ierr); >>>>>>> ? PetscFunctionReturn(0); >>>>>>> ?} >>>>>>> >>>>>>> At the moment when program crashes call stack looks like: >>>>>>> >>>>>>> ? ? __libc_memalign, >>>>>>> FP=7fff342ca010 >>>>>>> PetscMallocAlign, >>>>>>> ?FP=7fff342ca080 >>>>>>> PetscTrMallocDefault, >>>>>>> ?FP=7fff342ca180 >>>>>>> PetscStrallocpy, >>>>>>> FP=7fff342ca230 >>>>>>> PetscFListGetPathAndFunction, >>>>>>> ?FP=7fff342cb2e0 >>>>>>> PetscFListFind, >>>>>>> ?FP=7fff342cb520 >>>>>>> PetscObjectQueryFunction_Petsc, >>>>>>> ?FP=7fff342cb590 >>>>>>> PetscObjectQueryFunction, >>>>>>> ?FP=7fff342cb620 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cb720 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cb820 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cb920 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cba20 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cbb20 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cbc20 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cbd20 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cbe20 >>>>>>> MatMumpsSetIcntl, >>>>>>> ?FP=7fff342cbf20 >>>>>>> ... (Hundreds of MatMumpsSetIcntl) ... >>>>>>> >>>>>>> What can I do about that? >>>>>>> >>>>>>> Regards, >>>>>>> Alexander >>>>> >>>>> >> > From tim.gallagher at gatech.edu Fri Dec 2 13:08:43 2011 From: tim.gallagher at gatech.edu (Tim Gallagher) Date: Fri, 02 Dec 2011 14:08:43 -0500 (EST) Subject: [petsc-users] DMDA Question In-Reply-To: <620af1e0-2f1f-4cbf-ae64-c35522d388b3@mail2.gatech.edu> Message-ID: Hi, I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? Any advice would be appreciated, Tim From bsmith at mcs.anl.gov Fri Dec 2 13:12:54 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 13:12:54 -0600 Subject: [petsc-users] DMDA Question In-Reply-To: References: Message-ID: <67C962A8-AD29-4474-A5BE-89244044834B@mcs.anl.gov> Tim, Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. Do I understand you want two DMDA 1) one that has locations for cell vertices (or face centers)? 2) and one that locations for cell centers? Once we understand what you want we may be able to make a suggestion. Barry On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > Hi, > > I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: > > Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. > > I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? > > Any advice would be appreciated, > > Tim From tim.gallagher at gatech.edu Fri Dec 2 13:19:47 2011 From: tim.gallagher at gatech.edu (Tim Gallagher) Date: Fri, 02 Dec 2011 14:19:47 -0500 (EST) Subject: [petsc-users] DMDA Question In-Reply-To: <67C962A8-AD29-4474-A5BE-89244044834B@mcs.anl.gov> Message-ID: Yes, I would like two DMDA's, one for the nodes and the other for the cell centers. The grid is structured, but is non-uniform. I think I want that anyway, is that the best structure for a case like this? Thanks, Tim ----- Original Message ----- From: "Barry Smith" To: gtg085x at mail.gatech.edu, "PETSc users list" Sent: Friday, December 2, 2011 2:12:54 PM Subject: Re: [petsc-users] DMDA Question Tim, Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. Do I understand you want two DMDA 1) one that has locations for cell vertices (or face centers)? 2) and one that locations for cell centers? Once we understand what you want we may be able to make a suggestion. Barry On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > Hi, > > I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: > > Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. > > I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? > > Any advice would be appreciated, > > Tim From ecoon at lanl.gov Fri Dec 2 14:04:12 2011 From: ecoon at lanl.gov (Ethan Coon) Date: Fri, 02 Dec 2011 13:04:12 -0700 Subject: [petsc-users] DMDA Question In-Reply-To: References: Message-ID: <1322856252.12219.19.camel@echo.lanl.gov> On Fri, 2011-12-02 at 14:08 -0500, Tim Gallagher wrote: > Hi, > > I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: > > Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. > > I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. There isn't really a more elegant way out there (as far as I know). The usual approach for this is to do as you have suggested, with the following alteration -- information about a cell should only be calculated by the process that owns the cell. So the approach would be: - set the coordinates of the DA as normal via DMDASetCoordinates() - call DMDAGetCoordinateDA(), get a local vec based upon that DA, and call DMGlobalToLocalBegin/End() to scatter from the (global) coordinate vector into your new local vec. - then the cell 4-5 center is calculated by process 0 (who owns node 4 and has node 5 as a ghost), set the cell coordinates - reverse the process if needed, calling DMDAGetCoordinateDA() on the cell DA, and DMGlobalToLocalBegin/End() to get cell "4-5"'s coordinates on process 1 (where it is a ghost cell). Ethan > Has anybody used a DMDA to create a dual DMDA? > > Any advice would be appreciated, > > Tim -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------ From bsmith at mcs.anl.gov Fri Dec 2 16:14:59 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 16:14:59 -0600 Subject: [petsc-users] DMDA Question In-Reply-To: References: Message-ID: On Dec 2, 2011, at 1:19 PM, Tim Gallagher wrote: > Yes, I would like two DMDA's, one for the nodes and the other for the cell centers. The grid is structured, but is non-uniform. Do you have degrees of freedom on the "nodes" and cell centers, or are the only degrees of freedom on the cell centers and the nodes are used for intermediate computations? That is does your nonlinear system involve unknowns on both the nodes and centers? Barry > > I think I want that anyway, is that the best structure for a case like this? > > Thanks, > > Tim > > ----- Original Message ----- > From: "Barry Smith" > To: gtg085x at mail.gatech.edu, "PETSc users list" > Sent: Friday, December 2, 2011 2:12:54 PM > Subject: Re: [petsc-users] DMDA Question > > > Tim, > > Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. > > Do I understand you want two DMDA > > 1) one that has locations for cell vertices (or face centers)? > > 2) and one that locations for cell centers? > > Once we understand what you want we may be able to make a suggestion. > > Barry > > > On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > >> Hi, >> >> I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: >> >> Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. >> >> I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? >> >> Any advice would be appreciated, >> >> Tim > From tim.gallagher at gatech.edu Fri Dec 2 16:52:08 2011 From: tim.gallagher at gatech.edu (Tim Gallagher) Date: Fri, 02 Dec 2011 17:52:08 -0500 (EST) Subject: [petsc-users] DMDA Question In-Reply-To: Message-ID: <495cb3b5-7afa-4962-b928-e7c5a0a1fe6b@mail2.gatech.edu> There are only unknowns at the cell centers. The coordinates of the nodes may be used during computation though. I'd have to think if it's possible to specify the nodes as dof's of each cell center... but that may duplicate a lot of data and be sort of convoluted to understand what was going on in the code. Tim ----- Original Message ----- From: "Barry Smith" To: gtg085x at mail.gatech.edu, "PETSc users list" Sent: Friday, December 2, 2011 5:14:59 PM Subject: Re: [petsc-users] DMDA Question On Dec 2, 2011, at 1:19 PM, Tim Gallagher wrote: > Yes, I would like two DMDA's, one for the nodes and the other for the cell centers. The grid is structured, but is non-uniform. Do you have degrees of freedom on the "nodes" and cell centers, or are the only degrees of freedom on the cell centers and the nodes are used for intermediate computations? That is does your nonlinear system involve unknowns on both the nodes and centers? Barry > > I think I want that anyway, is that the best structure for a case like this? > > Thanks, > > Tim > > ----- Original Message ----- > From: "Barry Smith" > To: gtg085x at mail.gatech.edu, "PETSc users list" > Sent: Friday, December 2, 2011 2:12:54 PM > Subject: Re: [petsc-users] DMDA Question > > > Tim, > > Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. > > Do I understand you want two DMDA > > 1) one that has locations for cell vertices (or face centers)? > > 2) and one that locations for cell centers? > > Once we understand what you want we may be able to make a suggestion. > > Barry > > > On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > >> Hi, >> >> I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: >> >> Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. >> >> I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? >> >> Any advice would be appreciated, >> >> Tim > From wolfshow at gmail.com Fri Dec 2 19:24:10 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Sat, 3 Dec 2011 09:24:10 +0800 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file Message-ID: <201112030924087476065@gmail.com> >Date: Fri, 2 Dec 2011 08:07:54 -0600 >From: Jed Brown >Subject: Re: [petsc-users] How to output the values of a parallel >PETSc vector to a ASCII file >To: PETSc users list >Message-ID: >Content-Type: text/plain; charset="utf-8" >On Fri, Dec 2, 2011 at 08:04, Feng-Chao Wang wrote: > ** > Dear all, > > I intend to postprocess the solution of a PDE (stored in a > parallel PETSc vector ) using tecplot. Thus I want to output the values of > the parallel vector as well as other data (such as the grid information) to > a ASCII file. > > When I deal this with > > VecGetArray(U, &u); > > for (i=0; i { > PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); > } > > If more than 1 processor are used, I found that only part of the value of > u was output correctly (that stored on the first processor). > I found in the manual page of "PetscViewerASCIIPrintf" that "Not > Collective, but only first processor in set has any effect" > >If you can afford the memory: >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html Thank you very much! I tried the Scatter method. I found that even I scatter the parallel vector to a sequential vector, when I want to output this sequential vector to the file, I still can not use "PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]);", because the vector can not be directly accessed for output? Thus I used VecGetArray(U, &u); and then PetscViewerASCIIPrintf. I still got part of the sequential vector, because the array u is stored on all the processor? >Slower: >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscSynchronizedPrintf.html I tried the PetscSynchronizedPrintf and got the same result. As a beginner of PETSc, I wonder if I can output the value of a vector U to the ASCII file using Printf("%f", U[i])? If so, I can get the result I want. Or, I must use VecGetArray(U, &u); before I want to access the data stored in the vector? Thank you! >Binary output can be written in parallel using >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecView.html Yes, the VecView can output the correct data stored in the parallel vector to a ASCII file. However, I want to output not only the vector value but also the grid information according to the tecplot file format, for example, there are three column of data in one file: x[0] y[0] vec_value[0] x[1] y[1] vec_value[1] x[2] y[2] vec_value[2] x[3] y[3] vec_value[3] x[4] y[4] vec_value[4] x[5] y[5] vec_value[5] ... ... ... ****************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 2 19:29:18 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 2 Dec 2011 19:29:18 -0600 Subject: [petsc-users] How to output the values of a parallel PETSc vector to a ASCII file In-Reply-To: <201112030924087476065@gmail.com> References: <201112030924087476065@gmail.com> Message-ID: 2011/12/2 Feng-Chao Wang > ** > >Date: Fri, 2 Dec 2011 08:07:54 -0600 > >From: Jed Brown > >Subject: Re: [petsc-users] How to output the values of a parallel > >PETSc vector to a ASCII file > >To: PETSc users list > >Message-ID: > > >Content-Type: text/plain; charset="utf-8" > >On Fri, Dec 2, 2011 at 08:04, Feng-Chao Wang wrote: > > ** > > Dear all, > > > > I intend to postprocess the solution of a PDE (stored in a > > > parallel PETSc vector ) using tecplot. Thus I want to output the values of > > > the parallel vector as well as other data (such as the grid information) to > > a ASCII file. > > > > When I deal this with > > > > VecGetArray(U, &u); > > > > for (i=0; i > { > > PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); > > } > > > > If more than 1 processor are used, I found that only part of the value of > > u was output correctly (that stored on the first processor). > > I found in the manual page of "PetscViewerASCIIPrintf" that "Not > > Collective, but only first processor in set has any effect" > > > >If you can afford the memory: > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html > > Thank you very much! > I tried the Scatter method. I found that even I scatter the parallel > vector to a sequential vector, when I want to output this sequential vector > to the file, I still can not use > "PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]);", because the > vector can not be directly accessed for output? > Yes, it is an object. You really must read the manual chapter on Vec objects. > > Thus I used VecGetArray(U, &u); and then PetscViewerASCIIPrintf. I still > got part of the sequential vector, because the array u is stored on all the > processor? > This is incorrect. The entire array resides on process zero Matt > >Slower: > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscSynchronizedPrintf.html > > I tried the PetscSynchronizedPrintf and got the same result. As a beginner > of PETSc, I wonder if I can output the value of a vector U to the ASCII > file using Printf("%f", U[i])? > If so, I can get the result I want. > > Or, I must use VecGetArray(U, &u); before I want to access the data stored > in the vector? > > Thank you! > > >Binary output can be written in parallel using > >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecView.html > > Yes, the VecView can output the correct data stored in the parallel vector > to a ASCII file. However, I want to output not only the vector value but > also the grid information according to the tecplot file format, for > example, there are three column of data in one file: > > x[0] y[0] vec_value[0] > x[1] y[1] vec_value[1] > x[2] y[2] vec_value[2] > x[3] y[3] vec_value[3] > x[4] y[4] vec_value[4] > x[5] y[5] vec_value[5] > ... ... ... > > > > ****************************************** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Fri Dec 2 19:45:42 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Sat, 3 Dec 2011 09:45:42 +0800 Subject: [petsc-users] petsc-users Digest, Vol 36, Issue 13 Message-ID: <201112030945408105685@gmail.com> >Message: 8 >Date: Sat, 3 Dec 2011 09:24:10 +0800 >From: "Feng-Chao Wang" >Subject: Re: [petsc-users] How to output the values of a parallel >PETSc vector to a ASCII file >To: "petsc-users" >Message-ID: <201112030924087476065 at gmail.com> >Content-Type: text/plain; charset="gb2312" >Date: Fri, 2 Dec 2011 08:07:54 -0600 >From: Jed Brown >Subject: Re: [petsc-users] How to output the values of a parallel >PETSc vector to a ASCII file >To: PETSc users list >Message-ID: >Content-Type: text/plain; charset="utf-8" >On Fri, Dec 2, 2011 at 08:04, Feng-Chao Wang wrote: > ** > Dear all, > > I intend to postprocess the solution of a PDE (stored in a > parallel PETSc vector ) using tecplot. Thus I want to output the values of > the parallel vector as well as other data (such as the grid information) to > a ASCII file. > > When I deal this with > > VecGetArray(U, &u); > > for (i=0; i { > PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]); > } > > If more than 1 processor are used, I found that only part of the value of > u was output correctly (that stored on the first processor). > I found in the manual page of "PetscViewerASCIIPrintf" that "Not > Collective, but only first processor in set has any effect" > >If you can afford the memory: >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html Thank you very much! I tried the Scatter method. I found that even I scatter the parallel vector to a sequential vector, when I want to output this sequential vector to the file, I still can not use "PetscViewerASCIIPrintf(viewer,"%F %F %F", x, y, u[i]);", because the vector can not be directly accessed for output? Thus I used VecGetArray(U, &u); and then PetscViewerASCIIPrintf. I still got part of the sequential vector, because the array u is stored on all the processor? Sorry, I think I made a mistake. After I scatter the parallel vector to a sequential vector and then use VecGetArray(U, &u); and then output the u[i] to the file. I got the correct output. Thank Jed Brown and Matthew Knepley very much for the reply. >Slower: >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscSynchronizedPrintf.html I tried the PetscSynchronizedPrintf and got the same result. As a beginner of PETSc, I wonder if I can output the value of a vector U to the ASCII file using Printf("%f", U[i])? If so, I can get the result I want. Or, I must use VecGetArray(U, &u); before I want to access the data stored in the vector? Thank you! >Binary output can be written in parallel using >http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecView.html Yes, the VecView can output the correct data stored in the parallel vector to a ASCII file. However, I want to output not only the vector value but also the grid information according to the tecplot file format, for example, there are three column of data in one file: x[0] y[0] vec_value[0] x[1] y[1] vec_value[1] x[2] y[2] vec_value[2] x[3] y[3] vec_value[3] x[4] y[4] vec_value[4] x[5] y[5] vec_value[5] ... ... ... ****************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 36, Issue 13 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 2 22:43:23 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 22:43:23 -0600 Subject: [petsc-users] DMDA Question In-Reply-To: <495cb3b5-7afa-4962-b928-e7c5a0a1fe6b@mail2.gatech.edu> References: <495cb3b5-7afa-4962-b928-e7c5a0a1fe6b@mail2.gatech.edu> Message-ID: <5AA0BBAE-30F4-4FCE-B8CB-934470AEA338@mcs.anl.gov> On Dec 2, 2011, at 4:52 PM, Tim Gallagher wrote: > There are only unknowns at the cell centers. The coordinates of the nodes may be used during computation though. > > I'd have to think if it's possible to specify the nodes as dof's of each cell center... but that may duplicate a lot of data and be sort of convoluted to understand what was going on in the code. Tim, So the issue is that there is no way to insure that two different DA's with different dimensions get the same parallel layout so the coordinate information you need is always easily available with the cell data. Someday we hope to have support for staggered grids with DAs and that would make your task much much easier. Sorry it doesn't exist now. Here is what I would do now because it is so easy to do and the parallelism becomes trivial. Create 2 DMDAs both of the same size but have the first have a dof equal to your number of equations and have the second one have a dof of 8 (for 2d problems) and 16 for 3d problems. Both DMDA's will have the same parallel layout and the ghosted versions while also have the same layout. Now store the x,y coordinates of all four vertices of each cell (in 2d) into the 8 locations of a vector associated with the second DMDA. Now you will access coordinates of cell vertices with the same indexing that you use for the equations (after calling DMDAVecGetArray()). The code will be very clean. The only drawback is the extra storage of the coordinates but frankly that is likely to be a relatively small amount of additional memory compared to everything else need in the simulation. Note: do not use the DMDACoordinates interfaces that is not appropriate for what you need, just keep the coordinates as I described above. Barry > > Tim > > ----- Original Message ----- > From: "Barry Smith" > To: gtg085x at mail.gatech.edu, "PETSc users list" > Sent: Friday, December 2, 2011 5:14:59 PM > Subject: Re: [petsc-users] DMDA Question > > > On Dec 2, 2011, at 1:19 PM, Tim Gallagher wrote: > >> Yes, I would like two DMDA's, one for the nodes and the other for the cell centers. The grid is structured, but is non-uniform. > > Do you have degrees of freedom on the "nodes" and cell centers, or are the only degrees of freedom on the cell centers and the nodes are used for intermediate computations? That is does your nonlinear system involve unknowns on both the nodes and centers? > > Barry > > >> >> I think I want that anyway, is that the best structure for a case like this? >> >> Thanks, >> >> Tim >> >> ----- Original Message ----- >> From: "Barry Smith" >> To: gtg085x at mail.gatech.edu, "PETSc users list" >> Sent: Friday, December 2, 2011 2:12:54 PM >> Subject: Re: [petsc-users] DMDA Question >> >> >> Tim, >> >> Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. >> >> Do I understand you want two DMDA >> >> 1) one that has locations for cell vertices (or face centers)? >> >> 2) and one that locations for cell centers? >> >> Once we understand what you want we may be able to make a suggestion. >> >> Barry >> >> >> On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: >> >>> Hi, >>> >>> I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: >>> >>> Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. >>> >>> I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? >>> >>> Any advice would be appreciated, >>> >>> Tim >> > From gshy2014 at gmail.com Fri Dec 2 22:57:14 2011 From: gshy2014 at gmail.com (Shiyuan) Date: Fri, 2 Dec 2011 22:57:14 -0600 Subject: [petsc-users] KSP with MatNullSpace In-Reply-To: <9F0F599F-F0FD-4E60-B601-0C422839CF18@mcs.anl.gov> References: <9F0F599F-F0FD-4E60-B601-0C422839CF18@mcs.anl.gov> Message-ID: I notice that when the vector indicating the nullspace is not very accurate (norm(A*phi) very small but not zero), KSP (CG) will return KSP_INDEFINITE_PC. When b is very close to zero in AX=b, KSP (CG) has the same error. Is it controlled by some tolerance? How can I set the tolerance? Thanks. Shiyuan On Wed, Nov 23, 2011 at 5:00 PM, Barry Smith wrote: > > On Nov 23, 2011, at 1:31 PM, Shiyuan wrote: > > > I've check it in Matlab. the matrix A is symmetric and has no negative > eigenvalues( only a zero eigen value). I've also checked that the nullspace > is correct( norm(A*phi)<1e-11); > > Hmm. You can run with -ksp_view_binary and email to > petsc-maint at mcs.anl.gov the resulting file called binaryoutput. > > Barry > > > > Thanks. > > > > On Wed, Nov 23, 2011 at 1:26 PM, Barry Smith wrote: > > > > Print out the matrix for a small grid then check in Matlab that it is > symmetric and has no negative eigenvalues. > > > > Barry > > > > > > On Nov 23, 2011, at 9:50 AM, Shiyuan wrote: > > > > > Hi, > > > I want to solve a singular system with a known nullspace. However, > I the KSP solve diverges with KSP_INDEFINTE_PC even if I disable the > preconditioning by PCNONE. > > > this is how I setup the system. What did I do wrong? Any possible > causes? Thanks. > > > > > > > ierr=MatNullSpaceCreate(PETSC_COMM_SELF,PETSC_FALSE,1,&phi,&nsp);CHKERRV(ierr); > > > ierr=KSPCreate(PETSC_COMM_SELF,&ksp);CHKERRV(ierr); > > > > ierr=KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRV(ierr); > > > ierr=KSPSetNullSpace(ksp,nsp);CHKERRV(ierr); > > > ierr=KSPSetType(ksp,KSPCG);CHKERRV(ierr); > > > ierr=KSPGetPC(ksp,&prec);CHKERRV(ierr); > > > ierr=PCSetType(prec,PCNONE);CHKERRV(ierr); > > > ierr=KSPSetTolerances(ksp,1e-5,1e-20,1e5,10000);CHKERRV(ierr); > > > ierr=KSPSetFromOptions(ksp);CHKERRV(ierr); > > > ierr=KSPSetUp(ksp);CHKERRV(ierr); > > > > > > Shiyuan > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 2 23:17:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 2 Dec 2011 23:17:21 -0600 Subject: [petsc-users] KSP with MatNullSpace In-Reply-To: References: <9F0F599F-F0FD-4E60-B601-0C422839CF18@mcs.anl.gov> Message-ID: On Dec 2, 2011, at 10:57 PM, Shiyuan wrote: > I notice that when the vector indicating the nullspace is not very accurate (norm(A*phi) very small but not zero), KSP (CG) will return KSP_INDEFINITE_PC. When b is very close to zero in AX=b, KSP (CG) has the same error. Is it controlled by some tolerance? How can I set the tolerance? No it is not controlled by some tolerance, it cannot be controlled by some tolerance because the indication of indefiniteness is determined by } else if ((i > 0) && (beta*betaold < 0.0)) { ksp->reason = KSP_DIVERGED_INDEFINITE_PC; ierr = PetscInfo(ksp,"diverging due to indefinite preconditioner\n");CHKERRQ(ierr); break; as soon as beta*betaold is negative the entire CG algorithm breaks down and there is no way to recover. Barry > Thanks. > > Shiyuan > > On Wed, Nov 23, 2011 at 5:00 PM, Barry Smith wrote: > > On Nov 23, 2011, at 1:31 PM, Shiyuan wrote: > > > I've check it in Matlab. the matrix A is symmetric and has no negative eigenvalues( only a zero eigen value). I've also checked that the nullspace is correct( norm(A*phi)<1e-11); > > Hmm. You can run with -ksp_view_binary and email to petsc-maint at mcs.anl.gov the resulting file called binaryoutput. > > Barry > > > > Thanks. > > > > On Wed, Nov 23, 2011 at 1:26 PM, Barry Smith wrote: > > > > Print out the matrix for a small grid then check in Matlab that it is symmetric and has no negative eigenvalues. > > > > Barry > > > > > > On Nov 23, 2011, at 9:50 AM, Shiyuan wrote: > > > > > Hi, > > > I want to solve a singular system with a known nullspace. However, I the KSP solve diverges with KSP_INDEFINTE_PC even if I disable the preconditioning by PCNONE. > > > this is how I setup the system. What did I do wrong? Any possible causes? Thanks. > > > > > > ierr=MatNullSpaceCreate(PETSC_COMM_SELF,PETSC_FALSE,1,&phi,&nsp);CHKERRV(ierr); > > > ierr=KSPCreate(PETSC_COMM_SELF,&ksp);CHKERRV(ierr); > > > ierr=KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRV(ierr); > > > ierr=KSPSetNullSpace(ksp,nsp);CHKERRV(ierr); > > > ierr=KSPSetType(ksp,KSPCG);CHKERRV(ierr); > > > ierr=KSPGetPC(ksp,&prec);CHKERRV(ierr); > > > ierr=PCSetType(prec,PCNONE);CHKERRV(ierr); > > > ierr=KSPSetTolerances(ksp,1e-5,1e-20,1e5,10000);CHKERRV(ierr); > > > ierr=KSPSetFromOptions(ksp);CHKERRV(ierr); > > > ierr=KSPSetUp(ksp);CHKERRV(ierr); > > > > > > Shiyuan > > > > > > From gdiso at ustc.edu Sat Dec 3 06:18:02 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sat, 3 Dec 2011 20:18:02 +0800 Subject: [petsc-users] many valgrind error of petsc string functions References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> Message-ID: <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> In the function of PetscOptionsFindPair_Private (options.c line 1098), there are still some code with the same problem. ----- Original Message ----- From: "Jed Brown" To: "PETSc users list" Sent: Friday, December 02, 2011 8:28 PM Subject: Re: [petsc-users] many valgrind error of petsc string functions > 2011/12/2 Gong Ding > >> Valgrind reported many errors about string processing functionss of petsc. >> These error seems harmless, but really ugly. >> > > Agreed, I think you have a really long path (but still much shorter than > the system path length). If this is the problem, then the fix I just pushed > to petsc-3.2 will fix it. > > http://petsc.cs.iit.edu/petsc/releases/petsc-3.2/rev/36bd87d582e9 > > You can pull this with mercurial or apply manually. Let me know if this > works. > From gdiso at ustc.edu Sat Dec 3 06:21:06 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sat, 3 Dec 2011 20:21:06 +0800 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> Message-ID: <5B58FB78E1E048EF803121C203D9A793@cogendaeda> I indid use --download-parmetis and --download-mumps. The parmetis version is 3.2.0-p1. Maybe mumps is not compatable with this version? This is a problem in Metis/Parmetis. PETSc 3.2 MUST be used with parmetis 3.2.0 Are you perhaps using Metis/Parmetis metis-5.0.2 and parmetis-4.0.2? Did you use --download-parmetis or install it yourself? You should use the --download to the the right one Barry On Dec 2, 2011, at 3:47 AM, Gong Ding wrote: > petsc-3.1-p8 & mumps 4.9 works well. > However, petsc-3.2 p3-p5 & mumps 4.10 seems have memory problem. > > The code occasionally crash on Linux/X64 and alwasy crash on AIX/PPC. > This problem may caused by mumps4.10 or parmetis, but I havn't test. > > The valgrind report: > > ==5139== Invalid read of size 8 > ==5139== at 0x57BE749: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x2269296: __GrowBisection (initpart.c:200) > ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== Address 0x8718408 is 440 bytes inside a block of size 444 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > ==5139== by 0x2269014: __GrowBisection (initpart.c:101) > ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== > ==5139== Invalid read of size 8 > ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x73F8CCCCC: ??? > ==5139== by 0x300000001F: ??? > ==5139== by 0x7FEFF8B4F: ??? > ==5139== by 0x7FEFF8C37: ??? > ==5139== by 0xE326EEF: ??? > ==5139== by 0x216F: ??? > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== Address 0x9288b38 is 0 bytes after a block of size 15,224 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > ==5139== by 0x223DB60: __MlevelNodeBisectionMultiple (ometis.c:402) > ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== > ==5139== Invalid read of size 8 > ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5139== by 0x21913DD: dmumps_557_ (dmumps_part2.F:2147) > ==5139== by 0x218E243: dmumps_195_ (dmumps_part2.F:1535) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > ==5139== Address 0xf24f6e8 is 3,163,816 bytes inside a block of size 3,163,820 alloc'd > ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5139== by 0x5CB3C43: for_allocate (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > ==5139== by 0x5CB3B50: for_alloc_allocatable (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > ==5139== by 0x218CD64: dmumps_195_ (dmumps_part2.F:1072) > ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > From jedbrown at mcs.anl.gov Sat Dec 3 09:04:39 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 3 Dec 2011 09:04:39 -0600 Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> Message-ID: On Sat, Dec 3, 2011 at 06:18, Gong Ding wrote: > In the function of PetscOptionsFindPair_Private (options.c line 1098), > there are still some code with the same problem. Are you passing a full path to the option? Or just many very long prefixes? This absolutely needs to be fixed, but I think there are a lot of places where fixed-length temporary buffers are used for building options keys and values. I would like to know how you are triggering these overflows so that I can stand a decent chance of getting all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Dec 3 09:55:00 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 3 Dec 2011 09:55:00 -0600 Subject: [petsc-users] DMDA Question In-Reply-To: <5AA0BBAE-30F4-4FCE-B8CB-934470AEA338@mcs.anl.gov> References: <495cb3b5-7afa-4962-b928-e7c5a0a1fe6b@mail2.gatech.edu> <5AA0BBAE-30F4-4FCE-B8CB-934470AEA338@mcs.anl.gov> Message-ID: On Fri, Dec 2, 2011 at 10:43 PM, Barry Smith wrote: > > On Dec 2, 2011, at 4:52 PM, Tim Gallagher wrote: > > > There are only unknowns at the cell centers. The coordinates of the > nodes may be used during computation though. > > > > I'd have to think if it's possible to specify the nodes as dof's of each > cell center... but that may duplicate a lot of data and be sort of > convoluted to understand what was going on in the code. > > Tim, > > So the issue is that there is no way to insure that two different DA's > with different dimensions get the same parallel layout so the coordinate > information you need is always easily available with the cell data. > > Someday we hope to have support for staggered grids with DAs and that > would make your task much much easier. Sorry it doesn't exist now. > > Here is what I would do now because it is so easy to do and the > parallelism becomes trivial. Create 2 DMDAs both of the same size but have > the first have a dof equal to your number of equations and have the second > one have a dof of 8 (for 2d problems) and 16 for 3d problems. Both DMDA's > will have the same parallel layout and the ghosted versions while also have > the same layout. Now store the x,y coordinates of all four vertices of each > cell (in 2d) into the 8 locations of a vector associated with the second > DMDA. Now you will access coordinates of cell vertices with the same > indexing that you use for the equations (after calling DMDAVecGetArray()). > The code will be very clean. > This is one option. For slightly less clean code that use less memory, you can adopt the convention in SNES ex30. Here staggered grid unknowns are all kept on the same DA, enough ghosting is used to make sure you have everything, and a convention tells you what thing you are addressing for each location. It was done by a user who is capable of keeping a lot of complexity in his head, so its not necessarily the easiest way to do it but its possible. Matt > The only drawback is the extra storage of the coordinates but frankly > that is likely to be a relatively small amount of additional memory > compared to everything else need in the simulation. > > Note: do not use the DMDACoordinates interfaces that is not appropriate > for what you need, just keep the coordinates as I described above. > > Barry > > > > > Tim > > > > ----- Original Message ----- > > From: "Barry Smith" > > To: gtg085x at mail.gatech.edu, "PETSc users list" > > > Sent: Friday, December 2, 2011 5:14:59 PM > > Subject: Re: [petsc-users] DMDA Question > > > > > > On Dec 2, 2011, at 1:19 PM, Tim Gallagher wrote: > > > >> Yes, I would like two DMDA's, one for the nodes and the other for the > cell centers. The grid is structured, but is non-uniform. > > > > Do you have degrees of freedom on the "nodes" and cell centers, or are > the only degrees of freedom on the cell centers and the nodes are used for > intermediate computations? That is does your nonlinear system involve > unknowns on both the nodes and centers? > > > > Barry > > > > > >> > >> I think I want that anyway, is that the best structure for a case like > this? > >> > >> Thanks, > >> > >> Tim > >> > >> ----- Original Message ----- > >> From: "Barry Smith" > >> To: gtg085x at mail.gatech.edu, "PETSc users list" < > petsc-users at mcs.anl.gov> > >> Sent: Friday, December 2, 2011 2:12:54 PM > >> Subject: Re: [petsc-users] DMDA Question > >> > >> > >> Tim, > >> > >> Unfortunately the DMDA was not designed with the idea of staggered > grids (or mixed discretizations) in mind so they are not straightforward > like we would like them to be. > >> > >> Do I understand you want two DMDA > >> > >> 1) one that has locations for cell vertices (or face centers)? > >> > >> 2) and one that locations for cell centers? > >> > >> Once we understand what you want we may be able to make a suggestion. > >> > >> Barry > >> > >> > >> On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > >> > >>> Hi, > >>> > >>> I'm trying to create some grids for a finite volume simulation and am > a little stuck on the best way to tackle it. The approach now is to use > DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the > processors. Then I am trying to create another DMDA with the same > distribution to store the cell centers and actual solution vector. The > problem with is is the following: > >>> > >>> Consider 9 points and two processors. Processor one gets points [1,5] > with point 6 as a ghost and processor two gets points [6,9] with point 5 as > a ghost. But now to create/store cell centers, processor two needs point 4 > to construct a cell center for it's ghost. > >>> > >>> I can certainly fetch that point and do the calculation, but I feel > like there is a more elegant way to do this out there. Has anybody used a > DMDA to create a dual DMDA? > >>> > >>> Any advice would be appreciated, > >>> > >>> Tim > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gshy2014 at gmail.com Sat Dec 3 13:24:36 2011 From: gshy2014 at gmail.com (Shiyuan) Date: Sat, 3 Dec 2011 13:24:36 -0600 Subject: [petsc-users] KSP with MatNullSpace In-Reply-To: References: <9F0F599F-F0FD-4E60-B601-0C422839CF18@mcs.anl.gov> Message-ID: Hi, I traced the program on the level of KSPSolve_CG, and I don't see the MatNullSpace is used. Where is MatNullSpace used? Can KSP CG automatically handle the null space provided by MatNullSpace? Or I need to change to some other solver? The matrix is semi-positive-definite with only one zero eigenvalue. The zero eigenvector is obtained by an eigensolver. Any suggestion? Thanks. Shiyuan On Fri, Dec 2, 2011 at 11:17 PM, Barry Smith wrote: > > On Dec 2, 2011, at 10:57 PM, Shiyuan wrote: > > > I notice that when the vector indicating the nullspace is not very > accurate (norm(A*phi) very small but not zero), KSP (CG) will return > KSP_INDEFINITE_PC. When b is very close to zero in AX=b, KSP (CG) has the > same error. Is it controlled by some tolerance? How can I set the tolerance? > > No it is not controlled by some tolerance, it cannot be controlled by > some tolerance because the indication of indefiniteness is determined by > > } else if ((i > 0) && (beta*betaold < 0.0)) { > ksp->reason = KSP_DIVERGED_INDEFINITE_PC; > ierr = PetscInfo(ksp,"diverging due to indefinite > preconditioner\n");CHKERRQ(ierr); > break; > > as soon as beta*betaold is negative the entire CG algorithm breaks down > and there is no way to recover. > > Barry > > > > > Thanks. > > > > Shiyuan > > > > On Wed, Nov 23, 2011 at 5:00 PM, Barry Smith wrote: > > > > On Nov 23, 2011, at 1:31 PM, Shiyuan wrote: > > > > > I've check it in Matlab. the matrix A is symmetric and has no negative > eigenvalues( only a zero eigen value). I've also checked that the nullspace > is correct( norm(A*phi)<1e-11); > > > > Hmm. You can run with -ksp_view_binary and email to > petsc-maint at mcs.anl.gov the resulting file called binaryoutput. > > > > Barry > > > > > > > Thanks. > > > > > > On Wed, Nov 23, 2011 at 1:26 PM, Barry Smith > wrote: > > > > > > Print out the matrix for a small grid then check in Matlab that it > is symmetric and has no negative eigenvalues. > > > > > > Barry > > > > > > > > > On Nov 23, 2011, at 9:50 AM, Shiyuan wrote: > > > > > > > Hi, > > > > I want to solve a singular system with a known nullspace. > However, I the KSP solve diverges with KSP_INDEFINTE_PC even if I disable > the preconditioning by PCNONE. > > > > this is how I setup the system. What did I do wrong? Any possible > causes? Thanks. > > > > > > > > > ierr=MatNullSpaceCreate(PETSC_COMM_SELF,PETSC_FALSE,1,&phi,&nsp);CHKERRV(ierr); > > > > ierr=KSPCreate(PETSC_COMM_SELF,&ksp);CHKERRV(ierr); > > > > > ierr=KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRV(ierr); > > > > ierr=KSPSetNullSpace(ksp,nsp);CHKERRV(ierr); > > > > ierr=KSPSetType(ksp,KSPCG);CHKERRV(ierr); > > > > ierr=KSPGetPC(ksp,&prec);CHKERRV(ierr); > > > > ierr=PCSetType(prec,PCNONE);CHKERRV(ierr); > > > > ierr=KSPSetTolerances(ksp,1e-5,1e-20,1e5,10000);CHKERRV(ierr); > > > > ierr=KSPSetFromOptions(ksp);CHKERRV(ierr); > > > > ierr=KSPSetUp(ksp);CHKERRV(ierr); > > > > > > > > Shiyuan > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 3 13:32:03 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 3 Dec 2011 13:32:03 -0600 Subject: [petsc-users] KSP with MatNullSpace In-Reply-To: References: <9F0F599F-F0FD-4E60-B601-0C422839CF18@mcs.anl.gov> Message-ID: <32321D12-36A3-4CF2-989F-5D1CEC473FD9@mcs.anl.gov> On Dec 3, 2011, at 1:24 PM, Shiyuan wrote: > Hi, I traced the program on the level of KSPSolve_CG, and I don't see the MatNullSpace is used. Where is MatNullSpace used? Can KSP CG automatically handle the null space provided by MatNullSpace? It is used, it is just a little convoluted: #define KSP_PCApply(ksp,x,y) (!ksp->transpose_solve) ? (PCApply(ksp->pc,x,y) || KSP_RemoveNullSpace(ksp,y)) : PCApplyTranspose(ksp->pc,x,y) so it automatically removes the null space after applying the preconditioner. > Or I need to change to some other solver? The matrix is semi-positive-definite with only one zero eigenvalue. The zero eigenvector is obtained by an eigensolver. My guess is that the zero eigenvector obtained with the eigensolver is just not accurate enough. Is there any way to increase the accuracy of the eigenvector? Is there any explicit formula for the eigenvector you can derived based on the PDE? Barry For the null space removal to work with CG you need a very accurate form of the null space. > Any suggestion? Thanks. > > > Shiyuan > > On Fri, Dec 2, 2011 at 11:17 PM, Barry Smith wrote: > > On Dec 2, 2011, at 10:57 PM, Shiyuan wrote: > > > I notice that when the vector indicating the nullspace is not very accurate (norm(A*phi) very small but not zero), KSP (CG) will return KSP_INDEFINITE_PC. When b is very close to zero in AX=b, KSP (CG) has the same error. Is it controlled by some tolerance? How can I set the tolerance? > > No it is not controlled by some tolerance, it cannot be controlled by some tolerance because the indication of indefiniteness is determined by > > } else if ((i > 0) && (beta*betaold < 0.0)) { > ksp->reason = KSP_DIVERGED_INDEFINITE_PC; > ierr = PetscInfo(ksp,"diverging due to indefinite preconditioner\n");CHKERRQ(ierr); > break; > > as soon as beta*betaold is negative the entire CG algorithm breaks down and there is no way to recover. > > Barry > > > > > Thanks. > > > > Shiyuan > > > > On Wed, Nov 23, 2011 at 5:00 PM, Barry Smith wrote: > > > > On Nov 23, 2011, at 1:31 PM, Shiyuan wrote: > > > > > I've check it in Matlab. the matrix A is symmetric and has no negative eigenvalues( only a zero eigen value). I've also checked that the nullspace is correct( norm(A*phi)<1e-11); > > > > Hmm. You can run with -ksp_view_binary and email to petsc-maint at mcs.anl.gov the resulting file called binaryoutput. > > > > Barry > > > > > > > Thanks. > > > > > > On Wed, Nov 23, 2011 at 1:26 PM, Barry Smith wrote: > > > > > > Print out the matrix for a small grid then check in Matlab that it is symmetric and has no negative eigenvalues. > > > > > > Barry > > > > > > > > > On Nov 23, 2011, at 9:50 AM, Shiyuan wrote: > > > > > > > Hi, > > > > I want to solve a singular system with a known nullspace. However, I the KSP solve diverges with KSP_INDEFINTE_PC even if I disable the preconditioning by PCNONE. > > > > this is how I setup the system. What did I do wrong? Any possible causes? Thanks. > > > > > > > > ierr=MatNullSpaceCreate(PETSC_COMM_SELF,PETSC_FALSE,1,&phi,&nsp);CHKERRV(ierr); > > > > ierr=KSPCreate(PETSC_COMM_SELF,&ksp);CHKERRV(ierr); > > > > ierr=KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRV(ierr); > > > > ierr=KSPSetNullSpace(ksp,nsp);CHKERRV(ierr); > > > > ierr=KSPSetType(ksp,KSPCG);CHKERRV(ierr); > > > > ierr=KSPGetPC(ksp,&prec);CHKERRV(ierr); > > > > ierr=PCSetType(prec,PCNONE);CHKERRV(ierr); > > > > ierr=KSPSetTolerances(ksp,1e-5,1e-20,1e5,10000);CHKERRV(ierr); > > > > ierr=KSPSetFromOptions(ksp);CHKERRV(ierr); > > > > ierr=KSPSetUp(ksp);CHKERRV(ierr); > > > > > > > > Shiyuan > > > > > > > > > > > > From bsmith at mcs.anl.gov Sat Dec 3 13:36:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 3 Dec 2011 13:36:21 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <5B58FB78E1E048EF803121C203D9A793@cogendaeda> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> Message-ID: <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> It seems likely a bug in that older version of Parmetis or MUMPS. Since parametis has been major fixed up since that release it would be crazy to debug the old version. You should switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html and use the options --download-mumps --download-metis --download-parmetis if the problem still persists we will help you debug it but my guess is it will start working. Barry On Dec 3, 2011, at 6:21 AM, Gong Ding wrote: > I indid use --download-parmetis and --download-mumps. > The parmetis version is 3.2.0-p1. > Maybe mumps is not compatable with this version? > > > > This is a problem in Metis/Parmetis. PETSc 3.2 MUST be used with parmetis 3.2.0 > > Are you perhaps using Metis/Parmetis metis-5.0.2 and parmetis-4.0.2? Did you use --download-parmetis or install it yourself? You should use the --download to the the right one > > Barry > > > On Dec 2, 2011, at 3:47 AM, Gong Ding wrote: > >> petsc-3.1-p8 & mumps 4.9 works well. >> However, petsc-3.2 p3-p5 & mumps 4.10 seems have memory problem. >> >> The code occasionally crash on Linux/X64 and alwasy crash on AIX/PPC. >> This problem may caused by mumps4.10 or parmetis, but I havn't test. >> >> The valgrind report: >> >> ==5139== Invalid read of size 8 >> ==5139== at 0x57BE749: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) >> ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) >> ==5139== by 0x2269296: __GrowBisection (initpart.c:200) >> ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) >> ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) >> ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) >> ==5139== Address 0x8718408 is 440 bytes inside a block of size 444 alloc'd >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) >> ==5139== by 0x226E6A2: __GKmalloc (util.c:111) >> ==5139== by 0x226E6E4: __idxmalloc (util.c:60) >> ==5139== by 0x2269014: __GrowBisection (initpart.c:101) >> ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) >> ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) >> ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) >> ==5139== >> ==5139== Invalid read of size 8 >> ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) >> ==5139== by 0x73F8CCCCC: ??? >> ==5139== by 0x300000001F: ??? >> ==5139== by 0x7FEFF8B4F: ??? >> ==5139== by 0x7FEFF8C37: ??? >> ==5139== by 0xE326EEF: ??? >> ==5139== by 0x216F: ??? >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) >> ==5139== Address 0x9288b38 is 0 bytes after a block of size 15,224 alloc'd >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) >> ==5139== by 0x226E6A2: __GKmalloc (util.c:111) >> ==5139== by 0x226E6E4: __idxmalloc (util.c:60) >> ==5139== by 0x223DB60: __MlevelNodeBisectionMultiple (ometis.c:402) >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) >> ==5139== >> ==5139== Invalid read of size 8 >> ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) >> ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) >> ==5139== by 0x21913DD: dmumps_557_ (dmumps_part2.F:2147) >> ==5139== by 0x218E243: dmumps_195_ (dmumps_part2.F:1535) >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) >> ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) >> ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) >> ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) >> ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) >> ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) >> ==5139== Address 0xf24f6e8 is 3,163,816 bytes inside a block of size 3,163,820 alloc'd >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) >> ==5139== by 0x5CB3C43: for_allocate (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) >> ==5139== by 0x5CB3B50: for_alloc_allocatable (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) >> ==5139== by 0x218CD64: dmumps_195_ (dmumps_part2.F:1072) >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) >> ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) >> ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) >> ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) >> ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) >> ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) >> From xiaohl1986 at gmail.com Sat Dec 3 17:00:14 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Sat, 3 Dec 2011 17:00:14 -0600 Subject: [petsc-users] question about DMDA vector Message-ID: Hi I need to set the values of a vector genarated by DMGetGlobalVector. >From your manual page 50, I saw the following PetscScalar **f,**u; ... DMDAVecGetArray(DM da,Vec local,(void*)&u); DMDAVecGetArray(DM da,Vec global,(void*)&f); ... f[i][j] = u[i][j] - ... does it change the values in vector "global" after I change the values in "f"? or I have to use the routines in page 42 to set the vector values, like VecSetValues(Vec x,int n,int *indices,PetscScalar *values,INSERT VALUES); VecAssemblyBegin(Vec x); VecAssemblyEnd(Vec x); -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Dec 3 17:02:59 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 3 Dec 2011 17:02:59 -0600 Subject: [petsc-users] question about DMDA vector In-Reply-To: References: Message-ID: On Sat, Dec 3, 2011 at 17:00, Hailong Xiao wrote: > PetscScalar **f,**u; > ... > DMDAVecGetArray(DM da,Vec local,(void*)&u); > DMDAVecGetArray(DM da,Vec global,(void*)&f); > ... > f[i][j] = u[i][j] - ... > > > does it change the values in vector "global" after I change the values in > "f"? > This sets the values in global, but you can only address "owned" parts of the global vector. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiaohl1986 at gmail.com Sat Dec 3 17:07:42 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Sat, 3 Dec 2011 17:07:42 -0600 Subject: [petsc-users] question about DMDA vector In-Reply-To: References: Message-ID: So, I do not need to call the VecAssemblyBegin; and VecAssemblyEnd; routines in order to set the values of f? On Sat, Dec 3, 2011 at 5:02 PM, Jed Brown wrote: > On Sat, Dec 3, 2011 at 17:00, Hailong Xiao wrote: > >> PetscScalar **f,**u; >> ... >> DMDAVecGetArray(DM da,Vec local,(void*)&u); >> DMDAVecGetArray(DM da,Vec global,(void*)&f); >> ... >> f[i][j] = u[i][j] - ... >> >> >> does it change the values in vector "global" after I change the values in >> "f"? >> > > This sets the values in global, but you can only address "owned" parts of > the global vector. > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Dec 3 17:09:50 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 3 Dec 2011 17:09:50 -0600 Subject: [petsc-users] question about DMDA vector In-Reply-To: References: Message-ID: On Sat, Dec 3, 2011 at 17:07, Hailong Xiao wrote: > So, I do not need to call the VecAssemblyBegin; and VecAssemblyEnd; > routines in order to set the values of f? Only in order set values that you do not own. Alternatively, you can set values in a local vector and DMDALocalToGlobalBegin/End. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiaohl1986 at gmail.com Sat Dec 3 17:11:01 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Sat, 3 Dec 2011 17:11:01 -0600 Subject: [petsc-users] question about DMDA vector In-Reply-To: References: Message-ID: Thank you Jed. On Sat, Dec 3, 2011 at 5:09 PM, Jed Brown wrote: > > > On Sat, Dec 3, 2011 at 17:07, Hailong Xiao wrote: > >> So, I do not need to call the VecAssemblyBegin; and VecAssemblyEnd; >> routines in order to set the values of f? > > > Only in order set values that you do not own. > > Alternatively, you can set values in a local vector and > DMDALocalToGlobalBegin/End. > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Sat Dec 3 23:23:19 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sun, 4 Dec 2011 13:23:19 +0800 (CST) Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> Message-ID: <29405729.354881322976199174.JavaMail.coremail@mail.ustc.edu> I don't have any long options. For example. I takes a look at following code ==12208== Conditional jump or move depends on uninitialised value(s) ==12208== at 0x171A9FA: PetscStrlen (str.c:149) ==12208== by 0x16EFF89: PetscOptionsHasName (options.c:1242) ==12208== by 0x141AA4A: main (main.cc:101) ==12208== In the line 101 of main.cc I just call PetscOptionsHasName to check -p command line argument. And the -p argument don't have parameter. // performace log flag PetscBool log_flg; PetscOptionsHasName(PETSC_NULL,"-p", &log_flg); if(!log_flg) perflog.disable_logging(); I checked petsc source file options.c In the source file options.c line 1098 static PetscErrorCode PetscOptionsFindPair_Private(const char pre[],const char name[],char *value[],PetscBool *flg) the tmp buffer has a fixed size of 256. the strncpy function has a max number of 256. /* append prefix to name */ if (pre) { if (pre[0] == '-') SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_WRONG,"Prefix should not begin with a -"); ierr = PetscStrncpy(tmp,pre,256);CHKERRQ(ierr); ierr = PetscStrlen(tmp,&len);CHKERRQ(ierr); ierr = PetscStrncat(tmp,name+1,256-len-1);CHKERRQ(ierr); } else { ierr = PetscStrncpy(tmp,name+1,256);CHKERRQ(ierr); } This should OK if name string has a length less than 256. The length of -p option is only 2. However, valgrind reported here has uninitialised value. I try to init both tmp (line 1103) and tmp2 (line 1152) as tmp[256]={0}, tmp2[256]={0}, the valgrind complain disappeared. I'v no idea what's wrong here. Maybe something wrong in valgrind. On Sat, Dec 3, 2011 at 06:18, Gong Ding wrote: In the function of PetscOptionsFindPair_Private (options.c line 1098), there are still some code with the same problem. Are you passing a full path to the option? Or just many very long prefixes? This absolutely needs to be fixed, but I think there are a lot of places where fixed-length temporary buffers are used for building options keys and values. I would like to know how you are triggering these overflows so that I can stand a decent chance of getting all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Sun Dec 4 00:29:41 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sun, 4 Dec 2011 14:29:41 +0800 (CST) Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> Message-ID: <26575371.354901322980181499.JavaMail.coremail@mail.ustc.edu> I spend hours on checking the valgrind reporting but find nothing. Change PetscMalloc to malloc in PetscStrallocpy can reduce memory warning about mem alignment, but not critical. Sigh... PetscErrorCode PetscStrallocpy(const char s[],char *t[]) { PetscErrorCode ierr; size_t len=0; char *tmp = 0; PetscFunctionBegin; if (s) { ierr = PetscStrlen(s,&len);CHKERRQ(ierr); tmp = (char *)malloc((1+len)*sizeof(char)); ierr = PetscStrcpy(tmp,s);CHKERRQ(ierr); tmp[len] = 0; } *t = tmp; PetscFunctionReturn(0); } The most dangerous error may be caused by intel library, not petsc. Anyway, you can put it away. ==5545== Invalid read of size 8 ==5545== at 0x57A9568: __intel_sse2_strlen (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) ==5545== by 0x171AB46: PetscStrlen (str.c:149) ==5545== by 0x171B3B8: PetscStrallocpy (str.c:185) ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) ==5545== by 0x16CA59A: PetscFListAdd (reg.c:229) ==5545== by 0x16C9F61: PetscFListDuplicate (reg.c:596) ==5545== by 0x1890951: MatDuplicateNoCreate_SeqAIJ (aij.c:3964) ==5545== by 0x18BBB77: MatILUFactorSymbolic_SeqAIJ_ilu0 (aijfact.c:1639) ==5545== by 0x18B9F6B: MatILUFactorSymbolic_SeqAIJ (aijfact.c:1740) ==5545== by 0x185315A: MatILUFactorSymbolic (matrix.c:6092) ==5545== by 0x1C9CD28: PCSetUp_ILU (ilu.c:216) ==5545== by 0x1F5B32B: PCSetUp (precon.c:819) ==5545== Address 0x8416118 is 24 bytes inside a block of size 28 alloc'd ==5545== at 0x4A0776F: malloc (vg_replace_malloc.c:263) ==5545== by 0x171B3CD: PetscStrallocpy (str.c:186) ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) ==5545== by 0x16CA8D2: PetscFListAdd (reg.c:205) ==5545== by 0x16E4014: PetscObjectComposeFunction_Petsc (inherit.c:466) ==5545== by 0x16E5214: PetscObjectComposeFunction (inherit.c:600) ==5545== by 0x18A0BBA: MatCreate_SeqAIJ (aij.c:3814) ==5545== by 0x1871689: MatSetType (matreg.c:73) ==5545== by 0x1247AF0: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:220) ==5545== by 0x132CCAE: PoissonSolver::create_solver() (poisson.cc:51) ==5545== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) ==5545== by 0xEB9688: SolverControl::mainloop() (control.cc:200) On Sat, Dec 3, 2011 at 06:18, Gong Ding wrote: In the function of PetscOptionsFindPair_Private (options.c line 1098), there are still some code with the same problem. Are you passing a full path to the option? Or just many very long prefixes? This absolutely needs to be fixed, but I think there are a lot of places where fixed-length temporary buffers are used for building options keys and values. I would like to know how you are triggering these overflows so that I can stand a decent chance of getting all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Dec 4 12:04:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 4 Dec 2011 12:04:34 -0600 Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: <26575371.354901322980181499.JavaMail.coremail@mail.ustc.edu> References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> <26575371.354901322980181499.JavaMail.coremail@mail.ustc.edu> Message-ID: You could try building everything with the gnu compilers and see if these message all go away; I bet they will because they are caused by problems with the intel compilers. Barry On Dec 4, 2011, at 12:29 AM, Gong Ding wrote: > I spend hours on checking the valgrind reporting but find nothing. > Change PetscMalloc to malloc in PetscStrallocpy can reduce memory warning about mem alignment, but not critical. > Sigh... > > PetscErrorCode PetscStrallocpy(const char s[],char *t[]) > { > PetscErrorCode ierr; > size_t len=0; > char *tmp = 0; > > PetscFunctionBegin; > if (s) { > ierr = PetscStrlen(s,&len);CHKERRQ(ierr); > tmp = (char *)malloc((1+len)*sizeof(char)); > ierr = PetscStrcpy(tmp,s);CHKERRQ(ierr); > tmp[len] = 0; > } > *t = tmp; > PetscFunctionReturn(0); > } > > The most dangerous error may be caused by intel library, not petsc. > Anyway, you can put it away. > > ==5545== Invalid read of size 8 > ==5545== at 0x57A9568: __intel_sse2_strlen (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > ==5545== by 0x171AB46: PetscStrlen (str.c:149) > ==5545== by 0x171B3B8: PetscStrallocpy (str.c:185) > ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) > ==5545== by 0x16CA59A: PetscFListAdd (reg.c:229) > ==5545== by 0x16C9F61: PetscFListDuplicate (reg.c:596) > ==5545== by 0x1890951: MatDuplicateNoCreate_SeqAIJ (aij.c:3964) > ==5545== by 0x18BBB77: MatILUFactorSymbolic_SeqAIJ_ilu0 (aijfact.c:1639) > ==5545== by 0x18B9F6B: MatILUFactorSymbolic_SeqAIJ (aijfact.c:1740) > ==5545== by 0x185315A: MatILUFactorSymbolic (matrix.c:6092) > ==5545== by 0x1C9CD28: PCSetUp_ILU (ilu.c:216) > ==5545== by 0x1F5B32B: PCSetUp (precon.c:819) > ==5545== Address 0x8416118 is 24 bytes inside a block of size 28 alloc'd > ==5545== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > ==5545== by 0x171B3CD: PetscStrallocpy (str.c:186) > ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) > ==5545== by 0x16CA8D2: PetscFListAdd (reg.c:205) > ==5545== by 0x16E4014: PetscObjectComposeFunction_Petsc (inherit.c:466) > ==5545== by 0x16E5214: PetscObjectComposeFunction (inherit.c:600) > ==5545== by 0x18A0BBA: MatCreate_SeqAIJ (aij.c:3814) > ==5545== by 0x1871689: MatSetType (matreg.c:73) > ==5545== by 0x1247AF0: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:220) > ==5545== by 0x132CCAE: PoissonSolver::create_solver() (poisson.cc:51) > ==5545== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) > ==5545== by 0xEB9688: SolverControl::mainloop() (control.cc:200) > > > > > On Sat, Dec 3, 2011 at 06:18, Gong Ding wrote: > In the function of PetscOptionsFindPair_Private (options.c line 1098), there are still some code with the same problem. > > Are you passing a full path to the option? Or just many very long prefixes? > > This absolutely needs to be fixed, but I think there are a lot of places where fixed-length temporary buffers are used for building options keys and values. I would like to know how you are triggering these overflows so that I can stand a decent chance of getting all of them. From gdiso at ustc.edu Mon Dec 5 03:48:57 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 5 Dec 2011 17:48:57 +0800 (CST) Subject: [petsc-users] cmake METIS problem Message-ID: <5531331.357581323078537599.JavaMail.coremail@mail.ustc.edu> Hi, I try to install petsc-dev on AIX/PPC cluster. I'd like to use xlc, however, there is a dead gcc/g++ installed in /usr/bin. Since I am not the administrator, I can't do anything to remove/update it. The cmake configure file for metis tries to find c++ compiler, and reported error with that g++. Error running configure on METIS: Could not execute "cd /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && make distclean && make config prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" doubleprecision=1": rm -rf build/AIX-000092FAD400 rm -rf build/AIX-000092FAD400 mkdir -p build/AIX-000092FAD400 cd build/AIX-000092FAD400 && cmake /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 -- The C compiler identification is XL -- The CXX compiler identification is unknown -- Check for working C compiler: /usr/bin/mpcc_r -- Check for working C compiler: /usr/bin/mpcc_r -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken -- Configuring incomplete, errors occurred! I think the testing for c++ compiler can be removed here? From gdiso at ustc.edu Mon Dec 5 04:09:24 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 5 Dec 2011 18:09:24 +0800 (CST) Subject: [petsc-users] many valgrind error of petsc string functions In-Reply-To: References: <28198114.352991322819526907.JavaMail.coremail@mail.ustc.edu> <03AD4EBDAEAE4645A81C40CEDA1F014D@cogendaeda> <26575371.354901322980181499.JavaMail.coremail@mail.ustc.edu> Message-ID: <31741035.357621323079764045.JavaMail.coremail@mail.ustc.edu> Yes, you are right. I compiled everything with gcc, and valgrind stops complain now. > > You could try building everything with the gnu compilers and see if these message all go away; I bet they will because they are caused by problems with the intel compilers. > > > > Barry > > > > On Dec 4, 2011, at 12:29 AM, Gong Ding wrote: > > > > > I spend hours on checking the valgrind reporting but find nothing. > > > Change PetscMalloc to malloc in PetscStrallocpy can reduce memory warning about mem alignment, but not critical. > > > Sigh... > > > > > > PetscErrorCode PetscStrallocpy(const char s[],char *t[]) > > > { > > > PetscErrorCode ierr; > > > size_t len=0; > > > char *tmp = 0; > > > > > > PetscFunctionBegin; > > > if (s) { > > > ierr = PetscStrlen(s,&len);CHKERRQ(ierr); > > > tmp = (char *)malloc((1+len)*sizeof(char)); > > > ierr = PetscStrcpy(tmp,s);CHKERRQ(ierr); > > > tmp[len] = 0; > > > } > > > *t = tmp; > > > PetscFunctionReturn(0); > > > } > > > > > > The most dangerous error may be caused by intel library, not petsc. > > > Anyway, you can put it away. > > > > > > ==5545== Invalid read of size 8 > > > ==5545== at 0x57A9568: __intel_sse2_strlen (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > > ==5545== by 0x171AB46: PetscStrlen (str.c:149) > > > ==5545== by 0x171B3B8: PetscStrallocpy (str.c:185) > > > ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) > > > ==5545== by 0x16CA59A: PetscFListAdd (reg.c:229) > > > ==5545== by 0x16C9F61: PetscFListDuplicate (reg.c:596) > > > ==5545== by 0x1890951: MatDuplicateNoCreate_SeqAIJ (aij.c:3964) > > > ==5545== by 0x18BBB77: MatILUFactorSymbolic_SeqAIJ_ilu0 (aijfact.c:1639) > > > ==5545== by 0x18B9F6B: MatILUFactorSymbolic_SeqAIJ (aijfact.c:1740) > > > ==5545== by 0x185315A: MatILUFactorSymbolic (matrix.c:6092) > > > ==5545== by 0x1C9CD28: PCSetUp_ILU (ilu.c:216) > > > ==5545== by 0x1F5B32B: PCSetUp (precon.c:819) > > > ==5545== Address 0x8416118 is 24 bytes inside a block of size 28 alloc'd > > > ==5545== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > > > ==5545== by 0x171B3CD: PetscStrallocpy (str.c:186) > > > ==5545== by 0x16CAD2C: PetscFListGetPathAndFunction (reg.c:25) > > > ==5545== by 0x16CA8D2: PetscFListAdd (reg.c:205) > > > ==5545== by 0x16E4014: PetscObjectComposeFunction_Petsc (inherit.c:466) > > > ==5545== by 0x16E5214: PetscObjectComposeFunction (inherit.c:600) > > > ==5545== by 0x18A0BBA: MatCreate_SeqAIJ (aij.c:3814) > > > ==5545== by 0x1871689: MatSetType (matreg.c:73) > > > ==5545== by 0x1247AF0: FVM_NonlinearSolver::setup_nonlinear_data() (fvm_nonlinear_solver.cc:220) > > > ==5545== by 0x132CCAE: PoissonSolver::create_solver() (poisson.cc:51) > > > ==5545== by 0xEC7A61: SolverControl::do_solve(Parser::Card const&) (control.cc:1081) > > > ==5545== by 0xEB9688: SolverControl::mainloop() (control.cc:200) > > > > > > > > > > > > > > > On Sat, Dec 3, 2011 at 06:18, Gong Ding wrote: > > > In the function of PetscOptionsFindPair_Private (options.c line 1098), there are still some code with the same problem. > > > > > > Are you passing a full path to the option? Or just many very long prefixes? > > > > > > This absolutely needs to be fixed, but I think there are a lot of places where fixed-length temporary buffers are used for building options keys and values. I would like to know how you are triggering these overflows so that I can stand a decent chance of getting all of them. > > > > From knepley at gmail.com Mon Dec 5 05:49:22 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 05:49:22 -0600 Subject: [petsc-users] cmake METIS problem In-Reply-To: <5531331.357581323078537599.JavaMail.coremail@mail.ustc.edu> References: <5531331.357581323078537599.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/12/5 Gong Ding > Hi, > I try to install petsc-dev on AIX/PPC cluster. > I'd like to use xlc, however, there is a dead gcc/g++ installed in > /usr/bin. > Since I am not the administrator, I can't do anything to remove/update it. > Specify the compilers using --with-cc --with-cxx --with-fc. I suggest you specify the MPI compiler wrappers for this machine. Matt > The cmake configure file for metis tries to find c++ compiler, and > reported error with that g++. > > Error running configure on METIS: Could not execute "cd > /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && > make distclean && make config > prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" > doubleprecision=1": > rm -rf build/AIX-000092FAD400 > rm -rf build/AIX-000092FAD400 > mkdir -p build/AIX-000092FAD400 > cd build/AIX-000092FAD400 && cmake > /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 > -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= > -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP > -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 > -- The C compiler identification is XL > -- The CXX compiler identification is unknown > -- Check for working C compiler: /usr/bin/mpcc_r > -- Check for working C compiler: /usr/bin/mpcc_r -- works > -- Detecting C compiler ABI info > -- Detecting C compiler ABI info - done > -- Check for working CXX compiler: /usr/bin/g++ > -- Check for working CXX compiler: /usr/bin/g++ -- broken > -- Configuring incomplete, errors occurred! > > I think the testing for c++ compiler can be removed here? > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Mon Dec 5 05:50:05 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 5 Dec 2011 19:50:05 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> References: <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> Message-ID: <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> Yes, petsc-dev works. Please merge the update to petsc-3.2. > It seems likely a bug in that older version of Parmetis or MUMPS. Since parametis has been major fixed up since that release it would be crazy to debug the old version. > > > > You should switch to petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html and use the options --download-mumps --download-metis --download-parmetis if the problem still persists we will help you debug it but my guess is it will start working. > > > > Barry > > > > > > > > On Dec 3, 2011, at 6:21 AM, Gong Ding wrote: > > > > > I indid use --download-parmetis and --download-mumps. > > > The parmetis version is 3.2.0-p1. > > > Maybe mumps is not compatable with this version? > > > > > > > > > > > > This is a problem in Metis/Parmetis. PETSc 3.2 MUST be used with parmetis 3.2.0 > > > > > > Are you perhaps using Metis/Parmetis metis-5.0.2 and parmetis-4.0.2? Did you use --download-parmetis or install it yourself? You should use the --download to the the right one > > > > > > Barry > > > > > > > > > On Dec 2, 2011, at 3:47 AM, Gong Ding wrote: > > > > > >> petsc-3.1-p8 & mumps 4.9 works well. > > >> However, petsc-3.2 p3-p5 & mumps 4.10 seems have memory problem. > > >> > > >> The code occasionally crash on Linux/X64 and alwasy crash on AIX/PPC. > > >> This problem may caused by mumps4.10 or parmetis, but I havn't test. > > >> > > >> The valgrind report: > > >> > > >> ==5139== Invalid read of size 8 > > >> ==5139== at 0x57BE749: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > >> ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > >> ==5139== by 0x2269296: __GrowBisection (initpart.c:200) > > >> ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > > >> ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > > >> ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > > >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > > >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > > >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > > >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > > >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > > >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > > >> ==5139== Address 0x8718408 is 440 bytes inside a block of size 444 alloc'd > > >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > > >> ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > > >> ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > > >> ==5139== by 0x2269014: __GrowBisection (initpart.c:101) > > >> ==5139== by 0x2269C5F: __Init2WayPartition (initpart.c:36) > > >> ==5139== by 0x223DDFA: __MlevelNodeBisection (ometis.c:485) > > >> ==5139== by 0x223DB9B: __MlevelNodeBisectionMultiple (ometis.c:405) > > >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > > >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > > >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > > >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > > >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > > >> ==5139== > > >> ==5139== Invalid read of size 8 > > >> ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > >> ==5139== by 0x73F8CCCCC: ??? > > >> ==5139== by 0x300000001F: ??? > > >> ==5139== by 0x7FEFF8B4F: ??? > > >> ==5139== by 0x7FEFF8C37: ??? > > >> ==5139== by 0xE326EEF: ??? > > >> ==5139== by 0x216F: ??? > > >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > > >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > > >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > > >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > > >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > > >> ==5139== Address 0x9288b38 is 0 bytes after a block of size 15,224 alloc'd > > >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > > >> ==5139== by 0x226E6A2: __GKmalloc (util.c:111) > > >> ==5139== by 0x226E6E4: __idxmalloc (util.c:60) > > >> ==5139== by 0x223DB60: __MlevelNodeBisectionMultiple (ometis.c:402) > > >> ==5139== by 0x223D957: __MlevelNestedDissection (ometis.c:289) > > >> ==5139== by 0x223DA09: __MlevelNestedDissection (ometis.c:309) > > >> ==5139== by 0x223E837: METIS_NodeND (ometis.c:157) > > >> ==5139== by 0x2245E31: metis_nodend_ (frename.c:122) > > >> ==5139== by 0x2190666: dmumps_195_ (dmumps_part2.F:1435) > > >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > > >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > > >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > > >> ==5139== > > >> ==5139== Invalid read of size 8 > > >> ==5139== at 0x57BE609: __intel_new_memcpy (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > >> ==5139== by 0x57A0AF5: _intel_fast_memcpy.J (in /opt/intel/Compiler/11.1/038/lib/intel64/libirc.so) > > >> ==5139== by 0x21913DD: dmumps_557_ (dmumps_part2.F:2147) > > >> ==5139== by 0x218E243: dmumps_195_ (dmumps_part2.F:1535) > > >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > > >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > > >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > > >> ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > > >> ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > > >> ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > > >> ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > > >> ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > > >> ==5139== Address 0xf24f6e8 is 3,163,816 bytes inside a block of size 3,163,820 alloc'd > > >> ==5139== at 0x4A0776F: malloc (vg_replace_malloc.c:263) > > >> ==5139== by 0x5CB3C43: for_allocate (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > > >> ==5139== by 0x5CB3B50: for_alloc_allocatable (in /opt/intel/Compiler/11.1/038/lib/intel64/libifcore.so.5) > > >> ==5139== by 0x218CD64: dmumps_195_ (dmumps_part2.F:1072) > > >> ==5139== by 0x20C79D2: dmumps_26_ (dmumps_part5.F:313) > > >> ==5139== by 0x216A8DC: dmumps_ (dmumps_part1.F:409) > > >> ==5139== by 0x209407F: dmumps_f77_ (dmumps_part3.F:6651) > > >> ==5139== by 0x2073047: dmumps_c (mumps_c.c:422) > > >> ==5139== by 0x19755BF: MatLUFactorSymbolic_AIJMUMPS (mumps.c:893) > > >> ==5139== by 0x1831D89: MatLUFactorSymbolic (matrix.c:2823) > > >> ==5139== by 0x1C98F64: PCSetUp_LU (lu.c:135) > > >> ==5139== by 0x1F5B26B: PCSetUp (precon.c:819) > > >> > > > > From frtr at risoe.dtu.dk Mon Dec 5 06:48:48 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Mon, 5 Dec 2011 13:48:48 +0100 Subject: [petsc-users] MatView question Message-ID: Hi, I'm confused about the matview command, or rather, it's output. The following lines of code: int rank; MPI_Comm_rank(PETSC_COMM_WORLD,&rank); MatView((*FD).ddx,PETSC_VIEWER_STDOUT_WORLD); int fr,lr; MatGetOwnershipRange((*FD).ddx,&fr,&lr); printf("process %d has rows %d to %d\n",rank,fr,lr); Produce the following output: >mpirun -np 4 ./petsclapl Matrix Object: 1 MPI processes type: mpiaij row 0: (0, 5) (1, 5) (12, 0) (13, 0) (108, 0) (109, 0) row 1: (0, -5) (1, 0) (2, 5) (12, 0) (13, 0) (14, 0) (108, 0) (109, 0) (110, 0) [snip] process 0 has rows 0 to 30 process 1 has rows 30 to 60 process 2 has rows 60 to 90 process 3 has rows 90 to 120 My question is this: How come the MatView command states 1 MPI processes, while, when I use GetOwnershipRange, the (expected) division into ? of the rows at each process is reported? Is the MatView "processes" the process of the viewer? And more importantly, what is the actual position of my matrix - is it on 1 or on 4 processes? --- yours sincerily Frederik Treue -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 5 06:57:55 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 06:57:55 -0600 Subject: [petsc-users] MatView question In-Reply-To: References: Message-ID: On Mon, Dec 5, 2011 at 6:48 AM, Treue, Frederik wrote: > Hi,**** > > ** ** > > I?m confused about the matview command, or rather, it?s output. The > following lines of code:**** > > int rank;**** > > MPI_Comm_rank(PETSC_COMM_WORLD,&rank);**** > > MatView((*FD).ddx,PETSC_VIEWER_STDOUT_WORLD);**** > > int fr,lr;**** > > MatGetOwnershipRange((*FD).ddx,&fr,&lr);**** > > printf("process %d has rows %d to %d\n",rank,fr,lr);**** > > Produce the following output:**** > > >mpirun -np 4 ./petsclapl **** > > Matrix Object: 1 MPI processes**** > > type: mpiaij**** > > row 0: (0, 5) (1, 5) (12, 0) (13, 0) (108, 0) (109, 0) **** > > row 1: (0, -5) (1, 0) (2, 5) (12, 0) (13, 0) (14, 0) (108, 0) > (109, 0) (110, 0)**** > > [snip]**** > > process 0 has rows 0 to 30**** > > process 1 has rows 30 to 60**** > > process 2 has rows 60 to 90**** > > process 3 has rows 90 to 120**** > > ** ** > > My question is this: How come the MatView command states 1 MPI processes, > while, when I use GetOwnershipRange, the (expected) division into ? of the > rows at each process is reported? Is the MatView ?processes? the process of > the viewer? And more importantly, what is the actual position of my matrix > ? is it on 1 or on 4 processes? > This is a little confusing. We gather the matrix onto process 0 so we can print it, so it shows 1 process. Matt > > > ---**** > > yours sincerily**** > > Frederik Treue**** > > ** ** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Mon Dec 5 06:58:29 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 5 Dec 2011 20:58:29 +0800 (CST) Subject: [petsc-users] cmake METIS problem In-Reply-To: References: <5531331.357581323078537599.JavaMail.coremail@mail.ustc.edu> Message-ID: <17479811.357811323089909577.JavaMail.coremail@mail.ustc.edu> I do specify --with-cc --with-cxx --with-fc. Here is the configure options configure --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=2 --known-memcmp-ok=1 --known-endian=big --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --download-f-blas-lapack=1 --download-mumps=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-superlu=1 --download-superlu_dist=1 --with-c2html=0 --with-debugging=0 --with-clanguage=cxx --with-cc="mpcc_r" --with-fc="mpxlf_r" --with-cxx="mpCC_r" --CFLAGS="-q64 -O3 -qmaxmem=-1 -qstrict -qarch=auto -qtune=auto" --CXXFLAGS="-q64 -O3 -qmaxmem=-1 -qipa -qstrict -qarch=auto -qtune=auto" --FFLAGS="-q64 -O3 -qmaxmem=-1 -qarch=auto -qtune=auto" --with-sowing=0 --with-batch=1 --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-x=0 --with-pic=1 But cmake does not use given cxx. You see, cc="mpcc_r" is passed to cmake, but no cxx. Cmake tries to find cxx but failed. Hi, I try to install petsc-dev on AIX/PPC cluster. I'd like to use xlc, however, there is a dead gcc/g++ installed in /usr/bin. Since I am not the administrator, I can't do anything to remove/update it. Specify the compilers using --with-cc --with-cxx --with-fc. I suggest you specify the MPIcompiler wrappers for this machine. Matt The cmake configure file for metis tries to find c++ compiler, and reported error with that g++. Error running configure on METIS: Could not execute "cd /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && make distclean && make config prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" doubleprecision=1": rm -rf build/AIX-000092FAD400 rm -rf build/AIX-000092FAD400 mkdir -p build/AIX-000092FAD400 cd build/AIX-000092FAD400 && cmake /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 -- The C compiler identification is XL -- The CXX compiler identification is unknown -- Check for working C compiler: /usr/bin/mpcc_r -- Check for working C compiler: /usr/bin/mpcc_r -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken -- Configuring incomplete, errors occurred! I think the testing for c++ compiler can be removed here? -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.gallagher at gatech.edu Mon Dec 5 07:02:38 2011 From: tim.gallagher at gatech.edu (Tim Gallagher) Date: Mon, 05 Dec 2011 08:02:38 -0500 (EST) Subject: [petsc-users] cmake METIS problem In-Reply-To: <17479811.357811323089909577.JavaMail.coremail@mail.ustc.edu> Message-ID: You can set the CXX environment variable, CMake should always pick those up. Tim ----- Original Message ----- From: "Gong Ding" To: "PETSc users list" Sent: Monday, December 5, 2011 7:58:29 AM Subject: Re: [petsc-users] cmake METIS problem I do specify --with-cc --with-cxx --with-fc. Here is the configure options configure --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=2 --known-memcmp-ok=1 --known-endian=big --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --download-f-blas-lapack=1 --download-mumps=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-superlu=1 --download-superlu_dist=1 --with-c2html=0 --with-debugging=0 --with-clanguage=cxx --with-cc="mpcc_r" --with-fc="mpxlf_r" --with-cxx="mpCC_r" --CFLAGS="-q64 -O3 -qmaxmem=-1 -qstrict -qarch=auto -qtune=auto" --CXXFLAGS="-q64 -O3 -qmaxmem=-1 -qipa -qstrict -qarch=auto -qtune=auto" --FFLAGS="-q64 -O3 -qmaxmem=-1 -qarch=auto -qtune=auto" --with-sowing=0 --with-batch=1 --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-x=0 --with-pic=1 But cmake does not use given cxx. You see, cc="mpcc_r" is passed to cmake, but no cxx. Cmake tries to find cxx but failed.
Hi, I try to install petsc-dev on AIX/PPC cluster. I'd like to use xlc, however, there is a dead gcc/g++ installed in /usr/bin. Since I am not the administrator, I can't do anything to remove/update it. Specify the compilers using --with-cc --with-cxx --with-fc. I suggest you specify the MPI compiler wrappers for this machine. Matt
The cmake configure file for metis tries to find c++ compiler, and reported error with that g++. Error running configure on METIS: Could not execute "cd /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && make distclean && make config prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" doubleprecision=1": rm -rf build/AIX-000092FAD400 rm -rf build/AIX-000092FAD400 mkdir -p build/AIX-000092FAD400 cd build/AIX-000092FAD400 && cmake /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 -- The C compiler identification is XL -- The CXX compiler identification is unknown -- Check for working C compiler: /usr/bin/mpcc_r -- Check for working C compiler: /usr/bin/mpcc_r -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken -- Configuring incomplete, errors occurred! I think the testing for c++ compiler can be removed here?
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 5 07:06:38 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 07:06:38 -0600 Subject: [petsc-users] cmake METIS problem In-Reply-To: <17479811.357811323089909577.JavaMail.coremail@mail.ustc.edu> References: <5531331.357581323078537599.JavaMail.coremail@mail.ustc.edu> <17479811.357811323089909577.JavaMail.coremail@mail.ustc.edu> Message-ID: On Mon, Dec 5, 2011 at 6:58 AM, Gong Ding wrote: > > I do specify --with-cc --with-cxx --with-fc. Here is the configure options > configure --known-level1-dcache-size=32768 > --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=2 > --known-memcmp-ok=1 --known-endian=big --known-sizeof-char=1 > --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 > --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 > --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 > --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 > --known-mpi-long-double=1 --download-f-blas-lapack=1 --download-mumps=1 > --download-blacs=1 --download-metis=1 --download-parmetis=1 > --download-scalapack=1 --download-superlu=1 --download-superlu_dist=1 > --with-c2html=0 --with-debugging=0 --with-clanguage=cxx --with-cc="mpcc_r" > --with-fc="mpxlf_r" --with-cxx="mpCC_r" --CFLAGS="-q64 -O3 -qmaxmem=-1 > -qstrict -qarch=auto -qtune=auto" --CXXFLAGS="-q64 -O3 -qmaxmem=-1 -qipa > -qstrict -qarch=auto -qtune=auto" --FFLAGS="-q64 -O3 -qmaxmem=-1 > -qarch=auto -qtune=auto" --with-sowing=0 --with-batch=1 > --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-x=0 > --with-pic=1 > > But cmake does not use given cxx. You see, cc="mpcc_r" is passed to cmake, > but no cxx. > Cmake tries to find cxx but failed. > You can do make all-legacy until this is ficed. Matt > > > > > > Hi, >> I try to install petsc-dev on AIX/PPC cluster. >> I'd like to use xlc, however, there is a dead gcc/g++ installed in >> /usr/bin. >> Since I am not the administrator, I can't do anything to remove/update it. >> > > Specify the compilers using --with-cc --with-cxx --with-fc. I suggest you > specify the MPI > compiler wrappers for this machine. > > Matt > > >> The cmake configure file for metis tries to find c++ compiler, and >> reported error with that g++. >> >> Error running configure on METIS: Could not execute "cd >> /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && >> make distclean && make config >> prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" >> doubleprecision=1": >> rm -rf build/AIX-000092FAD400 >> rm -rf build/AIX-000092FAD400 >> mkdir -p build/AIX-000092FAD400 >> cd build/AIX-000092FAD400 && cmake >> /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 >> -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= >> -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP >> -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 >> -- The C compiler identification is XL >> -- The CXX compiler identification is unknown >> -- Check for working C compiler: /usr/bin/mpcc_r >> -- Check for working C compiler: /usr/bin/mpcc_r -- works >> -- Detecting C compiler ABI info >> -- Detecting C compiler ABI info - done >> -- Check for working CXX compiler: /usr/bin/g++ >> -- Check for working CXX compiler: /usr/bin/g++ -- broken >> -- Configuring incomplete, errors occurred! >> >> I think the testing for c++ compiler can be removed here? >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Mon Dec 5 07:11:59 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 5 Dec 2011 21:11:59 +0800 (CST) Subject: [petsc-users] cmake METIS problem In-Reply-To: References: Message-ID: <14239141.357851323090719334.JavaMail.coremail@mail.ustc.edu> No, petsc filters all the FC, CC and CXX env variable and gives a warning. p { margin: 0; }You can set the CXX environment variable, CMake should always pick those up. Tim From:"Gong Ding" To:"PETSc users list" Sent:Monday, December 5, 2011 7:58:29 AM Subject:Re: [petsc-users] cmake METIS problem I do specify --with-cc --with-cxx --with-fc. Here is the configure options configure --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=2 --known-memcmp-ok=1 --known-endian=big --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --download-f-blas-lapack=1 --download-mumps=1 --download-blacs=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-superlu=1 --download-superlu_dist=1 --with-c2html=0 --with-debugging=0 --with-clanguage=cxx --with-cc="mpcc_r" --with-fc="mpxlf_r" --with-cxx="mpCC_r" --CFLAGS="-q64 -O3 -qmaxmem=-1 -qstrict -qarch=auto -qtune=auto" --CXXFLAGS="-q64 -O3 -qmaxmem=-1 -qipa -qstrict -qarch=auto -qtune=auto" --FFLAGS="-q64 -O3 -qmaxmem=-1 -qarch=auto -qtune=auto" --with-sowing=0 --with-batch=1 --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-x=0 --with-pic=1 But cmake does not use given cxx. You see, cc="mpcc_r" is passed to cmake, but no cxx. Cmake tries to find cxx but failed. Hi, I try to install petsc-dev on AIX/PPC cluster. I'd like to use xlc, however, there is a dead gcc/g++ installed in /usr/bin. Since I am not the administrator, I can't do anything to remove/update it. Specify the compilers using --with-cc --with-cxx --with-fc. I suggest you specify the MPIcompiler wrappers for this machine. Matt The cmake configure file for metis tries to find c++ compiler, and reported error with that g++. Error running configure on METIS: Could not execute "cd /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 && make distclean && make config prefix=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP cc="mpcc_r" doubleprecision=1": rm -rf build/AIX-000092FAD400 rm -rf build/AIX-000092FAD400 mkdir -p build/AIX-000092FAD400 cd build/AIX-000092FAD400 && cmake /gpfs1/cogenda/cogenda/packages/petsc-dev/externalpackages/metis-5.0.2 -DCMAKE_VERBOSE_MAKEFILE=1 -DGKLIB_PATH= -DCMAKE_INSTALL_PREFIX=/gpfs1/cogenda/cogenda/packages/petsc-dev/IBM-XLCPP -DCMAKE_C_COMPILER=mpcc_r -DMETIS_USE_DOUBLEPRECISION=1 -- The C compiler identification is XL -- The CXX compiler identification is unknown -- Check for working C compiler: /usr/bin/mpcc_r -- Check for working C compiler: /usr/bin/mpcc_r -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/g++ -- Check for working CXX compiler: /usr/bin/g++ -- broken -- Configuring incomplete, errors occurred! I think the testing for c++ compiler can be removed here? -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 5 08:17:06 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 5 Dec 2011 15:17:06 +0100 Subject: [petsc-users] unsure about ex19 test results Message-ID: I am on a Cray system, where I have configured and built petsc 3.2 p5 myself, only using the vendor provided mpi, all the other packages (parmetis, hypre, mumps) are built with --download-package. "Make ... test" does not work here to verify the correct build because users do not have access to mpiexec. So I do the test by hand, and I am unsure about what I am getting. When executing the test ex19 with the system's scheduler I get this output: lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 Number of Newton iterations = 2 lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 Number of Newton iterations = 2 It's quite different from the expected output in ex19_1.testout: lid velocity = 0.0016, prandtl # = 1, grashof # = 1 Number of Newton iterations = 2 lid velocity = 0.0016, prandtl # = 1, grashof # = 1 Number of Newton iterations = 2 Does it mean that I have screwed up the build somehow, or is this somehow the effect of the batching system, or just numerical uncertainty due to the parallel computation? Note: I get the same result with 1 and a few MPI processes. Many thanks for any ideas. Dominik From knepley at gmail.com Mon Dec 5 08:24:43 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 08:24:43 -0600 Subject: [petsc-users] unsure about ex19 test results In-Reply-To: References: Message-ID: On Mon, Dec 5, 2011 at 8:17 AM, Dominik Szczerba wrote: > I am on a Cray system, where I have configured and built petsc 3.2 p5 > myself, only using the vendor provided mpi, all the other packages > (parmetis, hypre, mumps) are built with --download-package. "Make ... > test" does not work here to verify the correct build because users do > not have access to mpiexec. So I do the test by hand, and I am unsure > about what I am getting. When executing the test ex19 with the > system's scheduler I get this output: > > lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > > It's quite different from the expected output in ex19_1.testout: > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > lid velocity = 0.0016, prandtl # = 1, grashof # = 1 > Number of Newton iterations = 2 > Are you running with the right options? -dmmg_nlevels 4 -snes_monitor_short Matt > Does it mean that I have screwed up the build somehow, or is this > somehow the effect of the batching system, or just numerical > uncertainty due to the parallel computation? Note: I get the same > result with 1 and a few MPI processes. > > Many thanks for any ideas. > > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 5 08:32:31 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 5 Dec 2011 15:32:31 +0100 Subject: [petsc-users] unsure about ex19 test results In-Reply-To: References: Message-ID: Nice catch, thanks, no, I overlooked the extra options when reverse engineering the test from the makefiles. Now I get identical output. Thanks, Dominik On Mon, Dec 5, 2011 at 3:24 PM, Matthew Knepley wrote: > On Mon, Dec 5, 2011 at 8:17 AM, Dominik Szczerba > wrote: >> >> I am on a Cray system, where I have configured and built petsc 3.2 p5 >> myself, only using the vendor provided mpi, all the other packages >> (parmetis, hypre, mumps) are built with --download-package. "Make ... >> test" does not work here to verify the correct build because users do >> not have access to mpiexec. So I do the test by hand, and I am unsure >> about what I am getting. When executing the test ex19 with the >> system's scheduler I get this output: >> >> lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 >> Number of Newton iterations = 2 >> lid velocity = 0.0204082, prandtl # = 1, grashof # = 1 >> Number of Newton iterations = 2 >> >> It's quite different from the expected output in ex19_1.testout: >> >> lid velocity = 0.0016, prandtl # = 1, grashof # = 1 >> Number of Newton iterations = 2 >> lid velocity = 0.0016, prandtl # = 1, grashof # = 1 >> Number of Newton iterations = 2 > > > Are you running with the right options???-dmmg_nlevels 4 -snes_monitor_short > > ? Matt > >> >> Does it mean that I have screwed up the build somehow, or is this >> somehow the effect of the batching system, or just numerical >> uncertainty due to the parallel computation? Note: I get the same >> result with 1 and a few MPI processes. >> >> Many thanks for any ideas. >> >> Dominik > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From Debao.Shao at brion.com Mon Dec 5 08:46:47 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Mon, 5 Dec 2011 06:46:47 -0800 Subject: [petsc-users] Mat_CheckInode In-Reply-To: <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> Message-ID: <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> DA, 1, Will check_inode affect the runtime performance? For example, in my case, there are such different logs: [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using Inode routines [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode routines [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using Inode routines 2, Here is the log_summary, looks like PCSetup is big time-consuming function, the method I'm using is ILU(1) & GMRES, may I improve it? MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380 MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363 MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315 MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725 KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308 PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298 PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 The matrix size is around 50000*50000, nnz is ~2000000 3, Will the fill of ilu affect the runtime performance? [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 needed 1.11111 [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 needed 5.05 [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 needed 8.3410 How to set a proper value for fill, >8? Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 5 08:54:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 08:54:52 -0600 Subject: [petsc-users] Mat_CheckInode In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> Message-ID: On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: > DA, > > 1, Will check_inode affect the runtime performance? For example, in my > case, there are such different logs:**** > > [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using > Inode routines**** > > [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode > routines**** > > [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using > Inode routines > This is intended to "discover" block structure. If you have it, MatMult should be faster. > 2, Here is the log_summary, looks like PCSetup is big time-consuming > function, the method I?m using is ILU(1) & GMRES, may I improve it?**** > > MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 > 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380**** > > MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363**** > > MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 > 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382**** > > MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 > 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315**** > > MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0**** > > MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0**** > > MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0**** > > KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725**** > > KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0**** > > KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 > 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308**** > > PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 > 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298**** > > PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 > 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 **** > > The matrix size is around 50000*50000, nnz is ~2000000 > ILU is expensive, and levels make it much more expensive. Maybe try AMG? > 3, Will the fill of ilu affect the runtime performance?**** > > [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 > needed 1.11111**** > > [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 > needed 5.05**** > > [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 > needed 8.3410**** > > How to set a proper value for fill, >8? > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html Use -pc_factor_fill Matt > Thanks,**** > > Debao**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Dec 5 09:34:39 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 5 Dec 2011 09:34:39 -0600 Subject: [petsc-users] Mat_CheckInode In-Reply-To: References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> Message-ID: For >The matrix size is around 50000*50000, nnz is ~2000000 You may also try direct solver '-pc_type lu' either Petsc lu (sequential) or mumps, or superlu_dist. They may be faster than ilu(1). Hong On Mon, Dec 5, 2011 at 8:54 AM, Matthew Knepley wrote: > On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: >> >> DA, >> >> 1, Will check_inode affect the runtime performance? For example, in my >> case, there are such different logs: >> >> ???? [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using >> Inode routines >> >> ???? [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode >> routines >> >> ???? [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using >> Inode routines > > > This is intended to "discover" block structure. If you have it, MatMult > should be faster. > >> >> 2, Here is the log_summary, looks like PCSetup is big time-consuming >> function, the method I?m using is ILU(1) & GMRES, may I improve it? >> >> ???? MatMult??????????? 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00? 2? 4? 0? 0? 0?? 2? 4? 0? 0? 0?? 380 >> >> ???? MatMultAdd????????? 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 >> 0.0e+00 0.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0?? 363 >> >> ???? MatSolve?????????? 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00? 6 10? 0? 0? 0?? 6 10? 0? 0? 0?? 382 >> >> ???? MatLUFactorNum????? 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 61 85? 0? 0? 0? 61 85? 0? 0? 0?? 315 >> >> ???? MatILUFactorSym????? 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00? 3? 0? 0? 0? 0?? 3? 0? 0? 0? 0???? 0 >> >> ???? MatAssemblyBegin??? 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0???? 0 >> >> ???? MatAssemblyEnd????? 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0???? 0 >> >> ???? KSPGMRESOrthog???? 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 >> 0.0e+00 0.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0?? 725 >> >> ???? KSPSetup??????????? 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00? 0? 0? 0? 0? 0?? 0? 0? 0? 0? 0???? 0 >> >> ???? KSPSolve??????????? 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 73100? 0? 0? 0? 73100? 0? 0? 0?? 308 >> >> ???? PCSetUp???????????? 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 64 85? 0? 0? 0? 64 85? 0? 0? 0?? 298 >> >> ???? PCApply??????????? 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00? 6 10? 0? 0? 0?? 6 10? 0? 0? 0?? 382 >> >> The matrix size is around 50000*50000, nnz is ~2000000 > > > ILU is expensive, and levels make it much more expensive. Maybe try AMG? >> >> 3, Will the fill of ilu affect the runtime performance? >> >> ???? [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 >> needed 1.11111 >> >> ???? [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >> needed 5.05 >> >> ???? [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >> needed 8.3410 >> >> How to set a proper value for fill, >8? > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html > > Use -pc_factor_fill > > ? Matt > >> >> Thanks, >> >> Debao >> >> >> ________________________________ >> -- The information contained in this communication and any attachments is >> confidential and may be privileged, and is for the sole use of the intended >> recipient(s). Any unauthorized review, use, disclosure or distribution is >> prohibited. Unless explicitly stated otherwise in the body of this >> communication or the attachment thereto (if any), the information is >> provided on an AS-IS basis without any express or implied warranties or >> liabilities. To the extent you are relying on this information, you are >> doing so at your own risk. If you are not the intended recipient, please >> notify the sender immediately by replying to this message and destroy all >> copies of this message and any attachments. ASML is neither liable for the >> proper and complete transmission of the information contained in this >> communication, nor for any delay in its receipt. > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From sean at mcs.anl.gov Mon Dec 5 10:33:15 2011 From: sean at mcs.anl.gov (Sean Farley) Date: Mon, 5 Dec 2011 10:33:15 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> Message-ID: > > Yes, petsc-dev works. Please merge the update to petsc-3.2. Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to work with it. In general, software projects don't update a previous release on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at risoe.dtu.dk Mon Dec 5 11:55:05 2011 From: frtr at risoe.dtu.dk (Treue, Frederik) Date: Mon, 5 Dec 2011 18:55:05 +0100 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: > Ghosted has nothing to do with MPI communication! Ghosted vs None is only about the physical boundary, nothing to do with the boundary between domains. > Ghosted is only for the (nonlinear) function evaluation. It is not for storage of the Jacobian/matrix. > I suggest just using NONE and make your life and understanding easier. You can do Neuman and Dirichlet boundary conditions with none just fine. OK, I tried this, but as far as I can see, GHOSTED and NONE _does_ effect the MPI communication. I'm setting up a simple FD matrix for a first order differncial operator in (say) the x direction (periodic in the y direction), which should have dirichlet boundary conditions, ie. 0 0 0 0 0 0 -1 0 1 0 0 0 0 -1 0 1 0 0 0 0 -1 0 1 0 0 0 0 -1 0 1 0 0 0 0 0 0 Where the first and last row will be set later for BC. This works perfectly fine in both sequencial and GHOSTED, but fails with NONE - see the attached output files, which is simply from MatView on a 12x10 grid (the 0th and 11th columns will be ghostpoints). NOTHING has been changed between the two runs, but notice the spurious generation of elements in the middle of every (grid)row, ie. In row 6,18,30, etc. where the point in the middle of the (grid)row is connected to the next (grid)row - notice that the mpi division is exactly between 6 and 7, 18 and 19, 30 and 31 and so on, which makes me suspect that there is a connection to the MPI communication. Is there a logical explanation for this? It looks like a bug to me, but then, I haven't quite understood what the difference between GHOSTED and NONE is. --- yours sincerily Frederik Treue -------------- next part -------------- A non-text attachment was scrubbed... Name: NONEout Type: application/octet-stream Size: 10470 bytes Desc: NONEout URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GHOSTout Type: application/octet-stream Size: 10380 bytes Desc: GHOSTout URL: From knepley at gmail.com Mon Dec 5 12:12:21 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 Dec 2011 12:12:21 -0600 Subject: [petsc-users] newbie question on the parallel allocation of matrices In-Reply-To: References: Message-ID: On Mon, Dec 5, 2011 at 11:55 AM, Treue, Frederik wrote: > > Ghosted has nothing to do with MPI communication! Ghosted vs None is > only about the physical boundary, nothing to do with the boundary between > domains. > > > Ghosted is only for the (nonlinear) function evaluation. It is not > for storage of the Jacobian/matrix. > > > I suggest just using NONE and make your life and understanding easier. > You can do Neuman and Dirichlet boundary conditions with none just fine. > > > OK, I tried this, but as far as I can see, GHOSTED and NONE _does_ effect > the MPI communication. I'm setting up a simple FD matrix for a first order > differncial operator in (say) the x direction (periodic in the y > direction), which should have dirichlet boundary conditions, ie. > 1) I can't make any sense of what you are doing. If you describe it carefully, maybe this can be puzzled out. 2) NONE is very simple, it does nothing for boundaries, so there should be no surprises 3) When showing an example, make it as small as possible, and run on 2 procs. That makes everything obvious. I guarantee you, there are no bugs in this code. This is an understanding issue. Matt > 0 0 0 0 0 0 > -1 0 1 0 0 0 > 0 -1 0 1 0 0 > 0 0 -1 0 1 0 > 0 0 0 -1 0 1 > 0 0 0 0 0 0 > > Where the first and last row will be set later for BC. This works > perfectly fine in both sequencial and GHOSTED, but fails with NONE - see > the attached output files, which is simply from MatView on a 12x10 grid > (the 0th and 11th columns will be ghostpoints). > NOTHING has been changed between the two runs, but notice the spurious > generation of elements in the middle of every (grid)row, ie. In row > 6,18,30, etc. where the point in the middle of the (grid)row is connected > to the next (grid)row - notice that the mpi division is exactly between 6 > and 7, 18 and 19, 30 and 31 and so on, which makes me suspect that there is > a connection to the MPI communication. > > Is there a logical explanation for this? It looks like a bug to me, but > then, I haven't quite understood what the difference between GHOSTED and > NONE is. > > --- > yours sincerily > Frederik Treue > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Mon Dec 5 20:24:23 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Mon, 5 Dec 2011 18:24:23 -0800 Subject: [petsc-users] Mat_CheckInode In-Reply-To: References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E2666B4C0@EX03> Hi, Hong: Thanks a lot for your suggestion. The matrix scale of my case may vary from 10*10~1000000*1000000, so, I need both direct solver & iterative solver. For example, 1), <100*100, use direct dense solver; 2), <10000*10000, use direct sparse solver; 3), else, use iterative solver. Iterative solver(ILU(1) & GMRES) is very efficient on many large scale problems, the issue I encounter now is its runtime's unstable. For a 50000*50000 matrix, it may only need dozens of seconds, but sometimes it also may take thousands of seconds. Do you have any suggestion? Thanks, Debao -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Monday, December 05, 2011 11:35 PM To: PETSc users list Subject: Re: [petsc-users] Mat_CheckInode For >The matrix size is around 50000*50000, nnz is ~2000000 You may also try direct solver '-pc_type lu' either Petsc lu (sequential) or mumps, or superlu_dist. They may be faster than ilu(1). Hong On Mon, Dec 5, 2011 at 8:54 AM, Matthew Knepley wrote: > On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: >> >> DA, >> >> 1, Will check_inode affect the runtime performance? For example, in my >> case, there are such different logs: >> >> [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using >> Inode routines >> >> [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode >> routines >> >> [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using >> Inode routines > > > This is intended to "discover" block structure. If you have it, MatMult > should be faster. > >> >> 2, Here is the log_summary, looks like PCSetup is big time-consuming >> function, the method I'm using is ILU(1) & GMRES, may I improve it? >> >> MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380 >> >> MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363 >> >> MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >> >> MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315 >> >> MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >> >> MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725 >> >> KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 >> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308 >> >> PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 >> 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298 >> >> PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 >> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >> >> The matrix size is around 50000*50000, nnz is ~2000000 > > > ILU is expensive, and levels make it much more expensive. Maybe try AMG? >> >> 3, Will the fill of ilu affect the runtime performance? >> >> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 >> needed 1.11111 >> >> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >> needed 5.05 >> >> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >> needed 8.3410 >> >> How to set a proper value for fill, >8? > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html > > Use -pc_factor_fill > > Matt > >> >> Thanks, >> >> Debao >> >> >> ________________________________ >> -- The information contained in this communication and any attachments is >> confidential and may be privileged, and is for the sole use of the intended >> recipient(s). Any unauthorized review, use, disclosure or distribution is >> prohibited. Unless explicitly stated otherwise in the body of this >> communication or the attachment thereto (if any), the information is >> provided on an AS-IS basis without any express or implied warranties or >> liabilities. To the extent you are relying on this information, you are >> doing so at your own risk. If you are not the intended recipient, please >> notify the sender immediately by replying to this message and destroy all >> copies of this message and any attachments. ASML is neither liable for the >> proper and complete transmission of the information contained in this >> communication, nor for any delay in its receipt. > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From bsmith at mcs.anl.gov Mon Dec 5 20:35:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 5 Dec 2011 20:35:27 -0600 Subject: [petsc-users] Mat_CheckInode In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E2666B4C0@EX03> References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> <384FF55F15E3E447802DC8CCA85696980E2666B4C0@EX03> Message-ID: <6ABDCA90-F914-48AD-A8CA-B1EDAB8211DD@mcs.anl.gov> On Dec 5, 2011, at 8:24 PM, Debao Shao wrote: > Hi, Hong: > > Thanks a lot for your suggestion. > The matrix scale of my case may vary from 10*10~1000000*1000000, so, I need both direct solver & iterative solver. > For example, > 1), <100*100, use direct dense solver; > 2), <10000*10000, use direct sparse solver; > 3), else, use iterative solver. > > Iterative solver(ILU(1) & GMRES) is very efficient on many large scale problems, the issue I encounter now is its runtime's unstable. For a 50000*50000 matrix, it may only need dozens of seconds, but sometimes it also may take thousands of seconds. It shouldn't. There must be some issue with uncontrolled ill-conditioning on the matrix to it to take that long. If it truly is that badly conditioned then you may want a direct solver even at that size. http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged Barry > > Do you have any suggestion? > > Thanks, > Debao > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang > Sent: Monday, December 05, 2011 11:35 PM > To: PETSc users list > Subject: Re: [petsc-users] Mat_CheckInode > > For >> The matrix size is around 50000*50000, nnz is ~2000000 > > You may also try direct solver '-pc_type lu' either Petsc lu (sequential) > or mumps, or superlu_dist. > They may be faster than ilu(1). > > Hong > > On Mon, Dec 5, 2011 at 8:54 AM, Matthew Knepley wrote: >> On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: >>> >>> DA, >>> >>> 1, Will check_inode affect the runtime performance? For example, in my >>> case, there are such different logs: >>> >>> [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using >>> Inode routines >>> >>> [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode >>> routines >>> >>> [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using >>> Inode routines >> >> >> This is intended to "discover" block structure. If you have it, MatMult >> should be faster. >> >>> >>> 2, Here is the log_summary, looks like PCSetup is big time-consuming >>> function, the method I'm using is ILU(1) & GMRES, may I improve it? >>> >>> MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380 >>> >>> MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363 >>> >>> MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>> >>> MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315 >>> >>> MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>> >>> MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725 >>> >>> KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308 >>> >>> PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298 >>> >>> PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>> >>> The matrix size is around 50000*50000, nnz is ~2000000 >> >> >> ILU is expensive, and levels make it much more expensive. Maybe try AMG? >>> >>> 3, Will the fill of ilu affect the runtime performance? >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 >>> needed 1.11111 >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>> needed 5.05 >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>> needed 8.3410 >>> >>> How to set a proper value for fill, >8? >> >> >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html >> >> Use -pc_factor_fill >> >> Matt >> >>> >>> Thanks, >>> >>> Debao >>> >>> >>> ________________________________ >>> -- The information contained in this communication and any attachments is >>> confidential and may be privileged, and is for the sole use of the intended >>> recipient(s). Any unauthorized review, use, disclosure or distribution is >>> prohibited. Unless explicitly stated otherwise in the body of this >>> communication or the attachment thereto (if any), the information is >>> provided on an AS-IS basis without any express or implied warranties or >>> liabilities. To the extent you are relying on this information, you are >>> doing so at your own risk. If you are not the intended recipient, please >>> notify the sender immediately by replying to this message and destroy all >>> copies of this message and any attachments. ASML is neither liable for the >>> proper and complete transmission of the information contained in this >>> communication, nor for any delay in its receipt. >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From Debao.Shao at brion.com Mon Dec 5 20:42:55 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Mon, 5 Dec 2011 18:42:55 -0800 Subject: [petsc-users] Mat_CheckInode In-Reply-To: <6ABDCA90-F914-48AD-A8CA-B1EDAB8211DD@mcs.anl.gov> References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> <384FF55F15E3E447802DC8CCA85696980E2666B4C0@EX03> <6ABDCA90-F914-48AD-A8CA-B1EDAB8211DD@mcs.anl.gov> Message-ID: <384FF55F15E3E447802DC8CCA85696980E2666B4D1@EX03> Yes, for the slow case, most of times, it will be diverged. But, the problem is it takes long runtime before it give an unconverged answer. >From your experience, how large condition number shouldn't I use iterative solver? The problem I'm solving is a QP problem(symmetry & semi-positive definite) with constraints. Any suggestions? Thanks, Debao -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Tuesday, December 06, 2011 10:35 AM To: PETSc users list Subject: Re: [petsc-users] Mat_CheckInode On Dec 5, 2011, at 8:24 PM, Debao Shao wrote: > Hi, Hong: > > Thanks a lot for your suggestion. > The matrix scale of my case may vary from 10*10~1000000*1000000, so, I need both direct solver & iterative solver. > For example, > 1), <100*100, use direct dense solver; > 2), <10000*10000, use direct sparse solver; > 3), else, use iterative solver. > > Iterative solver(ILU(1) & GMRES) is very efficient on many large scale problems, the issue I encounter now is its runtime's unstable. For a 50000*50000 matrix, it may only need dozens of seconds, but sometimes it also may take thousands of seconds. It shouldn't. There must be some issue with uncontrolled ill-conditioning on the matrix to it to take that long. If it truly is that badly conditioned then you may want a direct solver even at that size. http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged Barry > > Do you have any suggestion? > > Thanks, > Debao > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang > Sent: Monday, December 05, 2011 11:35 PM > To: PETSc users list > Subject: Re: [petsc-users] Mat_CheckInode > > For >> The matrix size is around 50000*50000, nnz is ~2000000 > > You may also try direct solver '-pc_type lu' either Petsc lu (sequential) > or mumps, or superlu_dist. > They may be faster than ilu(1). > > Hong > > On Mon, Dec 5, 2011 at 8:54 AM, Matthew Knepley wrote: >> On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: >>> >>> DA, >>> >>> 1, Will check_inode affect the runtime performance? For example, in my >>> case, there are such different logs: >>> >>> [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using >>> Inode routines >>> >>> [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode >>> routines >>> >>> [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using >>> Inode routines >> >> >> This is intended to "discover" block structure. If you have it, MatMult >> should be faster. >> >>> >>> 2, Here is the log_summary, looks like PCSetup is big time-consuming >>> function, the method I'm using is ILU(1) & GMRES, may I improve it? >>> >>> MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380 >>> >>> MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363 >>> >>> MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>> >>> MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315 >>> >>> MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>> >>> MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725 >>> >>> KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> >>> KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308 >>> >>> PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298 >>> >>> PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>> >>> The matrix size is around 50000*50000, nnz is ~2000000 >> >> >> ILU is expensive, and levels make it much more expensive. Maybe try AMG? >>> >>> 3, Will the fill of ilu affect the runtime performance? >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 >>> needed 1.11111 >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>> needed 5.05 >>> >>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>> needed 8.3410 >>> >>> How to set a proper value for fill, >8? >> >> >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html >> >> Use -pc_factor_fill >> >> Matt >> >>> >>> Thanks, >>> >>> Debao >>> >>> >>> ________________________________ >>> -- The information contained in this communication and any attachments is >>> confidential and may be privileged, and is for the sole use of the intended >>> recipient(s). Any unauthorized review, use, disclosure or distribution is >>> prohibited. Unless explicitly stated otherwise in the body of this >>> communication or the attachment thereto (if any), the information is >>> provided on an AS-IS basis without any express or implied warranties or >>> liabilities. To the extent you are relying on this information, you are >>> doing so at your own risk. If you are not the intended recipient, please >>> notify the sender immediately by replying to this message and destroy all >>> copies of this message and any attachments. ASML is neither liable for the >>> proper and complete transmission of the information contained in this >>> communication, nor for any delay in its receipt. >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From bsmith at mcs.anl.gov Mon Dec 5 20:47:45 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 5 Dec 2011 20:47:45 -0600 Subject: [petsc-users] Mat_CheckInode In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E2666B4D1@EX03> References: <384FF55F15E3E447802DC8CCA85696980E26254051@EX03> <128B46D5-40E7-4E55-BE7F-9A7B13AEFF73@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B3D8@EX03> <384FF55F15E3E447802DC8CCA85696980E2666B4C0@EX03> <6ABDCA90-F914-48AD-A8CA-B1EDAB8211DD@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E2666B4D1@EX03> Message-ID: <6B33D00F-899D-4B2B-BDE4-FF9D025A74D3@mcs.anl.gov> On Dec 5, 2011, at 8:42 PM, Debao Shao wrote: > Yes, for the slow case, most of times, it will be diverged. But, the problem is it takes long runtime before it give an unconverged answer. > >> From your experience, how large condition number shouldn't I use iterative solver? > > The problem I'm solving is a QP problem(symmetry & semi-positive definite) with constraints. Direct solver. By having the constraints this makes it much less likely that the "canned" iterative solvers will work. You need to read up on iterative solvers for your type of problem which likely requiring treating different variables differently before exploring iterative methods. Barry > > Any suggestions? > > Thanks, > Debao > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Tuesday, December 06, 2011 10:35 AM > To: PETSc users list > Subject: Re: [petsc-users] Mat_CheckInode > > > On Dec 5, 2011, at 8:24 PM, Debao Shao wrote: > >> Hi, Hong: >> >> Thanks a lot for your suggestion. >> The matrix scale of my case may vary from 10*10~1000000*1000000, so, I need both direct solver & iterative solver. >> For example, >> 1), <100*100, use direct dense solver; >> 2), <10000*10000, use direct sparse solver; >> 3), else, use iterative solver. >> >> Iterative solver(ILU(1) & GMRES) is very efficient on many large scale problems, the issue I encounter now is its runtime's unstable. For a 50000*50000 matrix, it may only need dozens of seconds, but sometimes it also may take thousands of seconds. > > It shouldn't. There must be some issue with uncontrolled ill-conditioning on the matrix to it to take that long. If it truly is that badly conditioned then you may want a direct solver even at that size. http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged > > > Barry > >> >> Do you have any suggestion? >> >> Thanks, >> Debao >> >> -----Original Message----- >> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang >> Sent: Monday, December 05, 2011 11:35 PM >> To: PETSc users list >> Subject: Re: [petsc-users] Mat_CheckInode >> >> For >>> The matrix size is around 50000*50000, nnz is ~2000000 >> >> You may also try direct solver '-pc_type lu' either Petsc lu (sequential) >> or mumps, or superlu_dist. >> They may be faster than ilu(1). >> >> Hong >> >> On Mon, Dec 5, 2011 at 8:54 AM, Matthew Knepley wrote: >>> On Mon, Dec 5, 2011 at 8:46 AM, Debao Shao wrote: >>>> >>>> DA, >>>> >>>> 1, Will check_inode affect the runtime performance? For example, in my >>>> case, there are such different logs: >>>> >>>> [0] Mat_CheckInode(): Found 18602 nodes out of 18609 rows. Not using >>>> Inode routines >>>> >>>> [0] Mat_CheckInode(): Found 0 nodes of 0. Limit used: 5. Using Inode >>>> routines >>>> >>>> [0] Mat_CheckInode(): Found 14020 nodes out of 14020 rows. Not using >>>> Inode routines >>> >>> >>> This is intended to "discover" block structure. If you have it, MatMult >>> should be faster. >>> >>>> >>>> 2, Here is the log_summary, looks like PCSetup is big time-consuming >>>> function, the method I'm using is ILU(1) & GMRES, may I improve it? >>>> >>>> MatMult 30733 1.0 8.9553e+01 1.0 3.41e+10 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 380 >>>> >>>> MatMultAdd 9552 1.0 6.6610e+00 1.0 2.42e+09 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 363 >>>> >>>> MatSolve 36236 1.0 2.4699e+02 1.0 9.44e+10 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>>> >>>> MatLUFactorNum 2838 1.0 2.4328e+03 1.0 7.65e+11 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 61 85 0 0 0 61 85 0 0 0 315 >>>> >>>> MatILUFactorSym 173 1.0 1.3486e+02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>>> >>>> MatAssemblyBegin 5787 1.0 2.0547e-03 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> >>>> MatAssemblyEnd 5787 1.0 5.3559e+00 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> >>>> KSPGMRESOrthog 30679 1.0 5.5968e+00 1.0 4.06e+09 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 725 >>>> >>>> KSPSetup 2838 1.0 1.8219e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> >>>> KSPSolve 5503 1.0 2.9136e+03 1.0 8.99e+11 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 73100 0 0 0 73100 0 0 0 308 >>>> >>>> PCSetUp 2838 1.0 2.5682e+03 1.0 7.65e+11 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 64 85 0 0 0 64 85 0 0 0 298 >>>> >>>> PCApply 36236 1.0 2.4709e+02 1.0 9.44e+10 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 6 10 0 0 0 6 10 0 0 0 382 >>>> >>>> The matrix size is around 50000*50000, nnz is ~2000000 >>> >>> >>> ILU is expensive, and levels make it much more expensive. Maybe try AMG? >>>> >>>> 3, Will the fill of ilu affect the runtime performance? >>>> >>>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 0 Fill ratio:given 1 >>>> needed 1.11111 >>>> >>>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>>> needed 5.05 >>>> >>>> [0] MatILUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 1 >>>> needed 8.3410 >>>> >>>> How to set a proper value for fill, >8? >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetFill.html >>> >>> Use -pc_factor_fill >>> >>> Matt >>> >>>> >>>> Thanks, >>>> >>>> Debao >>>> >>>> >>>> ________________________________ >>>> -- The information contained in this communication and any attachments is >>>> confidential and may be privileged, and is for the sole use of the intended >>>> recipient(s). Any unauthorized review, use, disclosure or distribution is >>>> prohibited. Unless explicitly stated otherwise in the body of this >>>> communication or the attachment thereto (if any), the information is >>>> provided on an AS-IS basis without any express or implied warranties or >>>> liabilities. To the extent you are relying on this information, you are >>>> doing so at your own risk. If you are not the intended recipient, please >>>> notify the sender immediately by replying to this message and destroy all >>>> copies of this message and any attachments. ASML is neither liable for the >>>> proper and complete transmission of the information contained in this >>>> communication, nor for any delay in its receipt. >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From xiaohl1986 at gmail.com Mon Dec 5 20:59:14 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Tue, 6 Dec 2011 10:59:14 +0800 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: Hi I am solving an elliptic problem with Neumann boundary condition with dmda routines by snes solver. I set up the residue by DMDASetLocalFunction. Besides the way like fix the first value to be one by myself, is there way to do it by petsc routine? like the KSPSetNullSpace() rountine does? -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Dec 5 21:36:28 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 5 Dec 2011 19:36:28 -0800 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: There is not a PETSc function to set Dirichlet conditions. There are many ways to do it, you should pick a way they works well for your problem. I think you might misunderstand KSPSetNullSpace. On Dec 5, 2011 6:59 PM, "Hailong Xiao" wrote: > Hi > > I am solving an elliptic problem with Neumann boundary condition with dmda > routines by snes solver. > I set up the residue by DMDASetLocalFunction. > Besides the way like fix the first value to be one by myself, is there way > to do it by petsc routine? like the KSPSetNullSpace() rountine does? > > -- > Hailong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 5 21:51:54 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 5 Dec 2011 21:51:54 -0600 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: <2CF2F1FD-B04E-4C83-B42F-CCD92496F800@mcs.anl.gov> See src/ksp/ksp/examples/tutorials in petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html If you attach the null space to the matrix then the KSP objects automatically pull them out of the matrix and use them as needed. You don't need to set the value at a single point to fix it. Note it is much simplier in petsc-dev so switch to it. Barry On Dec 5, 2011, at 8:59 PM, Hailong Xiao wrote: > Hi > > I am solving an elliptic problem with Neumann boundary condition with dmda routines by snes solver. > I set up the residue by DMDASetLocalFunction. > Besides the way like fix the first value to be one by myself, is there way to do it by petsc routine? like the KSPSetNullSpace() rountine does? > > -- > Hailong From xiaohl1986 at gmail.com Mon Dec 5 22:10:07 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Mon, 5 Dec 2011 22:10:07 -0600 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: Hi Jed I am not going to set Dirichlet conditions. I just look at the webpage (maybe you wrote ) someone recommended http://www.cfd-online.com/Forums/main/15709-singular-linear-system-pressure-poisson-equation.html "You need to remove the null space. The best way is to tell the Krylov solver about this (see KSPSetNullSpace(...) in PETSc) but you can also fix the pressure at one point. That is, pick any row of your matrix and set it to a row of the identity and put 0 on the right hand side. This should also make the matrix nonsingular, but it disrupts the spectrum so may effect convergence rates (highly preconditioner dependent). " Since I am studying the performance of the preconditioners, it would be great for me that the spectrum would not be disrupted. It seems like KSPSetNullSpace(...) could better handle it. My code only has the application of the matrix not forming the matrix explicitly. Maybe I just fix one value is 1. Hailong On Mon, Dec 5, 2011 at 9:36 PM, Jed Brown wrote: > There is not a PETSc function to set Dirichlet conditions. There are many > ways to do it, you should pick a way they works well for your problem. > > I think you might misunderstand KSPSetNullSpace. > On Dec 5, 2011 6:59 PM, "Hailong Xiao" wrote: > >> Hi >> >> I am solving an elliptic problem with Neumann boundary condition >> with dmda routines by snes solver. >> I set up the residue by DMDASetLocalFunction. >> Besides the way like fix the first value to be one by myself, is there >> way to do it by petsc routine? like the KSPSetNullSpace() rountine does? >> >> -- >> Hailong >> > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Dec 5 22:14:03 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 5 Dec 2011 20:14:03 -0800 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: Ah, okay. With petsc-dev, just call MatSetNullSpace(). With earlier versions, SNESGetKSP, KSPSetNullSpace. On Dec 5, 2011 8:10 PM, "Hailong Xiao" wrote: > Hi Jed > > I am not going to set Dirichlet conditions. > I just look at the webpage (maybe you wrote ) someone recommended > > > http://www.cfd-online.com/Forums/main/15709-singular-linear-system-pressure-poisson-equation.html > > "You need to remove the null space. The best way is to tell the Krylov > solver about this (see KSPSetNullSpace(...) in PETSc) but you can also fix > the pressure at one point. That is, pick any row of your matrix and set it > to a row of the identity and put 0 on the right hand side. This should also > make the matrix nonsingular, but it disrupts the spectrum so may effect > convergence rates (highly preconditioner dependent). " > > Since I am studying the performance of the preconditioners, it would be > great for me that the spectrum would not be disrupted. > It seems like KSPSetNullSpace(...) could better handle it. > My code only has the application of the matrix not forming the matrix > explicitly. > > Maybe I just fix one value is 1. > Hailong > > > On Mon, Dec 5, 2011 at 9:36 PM, Jed Brown wrote: > >> There is not a PETSc function to set Dirichlet conditions. There are many >> ways to do it, you should pick a way they works well for your problem. >> >> I think you might misunderstand KSPSetNullSpace. >> On Dec 5, 2011 6:59 PM, "Hailong Xiao" wrote: >> >>> Hi >>> >>> I am solving an elliptic problem with Neumann boundary condition >>> with dmda routines by snes solver. >>> I set up the residue by DMDASetLocalFunction. >>> Besides the way like fix the first value to be one by myself, is there >>> way to do it by petsc routine? like the KSPSetNullSpace() rountine does? >>> >>> -- >>> Hailong >>> >> > > > -- > Hailong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiaohl1986 at gmail.com Mon Dec 5 22:17:02 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Mon, 5 Dec 2011 22:17:02 -0600 Subject: [petsc-users] question about Neumann boundary condition In-Reply-To: References: Message-ID: Thank you. Hailong On Mon, Dec 5, 2011 at 10:14 PM, Jed Brown wrote: > Ah, okay. With petsc-dev, just call MatSetNullSpace(). With earlier > versions, SNESGetKSP, KSPSetNullSpace. > On Dec 5, 2011 8:10 PM, "Hailong Xiao" wrote: > >> Hi Jed >> >> I am not going to set Dirichlet conditions. >> I just look at the webpage (maybe you wrote ) someone recommended >> >> >> http://www.cfd-online.com/Forums/main/15709-singular-linear-system-pressure-poisson-equation.html >> >> "You need to remove the null space. The best way is to tell the Krylov >> solver about this (see KSPSetNullSpace(...) in PETSc) but you can also fix >> the pressure at one point. That is, pick any row of your matrix and set it >> to a row of the identity and put 0 on the right hand side. This should also >> make the matrix nonsingular, but it disrupts the spectrum so may effect >> convergence rates (highly preconditioner dependent). " >> >> Since I am studying the performance of the preconditioners, it would be >> great for me that the spectrum would not be disrupted. >> It seems like KSPSetNullSpace(...) could better handle it. >> My code only has the application of the matrix not forming the matrix >> explicitly. >> >> Maybe I just fix one value is 1. >> Hailong >> >> >> On Mon, Dec 5, 2011 at 9:36 PM, Jed Brown wrote: >> >>> There is not a PETSc function to set Dirichlet conditions. There are >>> many ways to do it, you should pick a way they works well for your problem. >>> >>> I think you might misunderstand KSPSetNullSpace. >>> On Dec 5, 2011 6:59 PM, "Hailong Xiao" wrote: >>> >>>> Hi >>>> >>>> I am solving an elliptic problem with Neumann boundary condition >>>> with dmda routines by snes solver. >>>> I set up the residue by DMDASetLocalFunction. >>>> Besides the way like fix the first value to be one by myself, is there >>>> way to do it by petsc routine? like the KSPSetNullSpace() rountine >>>> does? >>>> >>>> -- >>>> Hailong >>>> >>> >> >> >> -- >> Hailong >> > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Tue Dec 6 00:52:33 2011 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 6 Dec 2011 14:52:33 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> Message-ID: <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> Bad news. MUMPS still crash on AIX/PPC. I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 task 0: Segmentation fault. I guess the interface of petsc-mumps has some problem, i.e. alignment. Yes, petsc-dev works. Please merge the update to petsc-3.2. Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to work with it. In general, software projects don't update a previous release on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 6 05:03:02 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 6 Dec 2011 05:03:02 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> Message-ID: On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: > Bad news. > MUMPS still crash on AIX/PPC. > I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 > task 0: Segmentation fault. > I guess the interface of petsc-mumps has some problem, i.e. alignment. > Please get a stack trace with the debugger. Matt > Yes, petsc-dev works. Please merge the update to petsc-3.2. > > > Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to > work with it. In general, software projects don't update a previous release > on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Parker2 at baesystems.com Tue Dec 6 07:43:03 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Tue, 6 Dec 2011 13:43:03 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: Hi, So having got petsc 3.2 install and all its deps, I now get this?. [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size 7 Any thoughts? Cheers, Andy From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: 02 December 2011 12:42 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Fri, Dec 2, 2011 at 06:40, Parker, Andrew (UK Filton) wrote: [0]PETSC ERROR: MatInvertBlockDiagonal_SeqBAIJ() line 129 in src/mat/impls/baij/seq/baij.c not supported for block size 7 This was fixed in petsc-3.2 ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 6 12:59:38 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 6 Dec 2011 12:59:38 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: On Tue, Dec 6, 2011 at 7:43 AM, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > Hi,**** > > ** ** > > So having got petsc 3.2 install and all its deps, I now get this?.**** > > ** ** > > [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in > src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size 7**** > > ** ** > > ** ** > > Any thoughts? > Sorry about that. There was confusion on our end. I have added the code you need to 3.2. It will go out with the next patch release. Alternatively, you can retrieve the source tarball from the repository site or use the repository directly. Thanks, Matt > > > Cheers,**** > > Andy**** > > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* 02 December 2011 12:42 > *To:* PETSc users list > *Subject:* Re: [petsc-users] MATBDIAG in > 3.1**** > > ** ** > > *** WARNING *** > > This message has originated outside your organisation, > either from an external partner or the Global Internet. > Keep this in mind if you answer this message.**** > > On Fri, Dec 2, 2011 at 06:40, Parker, Andrew (UK Filton) < > Andrew.Parker2 at baesystems.com> wrote:**** > > [0]PETSC ERROR: MatInvertBlockDiagonal_SeqBAIJ() line 129 in > src/mat/impls/baij/seq/baij.c not supported for block size 7**** > > ** ** > > This was fixed in petsc-3.2**** > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Dec 6 13:30:57 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 6 Dec 2011 13:30:57 -0600 (CST) Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: On Tue, 6 Dec 2011, Matthew Knepley wrote: > On Tue, Dec 6, 2011 at 7:43 AM, Parker, Andrew (UK Filton) < > Andrew.Parker2 at baesystems.com> wrote: > > > So having got petsc 3.2 install and all its deps, I now get this?.**** > > > > ** ** > > > > [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in > > src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size 7**** > > > > ** ** > > > > ** ** > > > > Any thoughts? > > > > Sorry about that. There was confusion on our end. I have added the code you > need to 3.2. It will > go out with the next patch release. Alternatively, you can retrieve the > source tarball from the > repository site or use the repository directly. I'm attaching this change as a patchfile - which can be applied to the current petsc-3.2 sources you have. cd petsc-3.2 patch -Np1 < bpjacobi.patch Satish -------------- next part -------------- tree baf1a1164d63 parent d7b881216ffe author Matthew G Knepley 1323197905 21600 committer Matthew G Knepley 1323197905 21600 revision 20397 branch default Added support for block size 7 to PBJACOBI diff --git a/src/ksp/pc/impls/pbjacobi/pbjacobi.c b/src/ksp/pc/impls/pbjacobi/pbjacobi.c --- a/src/ksp/pc/impls/pbjacobi/pbjacobi.c +++ b/src/ksp/pc/impls/pbjacobi/pbjacobi.c @@ -169,6 +169,35 @@ ierr = PetscLogFlops(66.0*m);CHKERRQ(ierr); PetscFunctionReturn(0); } +#undef __FUNCT__ +#define __FUNCT__ "PCApply_PBJacobi_7" +static PetscErrorCode PCApply_PBJacobi_7(PC pc,Vec x,Vec y) +{ + PC_PBJacobi *jac = (PC_PBJacobi*)pc->data; + PetscErrorCode ierr; + PetscInt i,m = jac->mbs; + const MatScalar *diag = jac->diag; + PetscScalar x0,x1,x2,x3,x4,x5,x6,*xx,*yy; + + PetscFunctionBegin; + ierr = VecGetArray(x,&xx);CHKERRQ(ierr); + ierr = VecGetArray(y,&yy);CHKERRQ(ierr); + for (i=0; iops->apply = PCApply_PBJacobi_6; break; + case 7: + pc->ops->apply = PCApply_PBJacobi_7; + break; default: SETERRQ1(((PetscObject)pc)->comm,PETSC_ERR_SUP,"not supported for block size %D",jac->bs); } From livne at uchicago.edu Tue Dec 6 17:08:13 2011 From: livne at uchicago.edu (Oren Livne) Date: Tue, 06 Dec 2011 17:08:13 -0600 Subject: [petsc-users] PetSc + MATLAB installation problem on Debian Squeeze Message-ID: <4EDEA05D.5000506@uchicago.edu> Dear All, I'm trying to install PetSc with the MATLAB interface on Debian Squeeze 6.26. "make all test" fails on an ld error due to a MATLAB library. Running the MATLAB library seems to cause a seg fault even though matlab runs fine (7.13 / R2011b). Could you kindly help? Output attached below. Thanks so much, Oren ------------------------------------------------------------------------------ $ uname -a Linux vm-debian 2.6.32-5-amd64 #1 SMP Mon Oct 3 03:59:20 UTC 2011 x86_64 GNU/Linux $ ./configure --with-shared-libraries --with-matlab-engine --with-matlab --download-f2cblaslapack --with-mpi=0 ... OK $ make all test ... libfast in: /home/oren/petsc-3.2-p5/tutorials/multiphysics Completed building libraries ========================================= making shared libraries in /home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib building libpetsc.so /usr/bin/ld: cannot find -licudata collect2: ld returned 1 exit status ========================================= Now to check if the libraries are working do: make PETSC_DIR=/home/oren/petsc-3.2-p5 PETSC_ARCH=arch-linux2-c-debug test ========================================= Running test examples to verify correct installation Using PETSC_DIR=/home/oren/petsc-3.2-p5 and PETSC_ARCH=arch-linux2-c-debug --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html gcc -o ex19.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -I/home/oren/petsc-3.2-p5/include -I/home/oren/petsc-3.2-p5/arch-linux2-c-debug/include -I/opt/MATLAB/R2011b/extern/include -I/home/oren/petsc-3.2-p5/include/mpiuni -D__INSDIR__=src/snes/examples/tutorials/ ex19.c gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ex19 ex19.o -Wl,-rpath,/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -L/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/opt/MATLAB/R2011b/sys/os/glnxa64:/opt/MATLAB/R2011b/bin/glnxa64:/opt/MATLAB/R2011b/extern/lib/glnxa64 -L/opt/MATLAB/R2011b/bin/glnxa64 -L/opt/MATLAB/R2011b/extern/lib/glnxa64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc -Wl,-rpath,/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -lf2clapack -lf2cblas -lm -lm -ldl /usr/bin/ld: cannot find -licudata collect2: ld returned 1 exit status make[3]: [ex19] Error 1 (ignored) /bin/rm -f ex19.o Completed test examples $ /opt/MATLAB/R2011b/bin/glnxa64/libicudata.so.44 Segmentation fault $ /opt/MATLAB/R2011b/bin/glnxa64/libicudata.so.44.2 Segmentation fault From bsmith at mcs.anl.gov Tue Dec 6 18:54:59 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 6 Dec 2011 18:54:59 -0600 Subject: [petsc-users] PetSc + MATLAB installation problem on Debian Squeeze In-Reply-To: <4EDEA05D.5000506@uchicago.edu> References: <4EDEA05D.5000506@uchicago.edu> Message-ID: <1823A41A-9BD9-4F47-9BD9-6AB42345A167@mcs.anl.gov> Oren, This comes from MATLAB having different libraries with different versions and different OSes and not having a mechanism to determine up front what libraries exist and are needed to be linked against. You need to first edit $PETSC_ARCH/conf/petscvariables and locate the line that contains -L/opt/MATLAB/R2011b/bin/glnxa64 -L/opt/MATLAB/R2011b/extern/lib/glnxa64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc then check in the directories /opt/MATLAB/R2011b/bin/glnxa64 and /opt/MATLAB/R2011b/extern/lib/glnxa64 for what libraries actually exist and remove from the list in petscvariables those that do not exist. Then try running make test again Please switch to petsc-maint at mcs.anl.gov for any additional emails needed on this thread of getting the MATLAB/PETSc stuff running. Barry On Dec 6, 2011, at 5:08 PM, Oren Livne wrote: > Dear All, > > I'm trying to install PetSc with the MATLAB interface on Debian Squeeze 6.26. "make all test" fails on an ld error due to a MATLAB library. Running the MATLAB library seems to cause a seg fault even though matlab runs fine (7.13 / R2011b). > > Could you kindly help? Output attached below. > > Thanks so much, > Oren > > ------------------------------------------------------------------------------ > $ uname -a > Linux vm-debian 2.6.32-5-amd64 #1 SMP Mon Oct 3 03:59:20 UTC 2011 x86_64 GNU/Linux > > $ ./configure --with-shared-libraries --with-matlab-engine --with-matlab --download-f2cblaslapack --with-mpi=0 > ... OK > > $ make all test > ... > libfast in: /home/oren/petsc-3.2-p5/tutorials/multiphysics > Completed building libraries > ========================================= > making shared libraries in /home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib > building libpetsc.so > /usr/bin/ld: cannot find -licudata > collect2: ld returned 1 exit status > ========================================= > Now to check if the libraries are working do: > make PETSC_DIR=/home/oren/petsc-3.2-p5 PETSC_ARCH=arch-linux2-c-debug test > ========================================= > Running test examples to verify correct installation > Using PETSC_DIR=/home/oren/petsc-3.2-p5 and PETSC_ARCH=arch-linux2-c-debug > --------------Error detected during compile or link!----------------------- > See http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html > gcc -o ex19.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -I/home/oren/petsc-3.2-p5/include -I/home/oren/petsc-3.2-p5/arch-linux2-c-debug/include -I/opt/MATLAB/R2011b/extern/include -I/home/oren/petsc-3.2-p5/include/mpiuni -D__INSDIR__=src/snes/examples/tutorials/ ex19.c > gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ex19 ex19.o -Wl,-rpath,/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -L/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/opt/MATLAB/R2011b/sys/os/glnxa64:/opt/MATLAB/R2011b/bin/glnxa64:/opt/MATLAB/R2011b/extern/lib/glnxa64 -L/opt/MATLAB/R2011b/bin/glnxa64 -L/opt/MATLAB/R2011b/extern/lib/glnxa64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc -Wl,-rpath,/home/oren/petsc-3.2-p5/arch-linux2-c-debug/lib -lf2clapack -lf2cblas -lm -lm -ldl > /usr/bin/ld: cannot find -licudata > collect2: ld returned 1 exit status > make[3]: [ex19] Error 1 (ignored) > /bin/rm -f ex19.o > Completed test examples > > $ /opt/MATLAB/R2011b/bin/glnxa64/libicudata.so.44 > Segmentation fault > > $ /opt/MATLAB/R2011b/bin/glnxa64/libicudata.so.44.2 > Segmentation fault > > > From xdliang at gmail.com Tue Dec 6 22:26:50 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 6 Dec 2011 23:26:50 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: Hello everyone, Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I want to have a 10-by-10 sparse matrix, where each row has maximum two nonzeros. I would like to have petsc ignore the 0.0 values by not allocating space for 0.0 values. In my problem, the nonzero pattern of sparse matrix will never change. I hope the ignore of zero entries will save lots of operations when I am doing matrix-vector product or spare-direct decomposition . Below is my script. When I check the mat_view_info, I have Matrix Object: type=mpiaij, rows=10, cols=10 total: nonzeros=20, allocated nonzeros=40 not using I-node (on process 0) routines Is there a way to have nonzeros=10, and allocated nonzeros=10, since the number of real nonzeros=10? Thanks. Xiangdong ----------------------------------------------------------- ?Mat A;? int N=10, i, ns,ne; ? ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); ? ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); ? for (i=ns;i wrote: > > ?The checks in the code are lines like > > ? ? ?if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; > > hence changing the source code ?to ignore small values is not difficult. ?But I think this is a terribly cheesy hack and would hope people would not do this. > > ? ?Barry > > > > On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: > >> I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? >> I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. >> >> Best, >> Mohamad >> >> >> >> >> >> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: >> >> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >> >> > Hello everyone, >> > >> > I have a question about whether zeros will change the sparse pattern. >> > Suppose I am generating two sparse matrices A and B. A is generated >> > with exactly number of nonzeros, say 10 nnz per row. When I generate >> > matrix B, I specify 12 nonzeros, but two of them are given zero values >> > by MatSetValue. Will A and B have same sparsity patterns and >> > performance ? In other words, will PETSc simply ignore these zeros >> > entries? ?I output these A and B in matlab, and it seems that these >> > zeros entries are ignored. I just want to double check on this. >> > >> > Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >> >> ? No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >> >> >> ? ?BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >> ? Barry >> >> > Thanks. >> > >> > Best, >> > Xiangdong >> >> > From gdiso at ustc.edu Tue Dec 6 22:54:36 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 7 Dec 2011 12:54:36 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> Message-ID: <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> I can only analysis the core file at present. It reported Program terminated with signal 11, Segmentation fault. #0 0x000000010217319c in dmumps_462 () Run program with -info, the output is Entering DMUMPS driver with JOB, N, NZ = 1 49 217 DMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** Resetting candidate strategy to 0 because NSLAVES=1 ... Structural symmetry (in percent)= 100 Density: NBdense, Average, Median = 0 3 3 ... No column permutation Ordering based on AMF Program terminated with signal SIGSEGV, Segmentation fault. The example in src/ksp/ksp/examples/tests/ex3 run ok -- it solves a small linear problem. Also, if I reduce the matrix size to ~20x20, no crash happen. I'll try to provide more information. On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: Bad news. MUMPS still crash on AIX/PPC. I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 task 0: Segmentation fault. I guess the interface of petsc-mumps has some problem, i.e. alignment. Please get a stack trace with the debugger. Matt Yes, petsc-dev works. Please merge the update to petsc-3.2. Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to work with it. In general, software projects don't update a previous release on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 6 23:11:13 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 6 Dec 2011 23:11:13 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: > Hello everyone, > > Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I > want to have a 10-by-10 sparse matrix, where each row has maximum two > nonzeros. I would like to have petsc ignore the 0.0 values by not > allocating space for 0.0 values. In my problem, the nonzero pattern of > sparse matrix will never change. I hope the ignore of zero entries > will save lots of operations when I am doing matrix-vector product or > spare-direct decomposition . > > Below is my script. When I check the mat_view_info, I have > > Matrix Object: > type=mpiaij, rows=10, cols=10 > total: nonzeros=20, allocated nonzeros=40 > not using I-node (on process 0) routines > > Is there a way to have nonzeros=10, and allocated nonzeros=10, since > the number of real nonzeros=10? Thanks. Those "extra" nonzeros are not actually in the sparse matrix so they do not cause extra multiples. You can safely ignore them. Barry > > Xiangdong > > ----------------------------------------------------------- > Mat A; int N=10, i, ns,ne; > ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); > ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); > for (i=ns;i ADD_VALUES); CHKERRQ(ierr); ierr=MatSetValue(A,i, (i+1)%N, 0.0, > ADD_VALUES); CHKERRQ(ierr); } > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ierr = > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > -------------------------------------------------------- > > > On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith wrote: >> >> The checks in the code are lines like >> >> if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; >> >> hence changing the source code to ignore small values is not difficult. But I think this is a terribly cheesy hack and would hope people would not do this. >> >> Barry >> >> >> >> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: >> >>> I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? >>> I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. >>> >>> Best, >>> Mohamad >>> >>> >>> >>> >>> >>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: >>> >>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >>> >>>> Hello everyone, >>>> >>>> I have a question about whether zeros will change the sparse pattern. >>>> Suppose I am generating two sparse matrices A and B. A is generated >>>> with exactly number of nonzeros, say 10 nnz per row. When I generate >>>> matrix B, I specify 12 nonzeros, but two of them are given zero values >>>> by MatSetValue. Will A and B have same sparsity patterns and >>>> performance ? In other words, will PETSc simply ignore these zeros >>>> entries? I output these A and B in matlab, and it seems that these >>>> zeros entries are ignored. I just want to double check on this. >>>> >>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >>> >>> No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >>> >>> >>> BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >>> Barry >>> >>>> Thanks. >>>> >>>> Best, >>>> Xiangdong >>> >>> >> From hzhang at mcs.anl.gov Tue Dec 6 23:16:27 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 6 Dec 2011 23:16:27 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> Message-ID: Get updated mumps interface in petsc-dev. Run your code with '-mat_mumps_icntl_4 4' which will display mumps info Use a debugger to see where the program terminates. Hong On Tue, Dec 6, 2011 at 10:54 PM, Gong Ding wrote: > I can only analysis the core file at present. > It reported > Program terminated with signal 11, Segmentation fault. > #0? 0x000000010217319c in dmumps_462 () > > Run program with -info, the output is > > Entering DMUMPS driver with JOB, N, NZ =?? 1????????? 49??????????? 217 > > ?DMUMPS 4.10.0 > L U Solver for unsymmetric matrices > Type of parallelism: Working host > > ?****** ANALYSIS STEP ******** > > ?Resetting candidate strategy to 0 because NSLAVES=1 > > ?... Structural symmetry (in percent)=? 100 > ?Density: NBdense, Average, Median?? =??? 0??? 3??? 3 > ?... No column permutation > ?Ordering based on AMF > > Program terminated with signal SIGSEGV, Segmentation fault. > > The example in src/ksp/ksp/examples/tests/ex3 run ok -- it solves a small > linear problem. > Also, if I reduce the matrix size to ~20x20, no crash happen. > > I'll try to provide more information. > > > > > > On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: >> >> Bad news. >> MUMPS still crash on AIX/PPC. >> I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 >> task 0: Segmentation fault. >> I guess the interface of petsc-mumps has some problem, i.e. alignment. > > > Please get a stack trace with the debugger. > > ? ?Matt > >>> >>> Yes, petsc-dev works. Please merge the update to petsc-3.2. >> >> >> Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to >> work with it. In general, software projects don't update a previous release >> on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From xdliang at gmail.com Tue Dec 6 23:39:38 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 00:39:38 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 12:11 AM, Barry Smith wrote: > > On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: > >> Hello everyone, >> >> Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I >> want to have a 10-by-10 sparse matrix, where each row has maximum two >> nonzeros. I would like to have petsc ignore the 0.0 ?values by not >> allocating space for 0.0 values. In my problem, the nonzero pattern of >> sparse matrix will never change. I hope the ignore of zero entries >> will save lots of operations when I am doing matrix-vector product or >> spare-direct decomposition . >> >> Below is my script. When I check the mat_view_info, I have >> >> Matrix Object: >> type=mpiaij, rows=10, cols=10 >> total: nonzeros=20, allocated nonzeros=40 >> ? not using I-node (on process 0) routines >> >> Is there a way to have nonzeros=10, and allocated nonzeros=10, since >> the number of real nonzeros=10? Thanks. > > ? Those "extra" nonzeros are not actually in the sparse matrix so they do not cause extra multiples. You can safely ignore them. I am sorry I am still not clear. Do "extra" nonzeros refer to ?nonzeros=20? or "allocated nonzeros=40"? With or without MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same mat_view_info. How can I see the differences in the spare matrix generated With/Without this option? Thanks. Xiangdong > > ?Barry > >> >> Xiangdong >> >> ----------------------------------------------------------- >> ?Mat A; ?int N=10, i, ns,ne; >> ? ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); >> ? ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); >> ? for (i=ns;i> ADD_VALUES); CHKERRQ(ierr); ? ? ?ierr=MatSetValue(A,i, (i+1)%N, 0.0, >> ADD_VALUES); CHKERRQ(ierr); ? ?} >> ? ? MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ? ? ierr = >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ? ? ierr = >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >> >> -------------------------------------------------------- >> >> >> On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith wrote: >>> >>> ?The checks in the code are lines like >>> >>> ? ? ?if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; >>> >>> hence changing the source code ?to ignore small values is not difficult. ?But I think this is a terribly cheesy hack and would hope people would not do this. >>> >>> ? ?Barry >>> >>> >>> >>> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: >>> >>>> I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? >>>> I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. >>>> >>>> Best, >>>> Mohamad >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: >>>> >>>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I have a question about whether zeros will change the sparse pattern. >>>>> Suppose I am generating two sparse matrices A and B. A is generated >>>>> with exactly number of nonzeros, say 10 nnz per row. When I generate >>>>> matrix B, I specify 12 nonzeros, but two of them are given zero values >>>>> by MatSetValue. Will A and B have same sparsity patterns and >>>>> performance ? In other words, will PETSc simply ignore these zeros >>>>> entries? ?I output these A and B in matlab, and it seems that these >>>>> zeros entries are ignored. I just want to double check on this. >>>>> >>>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >>>> >>>> ? No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >>>> >>>> >>>> ? ?BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >>>> ? Barry >>>> >>>>> Thanks. >>>>> >>>>> Best, >>>>> Xiangdong >>>> >>>> >>> > From gdiso at ustc.edu Wed Dec 7 00:48:09 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 7 Dec 2011 14:48:09 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> Message-ID: <1704494.361481323240489532.JavaMail.coremail@mail.ustc.edu> With a small matrix, mumps of AIX reports that Entering DMUMPS driver with JOB, N, NZ = 1 25 105 DMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** Resetting candidate strategy to 0 because NSLAVES=1 Entering analysis phase with ... N NZ LIW INFO(1) 25 105 287 0 Matrix entries: IRN() ICN() 1 1 1 2 1 3 2 1 2 2 2 4 2 5 3 1 3 3 3 4 3 6 4 2 4 3 4 4 4 7 4 8 5 2 5 5 5 7 5 9 6 3 6 6 6 8 6 10 7 4 7 5 7 7 7 11 7 12 8 4 8 6 8 8 8 12 8 13 9 5 9 9 9 11 9 14 10 6 10 10 10 13 10 15 11 7 11 9 11 11 11 16 11 17 12 7 12 8 12 12 12 17 12 18 13 8 13 10 13 13 13 18 13 19 14 9 14 14 14 16 15 10 15 15 15 19 16 11 16 14 16 16 16 20 17 11 17 12 17 17 17 20 17 21 18 12 18 13 18 18 18 21 18 22 19 13 19 15 19 19 19 22 20 16 20 17 20 20 20 23 21 17 21 18 21 21 21 23 21 24 22 18 22 19 22 22 22 24 23 20 23 21 23 23 23 25 24 21 24 22 24 24 24 25 25 23 25 24 25 25 ... Structural symmetry (in percent)= 100 Density: NBdense, Average, Median = 0 3 3 ... No column permutation Ordering based on AMF IKEEP(.,1)= 14 16 15 17 18 24 13 9 11 5 25 19 23 10 4 12 8 7 6 22 21 20 3 2 1 IKEEP(.,2)= 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 7 0 0 0 0 0 0 0 IKEEP(.,3)= 0 1 1 0 1 1 0 0 0 0 1 1 0 0 1 1 4 5 0 1 1 3 4 2 5 PTRAR(.,1)= 14 16 15 17 17 18 13 9 11 5 18 18 18 10 4 12 8 7 6 18 18 18 3 2 1 PTRAR(.,2)= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PTRAR(.,3)= 3 4 4 3 4 4 5 5 5 3 4 4 5 3 4 4 7 7 0 1 1 1 0 0 0 NFSIZ(.) = 3 4 4 7 0 0 5 5 4 4 0 7 0 3 3 4 5 5 4 0 0 0 4 4 3 FILS (.) = 0 -3 -1 5 -8 11 0 0 -14 -15 -23 22 6 0 0 -9 0 0 -10 13 20 21 -24 -25 0 FRERE(.) = -3 -4 -2 -12 26 26 2 16 -16 -19 26 0 26 -9 -10 7 4 17 18 26 26 26 19 -23 -24 ERROR: 0031-250 task 0: Segmentation fault The output is exactly the same as on linux, except the error. On Linux, the naxt line is Leaving analysis phase with ... INFOG(1) = 0 INFOG(2) = 0 -- (20) Number of entries in factors (estim.) = 185 -- (3) Storage of factors (REAL, estimated) = 185 -- (4) Storage of factors (INT , estimated) = 372 -- (5) Maximum frontal size (estimated) = 7 -- (6) Number of nodes in the tree = 18 -- (32) Type of analysis effectively used = 1 -- (7) Ordering option effectively used = 2 ICNTL(6) Maximum transversal option = 0 ICNTL(7) Pivot order option = 7 Percentage of memory relaxation (effective) = 20 Number of level 2 nodes = 0 Number of split nodes = 0 I compiled petsc-dev with debug , however, my code failed with [0] PetscCommDuplicate(): Using internal PETSc communicator 0 5 [0] PetscCommDuplicate(): Using internal PETSc communicator 0 5 [0] PetscCommDuplicate(): Using internal PETSc communicator 0 5 [0] PetscCommDuplicate(): Duplicating a communicator 1 6 max tags = 1073741823 The assert subroutine failed: context >= _min_context, file /project/sprelhya/build/rhyas002a/src/ppe/poe/include/mpi_cc_inlines.h, line 314 Any suggestion? > '-mat_mumps_icntl_4 4' which will display mumps info > > > > Use a debugger to see where the program terminates. > > > > Hong > > > > On Tue, Dec 6, 2011 at 10:54 PM, Gong Ding wrote: > > > I can only analysis the core file at present. > > > It reported > > > Program terminated with signal 11, Segmentation fault. > > > #0? 0x000000010217319c in dmumps_462 () > > > > > > Run program with -info, the output is > > > > > > Entering DMUMPS driver with JOB, N, NZ =?? 1????????? 49??????????? 217 > > > > > > ?DMUMPS 4.10.0 > > > L U Solver for unsymmetric matrices > > > Type of parallelism: Working host > > > > > > ?****** ANALYSIS STEP ******** > > > > > > ?Resetting candidate strategy to 0 because NSLAVES=1 > > > > > > ?... Structural symmetry (in percent)=? 100 > > > ?Density: NBdense, Average, Median?? =??? 0??? 3??? 3 > > > ?... No column permutation > > > ?Ordering based on AMF > > > > > > Program terminated with signal SIGSEGV, Segmentation fault. > > > > > > The example in src/ksp/ksp/examples/tests/ex3 run ok -- it solves a small > > > linear problem. > > > Also, if I reduce the matrix size to ~20x20, no crash happen. > > > > > > I'll try to provide more information. > > > > > > > > > > > > > > > > > > On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: > > >> > > >> Bad news. > > >> MUMPS still crash on AIX/PPC. > > >> I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 > > >> task 0: Segmentation fault. > > >> I guess the interface of petsc-mumps has some problem, i.e. alignment. > > > > > > > > > Please get a stack trace with the debugger. > > > > > > ? ?Matt > > > > > >>> > > >>> Yes, petsc-dev works. Please merge the update to petsc-3.2. > > >> > > >> > > >> Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to > > >> work with it. In general, software projects don't update a previous release > > >> on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments > > > is infinitely more interesting than any results to which their experiments > > > lead. > > > -- Norbert Wiener > > From gdiso at ustc.edu Wed Dec 7 02:53:47 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 7 Dec 2011 16:53:47 +0800 (CST) Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> Message-ID: <4644148.361751323248027736.JavaMail.coremail@mail.ustc.edu> I remove #define PETSCHAVE_DEBUG in the petscconf.h, and the debug version runs. The dbx report is: Segmentation fault in dmumps_462 at line 6743 in file "dmumps_part4.F" 6743 IF(DIM.EQ.1) THEN (dbx) where dmumps_462(internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??, internal error: assertion failed at line 3650 in file frame.c ??), line 6743 in "dmumps_part4.F" > > > > On Tue, Dec 6, 2011 at 10:54 PM, Gong Ding wrote: > > > I can only analysis the core file at present. > > > It reported > > > Program terminated with signal 11, Segmentation fault. > > > #0? 0x000000010217319c in dmumps_462 () > > > > > > Run program with -info, the output is > > > > > > Entering DMUMPS driver with JOB, N, NZ =?? 1????????? 49??????????? 217 > > > > > > ?DMUMPS 4.10.0 > > > L U Solver for unsymmetric matrices > > > Type of parallelism: Working host > > > > > > ?****** ANALYSIS STEP ******** > > > > > > ?Resetting candidate strategy to 0 because NSLAVES=1 > > > > > > ?... Structural symmetry (in percent)=? 100 > > > ?Density: NBdense, Average, Median?? =??? 0??? 3??? 3 > > > ?... No column permutation > > > ?Ordering based on AMF > > > > > > Program terminated with signal SIGSEGV, Segmentation fault. > > > > > > The example in src/ksp/ksp/examples/tests/ex3 run ok -- it solves a small > > > linear problem. > > > Also, if I reduce the matrix size to ~20x20, no crash happen. > > > > > > I'll try to provide more information. > > > > > > > > > > > > > > > > > > On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: > > >> > > >> Bad news. > > >> MUMPS still crash on AIX/PPC. > > >> I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 > > >> task 0: Segmentation fault. > > >> I guess the interface of petsc-mumps has some problem, i.e. alignment. > > > > > > > > > Please get a stack trace with the debugger. > > > > > > ? ?Matt > > > > > >>> > > >>> Yes, petsc-dev works. Please merge the update to petsc-3.2. > > >> > > >> > > >> Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to > > >> work with it. In general, software projects don't update a previous release > > >> on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments > > > is infinitely more interesting than any results to which their experiments > > > lead. > > > -- Norbert Wiener > > From Andrew.Parker2 at baesystems.com Wed Dec 7 02:54:00 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Wed, 7 Dec 2011 08:54:00 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: Legends, nice one! Got to that point in the souce code yesterday, saw that 7 was missing from the switch statement, and got worried. Is there any symmetry between the code in block size 5->6->7->8? I've asked that without looking, so could be stupid question. The reason I ask is that if there is this is a classic use of something like boost-preprocessor macros or a bit of templating, as in the compiler would expand it all out given a pattern at compile time? Just interest.... Cheers again, Andy -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: 06 December 2011 19:31 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Tue, 6 Dec 2011, Matthew Knepley wrote: > On Tue, Dec 6, 2011 at 7:43 AM, Parker, Andrew (UK Filton) < > Andrew.Parker2 at baesystems.com> wrote: > > > So having got petsc 3.2 install and all its deps, I now get > > this....**** > > > > ** ** > > > > [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in > > src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size > > 7**** > > > > ** ** > > > > ** ** > > > > Any thoughts? > > > > Sorry about that. There was confusion on our end. I have added the > code you need to 3.2. It will go out with the next patch release. > Alternatively, you can retrieve the source tarball from the repository > site or use the repository directly. I'm attaching this change as a patchfile - which can be applied to the current petsc-3.2 sources you have. cd petsc-3.2 patch -Np1 < bpjacobi.patch Satish ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** From knepley at gmail.com Wed Dec 7 03:36:55 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Dec 2011 03:36:55 -0600 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 2:54 AM, Parker, Andrew (UK Filton) < Andrew.Parker2 at baesystems.com> wrote: > Legends, nice one! > > Got to that point in the souce code yesterday, saw that 7 was missing > from the switch statement, and got worried. > > Is there any symmetry between the code in block size 5->6->7->8? I've > asked that without looking, so could be stupid question. The reason I > ask is that if there is this is a classic use of something like > boost-preprocessor macros or a bit of templating, as in the compiler > would expand it all out given a pattern at compile time? Just > interest.... > This is the kind of thing you could do with templates. However, that would mean 1) It only works in C++, and the logic is convoluted 2) You can't see the generated code 3) Its impossible to debug (and usually optimize) We are looking at other approaches. Thanks, Matt > Cheers again, > Andy > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: 06 December 2011 19:31 > To: PETSc users list > Subject: Re: [petsc-users] MATBDIAG in > 3.1 > > *** WARNING *** > > This message has originated outside your organisation, > either from an external partner or the Global Internet. > Keep this in mind if you answer this message. > > > On Tue, 6 Dec 2011, Matthew Knepley wrote: > > > On Tue, Dec 6, 2011 at 7:43 AM, Parker, Andrew (UK Filton) < > > Andrew.Parker2 at baesystems.com> wrote: > > > > > So having got petsc 3.2 install and all its deps, I now get > > > this....**** > > > > > > ** ** > > > > > > [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in > > > src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size > > > 7**** > > > > > > ** ** > > > > > > ** ** > > > > > > Any thoughts? > > > > > > > Sorry about that. There was confusion on our end. I have added the > > code you need to 3.2. It will go out with the next patch release. > > Alternatively, you can retrieve the source tarball from the repository > > > site or use the repository directly. > > > I'm attaching this change as a patchfile - which can be applied to the > current > petsc-3.2 sources you have. > > cd petsc-3.2 > patch -Np1 < bpjacobi.patch > > Satish > > ******************************************************************** > This email and any attachments are confidential to the intended > recipient and may also be privileged. If you are not the intended > recipient please delete it from your system and notify the sender. > You should not copy it or use it for any purpose nor disclose or > distribute its contents to any other person. > ******************************************************************** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Dec 7 03:44:26 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Dec 2011 03:44:26 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Tue, Dec 6, 2011 at 11:39 PM, Xiangdong Liang wrote: > On Wed, Dec 7, 2011 at 12:11 AM, Barry Smith wrote: > > > > On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: > > > >> Hello everyone, > >> > >> Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I > >> want to have a 10-by-10 sparse matrix, where each row has maximum two > >> nonzeros. I would like to have petsc ignore the 0.0 values by not > >> allocating space for 0.0 values. In my problem, the nonzero pattern of > >> sparse matrix will never change. I hope the ignore of zero entries > >> will save lots of operations when I am doing matrix-vector product or > >> spare-direct decomposition . > >> > >> Below is my script. When I check the mat_view_info, I have > >> > >> Matrix Object: > >> type=mpiaij, rows=10, cols=10 > >> total: nonzeros=20, allocated nonzeros=40 > >> not using I-node (on process 0) routines > >> > >> Is there a way to have nonzeros=10, and allocated nonzeros=10, since > >> the number of real nonzeros=10? Thanks. > > > > Those "extra" nonzeros are not actually in the sparse matrix so they > do not cause extra multiples. You can safely ignore them. > > I am sorry I am still not clear. Do "extra" nonzeros refer to > ?nonzeros=20? or "allocated nonzeros=40"? With or without > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same > mat_view_info. How can I see the differences in the spare matrix > generated With/Without this option? Thanks. If there are 2 nonzeros per row, there are 20 nonzeros not 10 in the matrix. According to the output, you allocated 40 nonzeros for the matrix. When you call MatAssemblyEnd(), we compress the matrix so that the 20 extra allocated nonzeros are not present. Matt > > Xiangdong > > > > > > Barry > > > >> > >> Xiangdong > >> > >> ----------------------------------------------------------- > >> Mat A; int N=10, i, ns,ne; > >> > ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); > >> ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); > >> for (i=ns;i >> ADD_VALUES); CHKERRQ(ierr); ierr=MatSetValue(A,i, (i+1)%N, 0.0, > >> ADD_VALUES); CHKERRQ(ierr); } > >> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ierr = > >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = > >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > >> > >> -------------------------------------------------------- > >> > >> > >> On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith > wrote: > >>> > >>> The checks in the code are lines like > >>> > >>> if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) > continue; > >>> > >>> hence changing the source code to ignore small values is not > difficult. But I think this is a terribly cheesy hack and would hope > people would not do this. > >>> > >>> Barry > >>> > >>> > >>> > >>> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: > >>> > >>>> I was always wondering how difficult it would be to define a constant > say, ZERO_DROP_TOLERANCE so that when the matrix is setup, > MatAssemblyBegin()/End(), it automatically ignores the values below this > threshold? > >>>> I understand the for the matrices that might change in successive > iterations, that might not be a good idea since the memory reallocation > could be very slow, but for a lot of cases which a constant matrix, that > could be simply helpful. > >>>> > >>>> Best, > >>>> Mohamad > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith > wrote: > >>>> > >>>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: > >>>> > >>>>> Hello everyone, > >>>>> > >>>>> I have a question about whether zeros will change the sparse pattern. > >>>>> Suppose I am generating two sparse matrices A and B. A is generated > >>>>> with exactly number of nonzeros, say 10 nnz per row. When I generate > >>>>> matrix B, I specify 12 nonzeros, but two of them are given zero > values > >>>>> by MatSetValue. Will A and B have same sparsity patterns and > >>>>> performance ? In other words, will PETSc simply ignore these zeros > >>>>> entries? I output these A and B in matlab, and it seems that these > >>>>> zeros entries are ignored. I just want to double check on this. > >>>>> > >>>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? > >>>> > >>>> No, by default PETSc will insert the zero values thus increasing > the nonzero pattern. If you set that option to the matrix then inserting > values it will not introduce an increase in the pattern. > >>>> > >>>> > >>>> BTW: if you then load the matrix via a binary file to MATLAB it > may be that MATLAB silently removes those locations so you don't see them. > >>>> Barry > >>>> > >>>>> Thanks. > >>>>> > >>>>> Best, > >>>>> Xiangdong > >>>> > >>>> > >>> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Parker2 at baesystems.com Wed Dec 7 03:55:40 2011 From: Andrew.Parker2 at baesystems.com (Parker, Andrew (UK Filton)) Date: Wed, 7 Dec 2011 09:55:40 -0000 Subject: [petsc-users] MATBDIAG in > 3.1 In-Reply-To: References: <9D55C83C-E7CA-4F9E-AA2D-CA4513B4E75E@mcs.anl.gov> Message-ID: Just my thoughts....I'm sure you've thought about this already.... 1) It would need to be compiled by a c++ compiler, but could be used as can other c++ functions by other languages given the export of the function and suitable wrapping.... 2) I agree on seeing the code, need to think about that. Although technically of course you could actually write a programme and compile it with the purpose of writing out a header file with a function in it, in that case you would use pre-processor boost magic. The idea would be to compile that code before petsc is compiled, run that code before petsc is compiled and that would produce a file with the function in it, say for block size 20 , then include that file and use that generated function when petsc is compiled. That way you would see the automated code..... 3) Using the pre-processor boost method probably produces close to the most optimal in-lined code you can get in my (personal) opinion.... Something like python to do the code-gen... Cheers, Andy From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: 07 December 2011 09:37 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Wed, Dec 7, 2011 at 2:54 AM, Parker, Andrew (UK Filton) wrote: Legends, nice one! Got to that point in the souce code yesterday, saw that 7 was missing from the switch statement, and got worried. Is there any symmetry between the code in block size 5->6->7->8? I've asked that without looking, so could be stupid question. The reason I ask is that if there is this is a classic use of something like boost-preprocessor macros or a bit of templating, as in the compiler would expand it all out given a pattern at compile time? Just interest.... This is the kind of thing you could do with templates. However, that would mean 1) It only works in C++, and the logic is convoluted 2) You can't see the generated code 3) Its impossible to debug (and usually optimize) We are looking at other approaches. Thanks, Matt Cheers again, Andy -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: 06 December 2011 19:31 To: PETSc users list Subject: Re: [petsc-users] MATBDIAG in > 3.1 *** WARNING *** This message has originated outside your organisation, either from an external partner or the Global Internet. Keep this in mind if you answer this message. On Tue, 6 Dec 2011, Matthew Knepley wrote: > On Tue, Dec 6, 2011 at 7:43 AM, Parker, Andrew (UK Filton) < > Andrew.Parker2 at baesystems.com> wrote: > > > So having got petsc 3.2 install and all its deps, I now get > > this....**** > > > > ** ** > > > > [0]PETSC ERROR: PCSetUp_PBJacobi() line 207 in > > src/ksp/pc/impls/pbjacobi/pbjacobi.c not supported for block size > > 7**** > > > > ** ** > > > > ** ** > > > > Any thoughts? > > > > Sorry about that. There was confusion on our end. I have added the > code you need to 3.2. It will go out with the next patch release. > Alternatively, you can retrieve the source tarball from the repository > site or use the repository directly. I'm attaching this change as a patchfile - which can be applied to the current petsc-3.2 sources you have. cd petsc-3.2 patch -Np1 < bpjacobi.patch Satish ******************************************************************** This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person. ******************************************************************** -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Dec 7 05:49:56 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 07 Dec 2011 12:49:56 +0100 Subject: [petsc-users] DMDA Question In-Reply-To: <67C962A8-AD29-4474-A5BE-89244044834B@mcs.anl.gov> References: <67C962A8-AD29-4474-A5BE-89244044834B@mcs.anl.gov> Message-ID: <4EDF52E4.9070304@gmail.com> Hi, Sorry to hijack the discussion but I'm currently thinking of implementing DMDA for staggered grids in 2D and later on in 3D. I am not sure if my problem is exactly the same as that of Tim but I don't really understand his problem. So for my case, I have a staggered grid for my finite volume CFD code. So the grid looks like this: Do I need to create 3 DMDA, for u,v,p? When computing the flux, I will also need to interpolate u and v velocities on the u or v cells. I hope you can suggest the best way to approach this problem. Btw, in the discussion, there's also some mention of DOF. However, I thought each variable (u,v,p) will have only 1 DOF, is that not so? Thanks! Yours sincerely, TAY wee-beng On 2/12/2011 8:12 PM, Barry Smith wrote: > Tim, > > Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. > > Do I understand you want two DMDA > > 1) one that has locations for cell vertices (or face centers)? > > 2) and one that locations for cell centers? > > Once we understand what you want we may be able to make a suggestion. > > Barry > > > On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > >> Hi, >> >> I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: >> >> Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. >> >> I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? >> >> Any advice would be appreciated, >> >> Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecdhigbj.png Type: image/png Size: 47018 bytes Desc: not available URL: From dharmareddy84 at gmail.com Wed Dec 7 06:26:40 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 7 Dec 2011 06:26:40 -0600 Subject: [petsc-users] petsc with scalapack and blacs from intel ? Message-ID: Hello, Can you tell me what should be the config options to use the scalapack and blacs provided in intel mkl with petsc ? thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Dec 7 06:42:27 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 07 Dec 2011 13:42:27 +0100 Subject: [petsc-users] petsc with scalapack and blacs from intel ? In-Reply-To: References: Message-ID: <4EDF5F33.3070602@gfz-potsdam.de> Hi Reddy, You should use --with-scalapack-lib/include and --with-blacs-lib/include For instance: --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include Regards, Alexander On 07.12.2011 13:26, Dharmendar Reddy wrote: > Hello, > Can you tell me what should be the config options to use the > scalapack and blacs provided in intel mkl with petsc ? > > thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Dec 7 07:26:20 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Dec 2011 07:26:20 -0600 Subject: [petsc-users] DMDA Question In-Reply-To: <4EDF52E4.9070304@gmail.com> References: <67C962A8-AD29-4474-A5BE-89244044834B@mcs.anl.gov> <4EDF52E4.9070304@gmail.com> Message-ID: On Wed, Dec 7, 2011 at 5:49 AM, TAY wee-beng wrote: > Hi, > > Sorry to hijack the discussion but I'm currently thinking of implementing > DMDA for staggered grids in 2D and later on in 3D. > > I am not sure if my problem is exactly the same as that of Tim but I don't > really understand his problem. > > So for my case, I have a staggered grid for my finite volume CFD code. So > the grid looks like this: > > > > Do I need to create 3 DMDA, for u,v,p? When computing the flux, I will > also need to interpolate u and v velocities on the u or v cells. I hope you > can suggest the best way to approach this problem. > Can you look at SNES ex30? I believe that discretization is very similar to what you want. Matt > Btw, in the discussion, there's also some mention of DOF. However, I > thought each variable (u,v,p) will have only 1 DOF, is that not so? > > Thanks! > > Yours sincerely, > > TAY wee-beng > > > On 2/12/2011 8:12 PM, Barry Smith wrote: > > Tim, > > Unfortunately the DMDA was not designed with the idea of staggered grids (or mixed discretizations) in mind so they are not straightforward like we would like them to be. > > Do I understand you want two DMDA > > 1) one that has locations for cell vertices (or face centers)? > > 2) and one that locations for cell centers? > > Once we understand what you want we may be able to make a suggestion. > > Barry > > > On Dec 2, 2011, at 1:08 PM, Tim Gallagher wrote: > > > Hi, > > I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following: > > Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. > > I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. Has anybody used a DMDA to create a dual DMDA? > > Any advice would be appreciated, > > Tim > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 47018 bytes Desc: not available URL: From hzhang at mcs.anl.gov Wed Dec 7 08:22:18 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 7 Dec 2011 08:22:18 -0600 Subject: [petsc-users] MUMPS solver crash with petsc-3.2 In-Reply-To: <4644148.361751323248027736.JavaMail.coremail@mail.ustc.edu> References: <31107580.352981322819243395.JavaMail.coremail@mail.ustc.edu> <5B58FB78E1E048EF803121C203D9A793@cogendaeda> <2AC4E4AA-7984-40E4-8376-5ADBB32B4820@mcs.anl.gov> <3145350.357701323085805845.JavaMail.coremail@mail.ustc.edu> <9508667.359421323154353523.JavaMail.coremail@mail.ustc.edu> <7872396.361311323233676626.JavaMail.coremail@mail.ustc.edu> <4644148.361751323248027736.JavaMail.coremail@mail.ustc.edu> Message-ID: On Wed, Dec 7, 2011 at 2:53 AM, Gong Ding wrote: > I remove #define PETSCHAVE_DEBUG in the petscconf.h, and the debug version runs. > > The dbx report is: > > Segmentation fault in dmumps_462 at line 6743 in file "dmumps_part4.F" > ?6743 ? ? ? ? IF(DIM.EQ.1) THEN This crashes inside mumps solver. Can you install mumps on this machine and run a stand-alone mumps example, e.g., ~mumps/examples/c_example.c? Hong > (dbx) where > dmumps_462(internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??, internal error: assertion failed at line 3650 in file frame.c > ??), line 6743 in "dmumps_part4.F" > > > >> >> >> >> On Tue, Dec 6, 2011 at 10:54 PM, Gong Ding wrote: >> >> > I can only analysis the core file at present. >> >> > It reported >> >> > Program terminated with signal 11, Segmentation fault. >> >> > #0? 0x000000010217319c in dmumps_462 () >> >> > >> >> > Run program with -info, the output is >> >> > >> >> > Entering DMUMPS driver with JOB, N, NZ =?? 1????????? 49??????????? 217 >> >> > >> >> > ?DMUMPS 4.10.0 >> >> > L U Solver for unsymmetric matrices >> >> > Type of parallelism: Working host >> >> > >> >> > ?****** ANALYSIS STEP ******** >> >> > >> >> > ?Resetting candidate strategy to 0 because NSLAVES=1 >> >> > >> >> > ?... Structural symmetry (in percent)=? 100 >> >> > ?Density: NBdense, Average, Median?? =??? 0??? 3??? 3 >> >> > ?... No column permutation >> >> > ?Ordering based on AMF >> >> > >> >> > Program terminated with signal SIGSEGV, Segmentation fault. >> >> > >> >> > The example in src/ksp/ksp/examples/tests/ex3 run ok -- it solves a small >> >> > linear problem. >> >> > Also, if I reduce the matrix size to ~20x20, no crash happen. >> >> > >> >> > I'll try to provide more information. >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Tue, Dec 6, 2011 at 12:52 AM, Gong Ding wrote: >> >> >> >> >> >> Bad news. >> >> >> MUMPS still crash on AIX/PPC. >> >> >> I tried petsc-3.1, 3.2 and dev. All of them crash with ERROR: 0031-250 >> >> >> task 0: Segmentation fault. >> >> >> I guess the interface of petsc-mumps has some problem, i.e. alignment. >> >> > >> >> > >> >> > Please get a stack trace with the debugger. >> >> > >> >> > ? ?Matt >> >> > >> >> >>> >> >> >>> Yes, petsc-dev works. Please merge the update to petsc-3.2. >> >> >> >> >> >> >> >> >> Parmetis 4.0 was released after PETSc 3.2 so it will not be updated to >> >> >> work with it. In general, software projects don't update a previous release >> >> >> on a dependent package update. Please use petsc-dev or wait until PETSc 3.3. >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their experiments >> >> > is infinitely more interesting than any results to which their experiments >> >> > lead. >> >> > -- Norbert Wiener >> >> From bsmith at mcs.anl.gov Wed Dec 7 08:24:54 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 7 Dec 2011 08:24:54 -0600 Subject: [petsc-users] [petsc-maint #98908] Slow MatSetValues in just one of the process In-Reply-To: References: Message-ID: <27D12AF2-16F4-4E8D-A73D-EBB7CB0693EE@mcs.anl.gov> http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly On Dec 7, 2011, at 6:19 AM, Manuel Ricardo Perez Cerquera wrote: > Dear all, > > I'm having a problem with my code, when I ran with 2 processors or more, > when I'm setting the values The process zero seems as like I didn't do any > kind of preallocation in its local part. I'm really confused because the > other processes go faster. I already check the d_nnz and o_nnz vectors and > they are ok. Does anybody has ever had this problem? I appreciate any clue. > Thanks > > Manuel. > > -- > Eng. Manuel Ricardo Perez Cerquera. MSc. Ph.D student > Antenna and EMC Lab (LACE) > Istituto Superiore Mario Boella (ISMB) > Politecnico di Torino > Via Pier Carlo Boggio 61, Torino 10138, Italy > Email: manuel.perezcerquera at polito.it > Phone: +39 0112276704 > Fax: +39 011 2276 299 > From bsmith at mcs.anl.gov Wed Dec 7 08:28:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 7 Dec 2011 08:28:24 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Dec 6, 2011, at 11:39 PM, Xiangdong Liang wrote: > On Wed, Dec 7, 2011 at 12:11 AM, Barry Smith wrote: >> >> On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: >> >>> Hello everyone, >>> >>> Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I >>> want to have a 10-by-10 sparse matrix, where each row has maximum two >>> nonzeros. I would like to have petsc ignore the 0.0 values by not >>> allocating space for 0.0 values. In my problem, the nonzero pattern of >>> sparse matrix will never change. I hope the ignore of zero entries >>> will save lots of operations when I am doing matrix-vector product or >>> spare-direct decomposition . >>> >>> Below is my script. When I check the mat_view_info, I have >>> >>> Matrix Object: >>> type=mpiaij, rows=10, cols=10 >>> total: nonzeros=20, allocated nonzeros=40 >>> not using I-node (on process 0) routines >>> >>> Is there a way to have nonzeros=10, and allocated nonzeros=10, since >>> the number of real nonzeros=10? Thanks. >> >> Those "extra" nonzeros are not actually in the sparse matrix so they do not cause extra multiples. You can safely ignore them. > > I am sorry I am still not clear. Do "extra" nonzeros refer to > ?nonzeros=20? or "allocated nonzeros=40"? With or without > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same > mat_view_info. You will see the same message. You are not setting any zero entries so the option doesn't matter in your case. > How can I see the differences in the spare matrix > generated With/Without this option? You will not see a difference in the output. Don't worry about it. The option does prevent zero entries from being inserted and thus slowing down the matrix operations. Barry > Thanks. > > Xiangdong > > >> >> Barry >> >>> >>> Xiangdong >>> >>> ----------------------------------------------------------- >>> Mat A; int N=10, i, ns,ne; >>> ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); >>> ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); >>> for (i=ns;i>> ADD_VALUES); CHKERRQ(ierr); ierr=MatSetValue(A,i, (i+1)%N, 0.0, >>> ADD_VALUES); CHKERRQ(ierr); } >>> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ierr = >>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = >>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >>> >>> -------------------------------------------------------- >>> >>> >>> On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith wrote: >>>> >>>> The checks in the code are lines like >>>> >>>> if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; >>>> >>>> hence changing the source code to ignore small values is not difficult. But I think this is a terribly cheesy hack and would hope people would not do this. >>>> >>>> Barry >>>> >>>> >>>> >>>> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: >>>> >>>>> I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? >>>>> I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. >>>>> >>>>> Best, >>>>> Mohamad >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: >>>>> >>>>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I have a question about whether zeros will change the sparse pattern. >>>>>> Suppose I am generating two sparse matrices A and B. A is generated >>>>>> with exactly number of nonzeros, say 10 nnz per row. When I generate >>>>>> matrix B, I specify 12 nonzeros, but two of them are given zero values >>>>>> by MatSetValue. Will A and B have same sparsity patterns and >>>>>> performance ? In other words, will PETSc simply ignore these zeros >>>>>> entries? I output these A and B in matlab, and it seems that these >>>>>> zeros entries are ignored. I just want to double check on this. >>>>>> >>>>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >>>>> >>>>> No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >>>>> >>>>> >>>>> BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >>>>> Barry >>>>> >>>>>> Thanks. >>>>>> >>>>>> Best, >>>>>> Xiangdong >>>>> >>>>> >>>> >> From bogdan at lmn.pub.ro Wed Dec 7 08:31:42 2011 From: bogdan at lmn.pub.ro (Bogdan Dita) Date: Wed, 7 Dec 2011 16:31:42 +0200 Subject: [petsc-users] Problem with Optimised version .vs. Debug version Message-ID: Hello, A few weeks ago i posted a question regarding a problem with the optimised version of PETSc, what i mean is that I obtained a higher solving time with the optimised version then with the debug version(~10-15% higher). This is not a performance issue regarding my own program because I got a similar result with PETSc provided codes(eg. codes from /src/ksp/ksp/examples/tutorials/). This kind of strange results i got only when testing code on the cluster, on my laptop all works great (opt_vers_time=debug_vers_time/2.2). I tried all kinds of PETSc optimised version configs options and even reinstalled mpi a few times, and still the same problem. Maybe someone out there has had a similar problem because I'm running out off ideas. Below are the config options for petsc_opt, petsc_debug and mpich2, maybe someone notices something i missed or has some idea regarding my problem. __________petsc_debug____________ -with-mpi-dir=/usr/lib64/mpich2 --with-clanguage=c --with-debugging=1 --with-shared-libraries=1 --with-scalar-type=complex --download-umfpack=yes --with-fortran-kernels=1 --download-superlu=yes --download-mumps=yes --download-scalapack=yes --download-superlu_dist=yes --download-blacs=yes --download-f-blas-lapack=yes --download-plapack=yes --download-parmetis=yes Working directory: /home/bogdan/petsc-new-dbg Machine platform: ('Linux', 'tau', '2.6.41.1-1.fc15.x86_64', '#1 SMP Fri Nov 11 21:36:28 UTC 2011', 'x86_64', 'x86_64') _________petsc-opt___________ --with-mpi-dir=/usr/lib64/mpich2 --with-clanguage=c --with-debugging=no --with-shared-libraries=1 --with-scalar-type=complex --download-umfpack=yes --with-fortran-kernels=1 --download-superlu=yes --download-mumps=yes --download-scalapack=yes --download-superlu_dist=yes --download-blacs=yes --download-f-blas-lapack=yes --download-plapack=yes --download-parmetis=yes Working directory: /home/bogdan/petsc-new-opt Machine platform:('Linux', 'tau', '2.6.41.1-1.fc15.x86_64', '#1 SMP Fri Nov 11 21:36:28 UTC 2011', 'x86_64', 'x86_64') _________mpich2___________*fedora15 provided MPICH2 Version: 1.4.1p1 MPICH2 Release date: Thu Sep 1 13:53:02 CDT 2011 MPICH2 Device: ch3:nemesis MPICH2 configure: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-sharedlibs=gcc --enable-shared --enable-lib-depend --disable-rpath --enable-fc --with-device=ch3:nemesis --with-pm=hydra:gforker --sysconfdir=/etc/mpich2-x86_64 --includedir=/usr/include/mpich2-x86_64 --bindir=/usr/lib64/mpich2/bin --libdir=/usr/lib64/mpich2/lib --datadir=/usr/share/mpich2 --mandir=/usr/share/man/mpich2 --docdir=/usr/share/mpich2/doc --htmldir=/usr/share/mpich2/doc --with-hwloc-prefix=system --with-java=/etc/alternatives/java_sdk FC=gfortran F77=gfortran CFLAGS=-m64 -O2 -fPIC CXXFLAGS=-m64 -O2 -fPIC FCFLAGS=-m64 -O2 -fPIC FFLAGS=-m64 -O2 -fPIC LDFLAGS=-Wl,-z,noexecstack MPICH2LIB_CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic MPICH2LIB_CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic MPICH2LIB_FCFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic MPICH2LIB_FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic MPICH2 CC: gcc -m64 -O2 -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -O2 MPICH2 CXX: c++ -m64 -O2 -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -O2 MPICH2 F77: gfortran -m64 -O2 -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -O2 MPICH2 FC: gfortran -m64 -O2 -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -O2 -------------------------------------- Cosmin-Bogdan DITA From jedbrown at mcs.anl.gov Wed Dec 7 09:53:54 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 7 Dec 2011 07:53:54 -0800 Subject: [petsc-users] Problem with Optimised version .vs. Debug version In-Reply-To: References: Message-ID: On Wed, Dec 7, 2011 at 06:31, Bogdan Dita wrote: > A few weeks ago i posted a question regarding a problem with the > optimised version of PETSc, what i mean is that I obtained a higher > solving time with the optimised version then with the debug > version(~10-15% higher). > What do you get from 'mpicc -show'? MPICH2 has a tendency to put optimization flags into the wrappers. It's possible that the "optimized" build of PETSc is effectively using -O2 (from MPICH2) and -g3 (from PETSc with-debugging=1), which might be faster than -O2 (from MPICH2) -O (from PETSc with-debugging=0, taking precedence) that you get when you. You can use PETSc's --COPTFLAGS to set your own optimization flags for PETSc. Let us know if this works. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Wed Dec 7 10:22:57 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 11:22:57 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 9:28 AM, Barry Smith wrote: > > On Dec 6, 2011, at 11:39 PM, Xiangdong Liang wrote: > >> On Wed, Dec 7, 2011 at 12:11 AM, Barry Smith wrote: >>> >>> On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: >>> >>>> Hello everyone, >>>> >>>> Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I >>>> want to have a 10-by-10 sparse matrix, where each row has maximum two >>>> nonzeros. I would like to have petsc ignore the 0.0 ?values by not >>>> allocating space for 0.0 values. In my problem, the nonzero pattern of >>>> sparse matrix will never change. I hope the ignore of zero entries >>>> will save lots of operations when I am doing matrix-vector product or >>>> spare-direct decomposition . >>>> >>>> Below is my script. When I check the mat_view_info, I have >>>> >>>> Matrix Object: >>>> type=mpiaij, rows=10, cols=10 >>>> total: nonzeros=20, allocated nonzeros=40 >>>> ? not using I-node (on process 0) routines >>>> >>>> Is there a way to have nonzeros=10, and allocated nonzeros=10, since >>>> the number of real nonzeros=10? Thanks. >>> >>> ? Those "extra" nonzeros are not actually in the sparse matrix so they do not cause extra multiples. You can safely ignore them. >> >> I am sorry I am still not clear. ? Do "extra" nonzeros refer to >> ?nonzeros=20? or "allocated nonzeros=40"? With or without >> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same >> mat_view_info. > > ? You will see the same message. You are not setting any zero entries so the option doesn't matter in your case. I think I did have 10 zeros entries. In my script, I have inserted zeros by MatSetValue(A,i, (i+1)%N, 0.0, ADD_VALUES); Why doesn't that option matter in my case? That option prevents PETSc from inserting zeros, doesn't it? Thank you. > > >> How can I see the differences in the spare matrix >> generated With/Without this option? > > ? You will not see a difference in the output. > > ? Don't worry about it. The option does prevent zero entries from being inserted and thus slowing down the matrix operations. > > > > ? Barry > >> Thanks. >> >> Xiangdong >> >> >>> >>> ?Barry >>> >>>> >>>> Xiangdong >>>> >>>> ----------------------------------------------------------- >>>> ?Mat A; ?int N=10, i, ns,ne; >>>> ? ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); >>>> ? ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); >>>> ? for (i=ns;i>>> ADD_VALUES); CHKERRQ(ierr); ? ? ?ierr=MatSetValue(A,i, (i+1)%N, 0.0, >>>> ADD_VALUES); CHKERRQ(ierr); ? ?} >>>> ? ? MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ? ? ierr = >>>> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ? ? ierr = >>>> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >>>> >>>> -------------------------------------------------------- >>>> >>>> >>>> On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith wrote: >>>>> >>>>> ?The checks in the code are lines like >>>>> >>>>> ? ? ?if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) continue; >>>>> >>>>> hence changing the source code ?to ignore small values is not difficult. ?But I think this is a terribly cheesy hack and would hope people would not do this. >>>>> >>>>> ? ?Barry >>>>> >>>>> >>>>> >>>>> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: >>>>> >>>>>> I was always wondering how difficult it would be to define a constant say, ZERO_DROP_TOLERANCE so that when the matrix is setup, MatAssemblyBegin()/End(), it automatically ignores the values below this threshold? >>>>>> I understand the for the matrices that might change in successive iterations, that might not be a good idea since the memory reallocation could be very slow, but for a lot of cases which a constant matrix, that could be simply helpful. >>>>>> >>>>>> Best, >>>>>> Mohamad >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith wrote: >>>>>> >>>>>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> I have a question about whether zeros will change the sparse pattern. >>>>>>> Suppose I am generating two sparse matrices A and B. A is generated >>>>>>> with exactly number of nonzeros, say 10 nnz per row. When I generate >>>>>>> matrix B, I specify 12 nonzeros, but two of them are given zero values >>>>>>> by MatSetValue. Will A and B have same sparsity patterns and >>>>>>> performance ? In other words, will PETSc simply ignore these zeros >>>>>>> entries? ?I output these A and B in matlab, and it seems that these >>>>>>> zeros entries are ignored. I just want to double check on this. >>>>>>> >>>>>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >>>>>> >>>>>> ? No, by default PETSc will insert the zero values thus increasing the nonzero pattern. If you set that option to the matrix then inserting values it will not introduce an increase in the pattern. >>>>>> >>>>>> >>>>>> ? ?BTW: if you then load the matrix via a binary file to MATLAB it may be that MATLAB silently removes those locations so you don't see them. >>>>>> ? Barry >>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> Best, >>>>>>> Xiangdong >>>>>> >>>>>> >>>>> >>> > From balay at mcs.anl.gov Wed Dec 7 10:53:29 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 7 Dec 2011 10:53:29 -0600 (CST) Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, 7 Dec 2011, Xiangdong Liang wrote: > I think I did have 10 zeros entries. In my script, I have inserted > zeros by MatSetValue(A,i, (i+1)%N, 0.0, ADD_VALUES); Why doesn't that > option matter in my case? That option prevents PETSc from inserting > zeros, doesn't it? Thank you. MatSetValue is a macro - perhaps its not processing the above properly. Try: PetscScalar zero=0.0 MatSetValue(A,i, (i+1)%N, zero, ADD_VALUES) And if its still not inzerting these zeros - check in a debugger. Satish From balay at mcs.anl.gov Wed Dec 7 10:56:35 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 7 Dec 2011 10:56:35 -0600 (CST) Subject: [petsc-users] petsc with scalapack and blacs from intel ? In-Reply-To: <4EDF5F33.3070602@gfz-potsdam.de> References: <4EDF5F33.3070602@gfz-potsdam.de> Message-ID: blacs is dependent on mpi - so if using libmkl_blacs_openmpi_lp64.a - make sure you use the right version of openmpi - thats compatible with it. [if not - choose the appropriate variant of blacs for your mpi] BTW: PETSc does not use blas/scalapack - its primarily needed by mumps. Satish On Wed, 7 Dec 2011, Alexander Grayver wrote: > Hi Reddy, > > You should use --with-scalapack-lib/include and --with-blacs-lib/include > > For instance: > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > Regards, > Alexander > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > Hello, > > Can you tell me what should be the config options to use the > > scalapack and blacs provided in intel mkl with petsc ? > > > > thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > From xdliang at gmail.com Wed Dec 7 11:02:48 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 12:02:48 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 11:53 AM, Satish Balay wrote: > On Wed, 7 Dec 2011, Xiangdong Liang wrote: > >> I think I did have 10 zeros entries. In my script, I have inserted >> zeros by MatSetValue(A,i, (i+1)%N, 0.0, ADD_VALUES); Why doesn't that >> option matter in my case? That option prevents PETSc from inserting >> zeros, doesn't it? Thank you. > > MatSetValue is a macro - perhaps its not processing the above properly. > > Try: > > PetscScalar zero=0.0 > MatSetValue(A,i, (i+1)%N, zero, ADD_VALUES) > > And if its still not inzerting these zeros - check in a debugger. Actually, I want PETSc does not insert zero in this case. Is there a way to do that? Thanks. > > Satish From Thomas.Witkowski at tu-dresden.de Wed Dec 7 11:04:44 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 07 Dec 2011 18:04:44 +0100 Subject: [petsc-users] Convert MatNest to MatMpiAij Message-ID: <20111207180444.nzies3ftes8goo48@mail.zih.tu-dresden.de> Is there a possibility to convert a MatNest to a MatMpiAij? Performance does not matter as this is for debugging purposes only. I tried to use MatConvert, but this does not work for these types. The MatNest I've in mind is a 3x3 matrix, each either a MatMpiAij or an empty matrix. Thomas From xdliang at gmail.com Wed Dec 7 11:04:52 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 12:04:52 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 4:44 AM, Matthew Knepley wrote: > On Tue, Dec 6, 2011 at 11:39 PM, Xiangdong Liang wrote: >> >> On Wed, Dec 7, 2011 at 12:11 AM, Barry Smith wrote: >> > >> > On Dec 6, 2011, at 10:26 PM, Xiangdong Liang wrote: >> > >> >> Hello everyone, >> >> >> >> Can anyone help on the usage of MAT_IGNORE_ZERO_ENTRIES? Suppose I >> >> want to have a 10-by-10 sparse matrix, where each row has maximum two >> >> nonzeros. I would like to have petsc ignore the 0.0 ?values by not >> >> allocating space for 0.0 values. In my problem, the nonzero pattern of >> >> sparse matrix will never change. I hope the ignore of zero entries >> >> will save lots of operations when I am doing matrix-vector product or >> >> spare-direct decomposition . >> >> >> >> Below is my script. When I check the mat_view_info, I have >> >> >> >> Matrix Object: >> >> type=mpiaij, rows=10, cols=10 >> >> total: nonzeros=20, allocated nonzeros=40 >> >> ? not using I-node (on process 0) routines >> >> >> >> Is there a way to have nonzeros=10, and allocated nonzeros=10, since >> >> the number of real nonzeros=10? Thanks. >> > >> > ? Those "extra" nonzeros are not actually in the sparse matrix so they >> > do not cause extra multiples. You can safely ignore them. >> >> I am sorry I am still not clear. ? Do "extra" nonzeros refer to >> ?nonzeros=20? or "allocated nonzeros=40"? With or without >> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same >> mat_view_info. How can I see the differences in the spare matrix >> generated With/Without this option? Thanks. > > > If there are 2 nonzeros per row, there are 20 nonzeros not 10 in the matrix. > According to > the output, you allocated 40 nonzeros for the matrix. Now, I understand the "allocated nonzeros=40". How can I bring the other one " total nonzero=20" down to " total nonzero=10", since 10 out of 20 values are 0.0? Thanks. > > When you call MatAssemblyEnd(), we compress the matrix so that the 20 extra > allocated nonzeros > are not present. > > ? Matt > >> >> >> Xiangdong >> >> >> > >> > ?Barry >> > >> >> >> >> Xiangdong >> >> >> >> ----------------------------------------------------------- >> >> ?Mat A; ?int N=10, i, ns,ne; >> >> >> >> ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,N,N,2,NULL,2,NULL,&A);CHKERRQ(ierr); >> >> ? ierr = MatGetOwnershipRange(A, &ns, &ne); CHKERRQ(ierr); >> >> ? for (i=ns;i> >> ADD_VALUES); CHKERRQ(ierr); ? ? ?ierr=MatSetValue(A,i, (i+1)%N, 0.0, >> >> ADD_VALUES); CHKERRQ(ierr); ? ?} >> >> ? ? MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); ? ? ierr = >> >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ? ? ierr = >> >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >> >> >> >> -------------------------------------------------------- >> >> >> >> >> >> On Thu, Dec 1, 2011 at 10:01 PM, Barry Smith >> >> wrote: >> >>> >> >>> ?The checks in the code are lines like >> >>> >> >>> ? ? ?if (value == 0.0 && ignorezeroentries && (is == ADD_VALUES)) >> >>> continue; >> >>> >> >>> hence changing the source code ?to ignore small values is not >> >>> difficult. ?But I think this is a terribly cheesy hack and would hope people >> >>> would not do this. >> >>> >> >>> ? ?Barry >> >>> >> >>> >> >>> >> >>> On Dec 1, 2011, at 8:42 PM, Mohamad M. Nasr-Azadani wrote: >> >>> >> >>>> I was always wondering how difficult it would be to define a constant >> >>>> say, ZERO_DROP_TOLERANCE so that when the matrix is setup, >> >>>> MatAssemblyBegin()/End(), it automatically ignores the values below this >> >>>> threshold? >> >>>> I understand the for the matrices that might change in successive >> >>>> iterations, that might not be a good idea since the memory reallocation >> >>>> could be very slow, but for a lot of cases which a constant matrix, that >> >>>> could be simply helpful. >> >>>> >> >>>> Best, >> >>>> Mohamad >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Dec 1, 2011 at 5:00 PM, Barry Smith >> >>>> wrote: >> >>>> >> >>>> On Dec 1, 2011, at 6:44 PM, Xiangdong Liang wrote: >> >>>> >> >>>>> Hello everyone, >> >>>>> >> >>>>> I have a question about whether zeros will change the sparse >> >>>>> pattern. >> >>>>> Suppose I am generating two sparse matrices A and B. A is generated >> >>>>> with exactly number of nonzeros, say 10 nnz per row. When I generate >> >>>>> matrix B, I specify 12 nonzeros, but two of them are given zero >> >>>>> values >> >>>>> by MatSetValue. Will A and B have same sparsity patterns and >> >>>>> performance ? In other words, will PETSc simply ignore these zeros >> >>>>> entries? ?I output these A and B in matlab, and it seems that these >> >>>>> zeros entries are ignored. I just want to double check on this. >> >>>>> >> >>>>> Is the option MAT_IGNORE_ZERO_ENTRIES on by default? >> >>>> >> >>>> ? No, by default PETSc will insert the zero values thus increasing >> >>>> the nonzero pattern. If you set that option to the matrix then inserting >> >>>> values it will not introduce an increase in the pattern. >> >>>> >> >>>> >> >>>> ? ?BTW: if you then load the matrix via a binary file to MATLAB it >> >>>> may be that MATLAB silently removes those locations so you don't see them. >> >>>> ? Barry >> >>>> >> >>>>> Thanks. >> >>>>> >> >>>>> Best, >> >>>>> Xiangdong >> >>>> >> >>>> >> >>> >> > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From jedbrown at mcs.anl.gov Wed Dec 7 11:05:42 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 7 Dec 2011 09:05:42 -0800 Subject: [petsc-users] Convert MatNest to MatMpiAij In-Reply-To: <20111207180444.nzies3ftes8goo48@mail.zih.tu-dresden.de> References: <20111207180444.nzies3ftes8goo48@mail.zih.tu-dresden.de> Message-ID: On Wed, Dec 7, 2011 at 09:04, Thomas Witkowski < Thomas.Witkowski at tu-dresden.de> wrote: > Is there a possibility to convert a MatNest to a MatMpiAij? Performance > does not matter as this is for debugging purposes only. I tried to use > MatConvert, but this does not work for these types. The MatNest I've in > mind is a 3x3 matrix, each either a MatMpiAij or an empty matrix. Yes, it makes sense. I will implement it soon. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Dec 7 11:07:06 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 7 Dec 2011 11:07:06 -0600 (CST) Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, 7 Dec 2011, Xiangdong Liang wrote: > >> >> Matrix Object: > >> >> type=mpiaij, rows=10, cols=10 > >> >> total: nonzeros=20, allocated nonzeros=40 > >> >> ? not using I-node (on process 0) routines > >> >> > >> >> Is there a way to have nonzeros=10, and allocated nonzeros=10, since > >> >> the number of real nonzeros=10? Thanks. > >> > > >> > ? Those "extra" nonzeros are not actually in the sparse matrix so they > >> > do not cause extra multiples. You can safely ignore them. > >> > >> I am sorry I am still not clear. ? Do "extra" nonzeros refer to > >> ?nonzeros=20? or "allocated nonzeros=40"? With or without > >> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same > >> mat_view_info. How can I see the differences in the spare matrix > >> generated With/Without this option? Thanks. > > > > > > If there are 2 nonzeros per row, there are 20 nonzeros not 10 in the matrix. > > According to > > the output, you allocated 40 nonzeros for the matrix. > > Now, I understand the "allocated nonzeros=40". How can I bring the > other one " total nonzero=20" down to " total nonzero=10", since 10 > out of 20 values are 0.0? Thanks. As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) - before any calls to MatSetValue/s() And verify with MatView() Satish From xdliang at gmail.com Wed Dec 7 11:10:50 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 12:10:50 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, Dec 7, 2011 at 12:07 PM, Satish Balay wrote: > On Wed, 7 Dec 2011, Xiangdong Liang wrote: > >> >> >> Matrix Object: >> >> >> type=mpiaij, rows=10, cols=10 >> >> >> total: nonzeros=20, allocated nonzeros=40 >> >> >> ? not using I-node (on process 0) routines >> >> >> >> >> >> Is there a way to have nonzeros=10, and allocated nonzeros=10, since >> >> >> the number of real nonzeros=10? Thanks. >> >> > >> >> > ? Those "extra" nonzeros are not actually in the sparse matrix so they >> >> > do not cause extra multiples. You can safely ignore them. >> >> >> >> I am sorry I am still not clear. ? Do "extra" nonzeros refer to >> >> ?nonzeros=20? or "allocated nonzeros=40"? With or without >> >> MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE), I see the same >> >> mat_view_info. How can I see the differences in the spare matrix >> >> generated With/Without this option? Thanks. >> > >> > >> > If there are 2 nonzeros per row, there are 20 nonzeros not 10 in the matrix. >> > According to >> > the output, you allocated 40 nonzeros for the matrix. >> >> Now, I understand the "allocated nonzeros=40". How can I bring the >> other one " total nonzero=20" down to " total nonzero=10", since 10 >> out of 20 values are 0.0? Thanks. > > As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) > - before any calls to MatSetValue/s() Great! It works now. I should pay attention to the order. Thank you. With mat_view_info, now it looks like: Matrix Object: type=mpiaij, rows=10, cols=10 total: nonzeros=10, allocated nonzeros=40 not using I-node (on process 0) routines > > And verify with MatView() > > Satish From balay at mcs.anl.gov Wed Dec 7 11:15:16 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 7 Dec 2011 11:15:16 -0600 (CST) Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Wed, 7 Dec 2011, Satish Balay wrote: > As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) > - before any calls to MatSetValue/s() > > And verify with MatView() Ok - here is what you can see with a slightly modified mat/examples/tests/ex38.c Notice "C[0,1] = 0.0" and thats not visible anymore with MatSetOption(MAT_IGNORE_ZERO_ENTRIES) Satish -------- asterix:/home/balay/spetsc/src/mat/examples/tests>cat ex38.c static char help[] = "Tests MatSetValues().\n\n"; #include #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **args) { Mat C; PetscInt i,n = 5,midx[3],nidx[2]; PetscErrorCode ierr; PetscBool flg; PetscScalar v[6]; PetscInitialize(&argc,&args,(char *)0,help); ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ierr = MatSetFromOptions(C);CHKERRQ(ierr); ierr = PetscOptionsHasName(PETSC_NULL,"-ignore_zero_entries",&flg);CHKERRQ(ierr); if (flg) { ierr = MatSetOption(C,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE);CHKERRQ(ierr);} for (i=0; i<6; i++) v[i] = (PetscReal)i; midx[0] = 0; midx[1] = 2; midx[2] = 3; nidx[0] = 1; nidx[1] = 3; ierr = MatSetValues(C,3,midx,2,nidx,v,ADD_VALUES);CHKERRQ(ierr); ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatView(C,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = MatDestroy(&C);CHKERRQ(ierr); ierr = PetscFinalize(); return 0; } asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info Matrix Object: 1 MPI processes type: seqaij rows=5, cols=5 total: nonzeros=6, allocated nonzeros=25 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 4 nodes, limit used is 5 Matrix Object: 1 MPI processes type: seqaij row 0: (1, 0) (3, 1) row 1: row 2: (1, 2) (3, 3) row 3: (1, 4) (3, 5) row 4: asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info -ignore_zero_entries Matrix Object: 1 MPI processes type: seqaij rows=5, cols=5 total: nonzeros=5, allocated nonzeros=25 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 4 nodes, limit used is 5 Matrix Object: 1 MPI processes type: seqaij row 0: (3, 1) row 1: row 2: (1, 2) (3, 3) row 3: (1, 4) (3, 5) row 4: asterix:/home/balay/spetsc/src/mat/examples/tests> From bsmith at mcs.anl.gov Wed Dec 7 12:54:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 7 Dec 2011 12:54:01 -0600 Subject: [petsc-users] Problem with Optimised version .vs. Debug version In-Reply-To: References: Message-ID: <40EF4C1B-511F-4FED-B2A7-EB972C3D838E@mcs.anl.gov> On Dec 7, 2011, at 9:53 AM, Jed Brown wrote: > On Wed, Dec 7, 2011 at 06:31, Bogdan Dita wrote: > A few weeks ago i posted a question regarding a problem with the > optimised version of PETSc, what i mean is that I obtained a higher > solving time with the optimised version then with the debug > version(~10-15% higher). > > What do you get from 'mpicc -show'? > > MPICH2 has a tendency to put optimization flags into the wrappers. It's possible that the "optimized" build of PETSc is effectively using -O2 (from MPICH2) and -g3 (from PETSc with-debugging=1), which might be faster than -O2 (from MPICH2) -O (from PETSc with-debugging=0, taking precedence) that you get when you. You can use PETSc's --COPTFLAGS to set your own optimization flags for PETSc. > > Let us know if this works. If this doesn't work produce the results from running the same code using the -log_summary option with and without debug. Then send the two outputs to petsc-maint at mcs.anl.gov this will tell us what parts of the code (all?) are slower and faster in the different versions. Barry From balay at mcs.anl.gov Wed Dec 7 12:57:36 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 7 Dec 2011 12:57:36 -0600 (CST) Subject: [petsc-users] several messages In-Reply-To: References: <4EDF5F33.3070602@gfz-potsdam.de> Message-ID: >>> /opt/petsc/petsc-3.2-p5/mpi_cScalar/include/petscconfiginfo.h(3): error: missing closing quote "; ^ <<< Can you send in this file as attachment? >>>>>>>>> Configure Options: --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --with-petsc-arch=mpi_cScalar --with-mpi-dir=/opt/mpich2-current --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/lib/intel64 --download-umfpack=1 --with-exodusii-dir=/opt/exodusii-4.98 --with-netcdf-dir=/opt/netcdf_v4.1.1 --download-superlu_dist=1 --download-parmetis=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1^M <<<<<<<< Not sure how the ^M char crept into the configure run - but perhaps its causing the petscconfiginfo.h file to be not created correctly. Satish On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > Hello, > Can you tell me what should be the config options to use the > scalapack and blacs provided in intel mkl with petsc ? > > thanks > Reddy > > On Wed, 7 Dec 2011, Alexander Grayver wrote: > Hi Reddy, > > You should use --with-scalapack-lib/include and --with-blacs-lib/include > > For instance: > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > Regards, > Alexander > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > Hello, > > Can you tell me what should be the config options to use the > > scalapack and blacs provided in intel mkl with petsc ? > > > > thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > I sent this email to petsc-users, it bounced back. Can someone please > clarify the following query.. > > ---------- Forwarded message ---------- > From: Dharmendar Reddy > Date: Wed, Dec 7, 2011 at 7:42 AM > Subject: Re: [petsc-users] petsc with scalapack and blacs from intel ? > To: PETSc users list > > > I tried --with-scalapack-dir=/opt/inte/mkl/ --with-blacs-dir=/opt/intel/mkl > and now with the method suggested by you > I am getting an error. I have attached the configure.log > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > --with-blacs-lib=['/opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a'] > and > --with-blacs-include=['/opt/intel/mkl/include'] did not work > ******************************************************************************* > > > On Wed, Dec 7, 2011 at 6:42 AM, Alexander Grayver > wrote: > > > ** > > Hi Reddy, > > > > You should use --with-scalapack-lib/include and --with-blacs-lib/include > > > > For instance: > > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > > > Regards, > > Alexander > > > > > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > > > Hello, > > Can you tell me what should be the config options to use the > > scalapack and blacs provided in intel mkl with petsc ? > > > > thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > > > > > > On Wed, 7 Dec 2011, Satish Balay wrote: > > --with-mpi-dir=/opt/mpich2-current > > I guess thats incompatible with /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a > > > its easier just to use --download-blacs --download-scalapack. > > Satish > > > On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > > > I sent this email to petsc-users, it bounced back. Can someone please > > clarify the following query.. > > > > ---------- Forwarded message ---------- > > From: Dharmendar Reddy > > Date: Wed, Dec 7, 2011 at 7:42 AM > > Subject: Re: [petsc-users] petsc with scalapack and blacs from intel ? > > To: PETSc users list > > > > > > I tried --with-scalapack-dir=/opt/inte/mkl/ --with-blacs-dir=/opt/intel/mkl > > and now with the method suggested by you > > I am getting an error. I have attached the configure.log > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > ------------------------------------------------------------------------------- > > --with-blacs-lib=['/opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a'] > > and > > --with-blacs-include=['/opt/intel/mkl/include'] did not work > > ******************************************************************************* > > > > > > On Wed, Dec 7, 2011 at 6:42 AM, Alexander Grayver > > wrote: > > > > > ** > > > Hi Reddy, > > > > > > You should use --with-scalapack-lib/include and --with-blacs-lib/include > > > > > > For instance: > > > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > > > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > > > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > > > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > > > > > Regards, > > > Alexander > > > > > > > > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > > > > > Hello, > > > Can you tell me what should be the config options to use the > > > scalapack and blacs provided in intel mkl with petsc ? > > > > > > thanks > > > Reddy > > > > > > -- > > > ----------------------------------------------------- > > > Dharmendar Reddy Palle > > > Graduate Student > > > Microelectronics Research center, > > > University of Texas at Austin, > > > 10100 Burnet Road, Bldg. 160 > > > MER 2.608F, TX 78758-4445 > > > e-mail: dharmareddy84 at gmail.com > > > Phone: +1-512-350-9082 > > > United States of America. > > > > > > > > > > > > > > > > > > On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > I was trying to install mumps and then i got messages that i need to > install scalapack and blacs. I will try the --download options.. > > On Wed, Dec 7, 2011 at 10:57 AM, Satish Balay wrote: > > > > --with-mpi-dir=/opt/mpich2-current > > > > I guess thats incompatible with > > /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a > > > > > > its easier just to use --download-blacs --download-scalapack. > > > > Satish > > > > > > On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > > > > > I sent this email to petsc-users, it bounced back. Can someone please > > > clarify the following query.. > > > > > > ---------- Forwarded message ---------- > > > From: Dharmendar Reddy > > > Date: Wed, Dec 7, 2011 at 7:42 AM > > > Subject: Re: [petsc-users] petsc with scalapack and blacs from intel ? > > > To: PETSc users list > > > > > > > > > I tried --with-scalapack-dir=/opt/inte/mkl/ > > --with-blacs-dir=/opt/intel/mkl > > > and now with the method suggested by you > > > I am getting an error. I have attached the configure.log > > > > > > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > details): > > > > > ------------------------------------------------------------------------------- > > > > > --with-blacs-lib=['/opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a'] > > > and > > > --with-blacs-include=['/opt/intel/mkl/include'] did not work > > > > > ******************************************************************************* > > > > > > > > > On Wed, Dec 7, 2011 at 6:42 AM, Alexander Grayver > > > wrote: > > > > > > > ** > > > > Hi Reddy, > > > > > > > > You should use --with-scalapack-lib/include and > > --with-blacs-lib/include > > > > > > > > For instance: > > > > > > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > > > > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > > > > > > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > > > > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > > > > > > > Regards, > > > > Alexander > > > > > > > > > > > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > > > > > > > Hello, > > > > Can you tell me what should be the config options to use the > > > > scalapack and blacs provided in intel mkl with petsc ? > > > > > > > > thanks > > > > Reddy > > > > > > > > -- > > > > ----------------------------------------------------- > > > > Dharmendar Reddy Palle > > > > Graduate Student > > > > Microelectronics Research center, > > > > University of Texas at Austin, > > > > 10100 Burnet Road, Bldg. 160 > > > > MER 2.608F, TX 78758-4445 > > > > e-mail: dharmareddy84 at gmail.com > > > > Phone: +1-512-350-9082 > > > > United States of America. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 7 Dec 2011, Satish Balay wrote: > blacs is dependent on mpi - so if using libmkl_blacs_openmpi_lp64.a - > make sure you use the right version of openmpi - thats compatible with > it. [if not - choose the appropriate variant of blacs for your mpi] > > BTW: PETSc does not use blas/scalapack - its primarily needed by mumps. > > Satish > > On Wed, 7 Dec 2011, Alexander Grayver wrote: > > > Hi Reddy, > > > > You should use --with-scalapack-lib/include and --with-blacs-lib/include > > > > For instance: > > --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > > > > Regards, > > Alexander > > > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > > > Hello, > > > Can you tell me what should be the config options to use the > > > scalapack and blacs provided in intel mkl with petsc ? > > > > > > thanks > > > Reddy > > > > > > -- > > > ----------------------------------------------------- > > > Dharmendar Reddy Palle > > > Graduate Student > > > Microelectronics Research center, > > > University of Texas at Austin, > > > 10100 Burnet Road, Bldg. 160 > > > MER 2.608F, TX 78758-4445 > > > e-mail: dharmareddy84 at gmail.com > > > Phone: +1-512-350-9082 > > > United States of America. > > > > > > > > > On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > I could configure with download options but now i get make error. Can you > help me with this.. > > > On Wed, Dec 7, 2011 at 11:06 AM, Dharmendar Reddy > wrote: > > > I was trying to install mumps and then i got messages that i need to > > install scalapack and blacs. I will try the --download options.. > > > > > > On Wed, Dec 7, 2011 at 10:57 AM, Satish Balay wrote: > > > >> > --with-mpi-dir=/opt/mpich2-current > >> > >> I guess thats incompatible with > >> /opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a > >> > >> > >> its easier just to use --download-blacs --download-scalapack. > >> > >> Satish > >> > >> > >> On Wed, 7 Dec 2011, Dharmendar Reddy wrote: > >> > >> > I sent this email to petsc-users, it bounced back. Can someone please > >> > clarify the following query.. > >> > > >> > ---------- Forwarded message ---------- > >> > From: Dharmendar Reddy > >> > Date: Wed, Dec 7, 2011 at 7:42 AM > >> > Subject: Re: [petsc-users] petsc with scalapack and blacs from intel ? > >> > To: PETSc users list > >> > > >> > > >> > I tried --with-scalapack-dir=/opt/inte/mkl/ > >> --with-blacs-dir=/opt/intel/mkl > >> > and now with the method suggested by you > >> > I am getting an error. I have attached the configure.log > >> > > >> > > >> ******************************************************************************* > >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > >> for > >> > details): > >> > > >> ------------------------------------------------------------------------------- > >> > > >> --with-blacs-lib=['/opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a'] > >> > and > >> > --with-blacs-include=['/opt/intel/mkl/include'] did not work > >> > > >> ******************************************************************************* > >> > > >> > > >> > On Wed, Dec 7, 2011 at 6:42 AM, Alexander Grayver > >> > wrote: > >> > > >> > > ** > >> > > Hi Reddy, > >> > > > >> > > You should use --with-scalapack-lib/include and > >> --with-blacs-lib/include > >> > > > >> > > For instance: > >> > > > >> --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a > >> > > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > >> > > > >> --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a > >> > > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > >> > > > >> > > Regards, > >> > > Alexander > >> > > > >> > > > >> > > On 07.12.2011 13:26, Dharmendar Reddy wrote: > >> > > > >> > > Hello, > >> > > Can you tell me what should be the config options to use the > >> > > scalapack and blacs provided in intel mkl with petsc ? > >> > > > >> > > thanks > >> > > Reddy > >> > > > >> > > -- > >> > > ----------------------------------------------------- > >> > > Dharmendar Reddy Palle > >> > > Graduate Student > >> > > Microelectronics Research center, > >> > > University of Texas at Austin, > >> > > 10100 Burnet Road, Bldg. 160 > >> > > MER 2.608F, TX 78758-4445 > >> > > e-mail: dharmareddy84 at gmail.com > >> > > Phone: +1-512-350-9082 > >> > > United States of America. > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > > > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > > > > From bogdan at lmn.pub.ro Wed Dec 7 18:16:13 2011 From: bogdan at lmn.pub.ro (Bogdan Dita) Date: Thu, 8 Dec 2011 02:16:13 +0200 Subject: [petsc-users] Optimised vs Debug Message-ID: <2ac65e14091561bacf5a13a44a3acee4.squirrel@wm.lmn.pub.ro> Thanks Jed, this is the kind of answer i really needed to hear(I hope it works), I was sure that something in the shadows of mpich2 didn't act acording to my expectations but honestly I didn't even knew where to start looking. I'll try this and then get back to you. To Barry: Regarding the output from -log_summary for the debug and opt versions, I already sent them a few weeks back in a mail to 'petsc-users', your answer was that the time increase came from the factorization of the matrix. If it will be necessary in my next post I'll attache the two files, but first I want to try Jed's idea. Thanks, Bogdan > On Wed, Dec 7, 2011 at 06:31, Bogdan Dita wrote: > >> A few weeks ago i posted a question regarding a problem with the >> optimised version of PETSc, what i mean is that I obtained a higher >> solving time with the optimised version then with the debug >> version(~10-15% higher). >> > > What do you get from 'mpicc -show'? > > MPICH2 has a tendency to put optimization flags into the wrappers. It's > possible that the "optimized" build of PETSc is effectively using -O2 > (from > MPICH2) and -g3 (from PETSc with-debugging=1), which might be faster than > -O2 (from MPICH2) -O (from PETSc with-debugging=0, taking precedence) that > you get when you. You can use PETSc's --COPTFLAGS to set your own > optimization flags for PETSc. > > Let us know if this works. > If this doesn't work produce the results from running the same code >using the >-log_summary option with and without debug. Then send the two outputs to >petsc-maint at mcs.anl.gov this will tell us what parts of the code (all?) >are slower >and faster in the different versions. > > Barry From xdliang at gmail.com Wed Dec 7 22:41:19 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 7 Dec 2011 23:41:19 -0500 Subject: [petsc-users] mat_view_info and the number of matrix objects Message-ID: Hello everyone, I am using mat_view_info to see the matrix information in ksp/ksp/ex2.c: http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html When I use single processor, I see two matrix-objects. One is the operator A, but what's the other one? It seems that the other one (always in seqaij or seqsbaij format) is from KSPSolve. What's the purpose of that matrix object? If I try two processors, I see five matrix-objects (more confused). Can you help me understand where these "five" matrices come from? Thanks. Best, Xiangdong Here is the script: ----------------------- ./ex2 -mat_view_info Matrix Object: 1 MPI processes type: seqaij rows=56, cols=56 total: nonzeros=250, allocated nonzeros=280 total number of mallocs used during MatSetValues calls =0 not using I-node routines Matrix Object: 1 MPI processes type: seqsbaij rows=56, cols=56 package used to perform factorization: petsc total: nonzeros=153, allocated nonzeros=153 total number of mallocs used during MatSetValues calls =0 block size is 1 Norm of error 0.000156044 iterations 6 -------------------------------- mpirun -np 2 ./ex2 -mat_view_info Matrix Object: 2 MPI processes type: mpiaij rows=56, cols=56 total: nonzeros=250, allocated nonzeros=560 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Matrix Object: 1 MPI processes type: seqaij rows=28, cols=28 total: nonzeros=118, allocated nonzeros=118 total number of mallocs used during MatSetValues calls =0 not using I-node routines Matrix Object: 1 MPI processes type: seqsbaij rows=28, cols=28 package used to perform factorization: petsc total: nonzeros=73, allocated nonzeros=73 total number of mallocs used during MatSetValues calls =0 block size is 1 Matrix Object: 1 MPI processes type: seqaij rows=28, cols=28 total: nonzeros=118, allocated nonzeros=118 total number of mallocs used during MatSetValues calls =0 not using I-node routines Matrix Object: 1 MPI processes type: seqsbaij rows=28, cols=28 package used to perform factorization: petsc total: nonzeros=73, allocated nonzeros=73 total number of mallocs used during MatSetValues calls =0 block size is 1 Norm of error 0.000411674 iterations 7 From bsmith at mcs.anl.gov Wed Dec 7 22:49:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 7 Dec 2011 22:49:41 -0600 Subject: [petsc-users] mat_view_info and the number of matrix objects In-Reply-To: References: Message-ID: On Dec 7, 2011, at 10:41 PM, Xiangdong Liang wrote: > Hello everyone, > > I am using mat_view_info to see the matrix information in ksp/ksp/ex2.c: > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html > > When I use single processor, I see two matrix-objects. One is the > operator A, but what's the other one? It seems that the other one > (always in seqaij or seqsbaij format) is from KSPSolve. What's the > purpose of that matrix object? > > If I try two processors, I see five matrix-objects (more confused). > Can you help me understand where these "five" matrices come from? > Thanks. > > Best, > Xiangdong The -mat_view_info is only of limited usefulness.Every time a matrix assembly is done or the end of a factorization if the -mat_view_info argument is passed it prints information about the matrix. The reason for the "two" is the second one is from the PC of ILU which generated a factored matrix and info is printed on that In parallel it also prints information about the sequential matrices that are stored inside the parallel matrix so you get the original matrix, then two sequential matrices inside it for each process. If you want information about a particular matrix in your code you are better off using MatGetInfo() then trying to use the command line option. Barry > > > > > Here is the script: > > ----------------------- > ./ex2 -mat_view_info > > Matrix Object: 1 MPI processes > type: seqaij > rows=56, cols=56 > total: nonzeros=250, allocated nonzeros=280 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Matrix Object: 1 MPI processes > type: seqsbaij > rows=56, cols=56 > package used to perform factorization: petsc > total: nonzeros=153, allocated nonzeros=153 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > Norm of error 0.000156044 iterations 6 > > -------------------------------- > > mpirun -np 2 ./ex2 -mat_view_info > > Matrix Object: 2 MPI processes > type: mpiaij > rows=56, cols=56 > total: nonzeros=250, allocated nonzeros=560 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Matrix Object: 1 MPI processes > type: seqaij > rows=28, cols=28 > total: nonzeros=118, allocated nonzeros=118 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Matrix Object: 1 MPI processes > type: seqsbaij > rows=28, cols=28 > package used to perform factorization: petsc > total: nonzeros=73, allocated nonzeros=73 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > Matrix Object: 1 MPI processes > type: seqaij > rows=28, cols=28 > total: nonzeros=118, allocated nonzeros=118 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Matrix Object: 1 MPI processes > type: seqsbaij > rows=28, cols=28 > package used to perform factorization: petsc > total: nonzeros=73, allocated nonzeros=73 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > Norm of error 0.000411674 iterations 7 From xdliang at gmail.com Thu Dec 8 01:11:44 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Thu, 8 Dec 2011 02:11:44 -0500 Subject: [petsc-users] petsc 3.2 compiling on cray Message-ID: Hello everyone, I am trying to compile petsc-3.2 on cray. However, when I specify blas and lapack library --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, libacml_mv.a], petsc complains like this: =============================================================================== Configuring PETSc to compile on your system =============================================================================== ******************************************************************************* ERROR in COMMAND LINE ARGUMENT to ./configure ------------------------------------------------------------------------------- The option libacml_mv.a] should probably be libacml-mv.a] ******************************************************************************* However, the library is indeed named as libacml_mv.a. I can compile successfully without specifying the lapack and blas libraries. Will it affect the performance? Thanks Xiangdong The configure option I use is: ./config/configure.py --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, libacml_mv.a] \ --with-cc=cc --with-cxx=CC --with-fc=ftn -with-shared-lib\ raries=0 --COPTFLAGS='-fastsse -O3 -tp istanbul-64' --CXXOPTFLAGS='-fastsse -O3\ -tp istanbul-64' --FOPTFLAGS='-fastsse -O3 -tp istanbul-64' --with-debugging=0\ --prefix=$MYINSTALL --PETSC_ARCH=arch-opt --with-clib-autodetect=0 --\ with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --download-blacs=yes --\ download-scalapack=yes --download-parmetis=yes --download-superlu=yes --downloa\ d-superlu_dist=yes --download-scotch=yes --download-pastix=yes --download-mumps\ =yes --download-spooles=yes --with-hdf5=yes --with-hdf5-dir=/opt/cray/hdf5/1.8.\ 5.0/hdf5-gnu/ --download-ptscotch --download-umfpack=yes --download-hypre=yes -\ -download-spai=yes From Thomas.Witkowski at tu-dresden.de Thu Dec 8 07:49:07 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 08 Dec 2011 14:49:07 +0100 Subject: [petsc-users] Problem: matrix is missing diagonal entry Message-ID: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> I've some strange problem: I create a mpiaij matrix, fill it with some values and want to make use of it in kspsolve. But I get the error [15]PETSC ERROR: Object is in wrong state! [15]PETSC ERROR: Matrix is missing diagonal entry 1129! I've checked the following things: - There is a MatSetValues call that contains the entry 1129/1129 with some value - MatView shows that there exists the value 1129/1129 - MatAssemblyBegin/End is called correctly - valgrind says: 0 errors - ddt also cannot find anything unusual I use petsc 3.2-p5, its compiled in debug mode. What could I check to solve this problem? Any hints? Thomas From Thomas.Witkowski at tu-dresden.de Thu Dec 8 07:51:33 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 08 Dec 2011 14:51:33 +0100 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> Message-ID: <20111208145133.da67vywg04gk88s0@mail.zih.tu-dresden.de> Maybe some more note on my problem: I've thounds of matrices in my code, where doing this works fine! The only difference here is, that row 1129 of the matrix is owned by rank 0, but all of its entries are created on rank 1. For all of the other matrices in my codes, at least the diagonal element is created on the rank that owns the matrix row. Thomas Zitat von Thomas Witkowski : > I've some strange problem: I create a mpiaij matrix, fill it with some > values and want to make use of it in kspsolve. But I get the error > > [15]PETSC ERROR: Object is in wrong state! > [15]PETSC ERROR: Matrix is missing diagonal entry 1129! > > I've checked the following things: > - There is a MatSetValues call that contains the entry 1129/1129 with > some value > - MatView shows that there exists the value 1129/1129 > - MatAssemblyBegin/End is called correctly > - valgrind says: 0 errors > - ddt also cannot find anything unusual > > I use petsc 3.2-p5, its compiled in debug mode. What could I check to > solve this problem? Any hints? > > Thomas From knepley at gmail.com Thu Dec 8 08:03:04 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Dec 2011 08:03:04 -0600 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: <20111208145133.da67vywg04gk88s0@mail.zih.tu-dresden.de> References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> <20111208145133.da67vywg04gk88s0@mail.zih.tu-dresden.de> Message-ID: On Thu, Dec 8, 2011 at 7:51 AM, Thomas Witkowski < Thomas.Witkowski at tu-dresden.de> wrote: > Maybe some more note on my problem: I've thounds of matrices in my code, > where doing this works fine! The only difference here is, that row 1129 of > the matrix is owned by rank 0, but all of its entries are created on rank > 1. For all of the other matrices in my codes, at least the diagonal element > is created on the rank that owns the matrix row. > Did you remember to call MatAssemblyBegin/End()? Matt > Thomas > > Zitat von Thomas Witkowski > >: > > > I've some strange problem: I create a mpiaij matrix, fill it with some >> values and want to make use of it in kspsolve. But I get the error >> >> [15]PETSC ERROR: Object is in wrong state! >> [15]PETSC ERROR: Matrix is missing diagonal entry 1129! >> >> I've checked the following things: >> - There is a MatSetValues call that contains the entry 1129/1129 with >> some value >> - MatView shows that there exists the value 1129/1129 >> - MatAssemblyBegin/End is called correctly >> - valgrind says: 0 errors >> - ddt also cannot find anything unusual >> >> I use petsc 3.2-p5, its compiled in debug mode. What could I check to >> solve this problem? Any hints? >> >> Thomas >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 8 08:03:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 8 Dec 2011 08:03:24 -0600 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> Message-ID: <4935A058-DAF6-41E6-A6D9-C549E854965D@mcs.anl.gov> Thomas, As always including the ENTIRE error message makes determining the issue much easier. It tells us what functions are being called at this point and the line numbers. Without we would only be speculating. In this case the error message is printed on process of rank 15 at location 1129 ON THAT PROCESS so the you need to determine the global row of location 1129 on the 15th process. That row is missing the diagonal. Barry The reason I know it is the not the global row 1129 but the 1129th row on process 15 is because the I know the check for diagonal entries is done in the factorization that is taking place on each process (each factorization is running at the same time on different processes). This would be clear with the entire error message. On Dec 8, 2011, at 7:49 AM, Thomas Witkowski wrote: > I've some strange problem: I create a mpiaij matrix, fill it with some values and want to make use of it in kspsolve. But I get the error > > [15]PETSC ERROR: Object is in wrong state! > [15]PETSC ERROR: Matrix is missing diagonal entry 1129! > > I've checked the following things: > - There is a MatSetValues call that contains the entry 1129/1129 with some value > - MatView shows that there exists the value 1129/1129 > - MatAssemblyBegin/End is called correctly > - valgrind says: 0 errors > - ddt also cannot find anything unusual > > I use petsc 3.2-p5, its compiled in debug mode. What could I check to solve this problem? Any hints? > > Thomas From knepley at gmail.com Thu Dec 8 08:05:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Dec 2011 08:05:49 -0600 Subject: [petsc-users] petsc 3.2 compiling on cray In-Reply-To: References: Message-ID: On Thu, Dec 8, 2011 at 1:11 AM, Xiangdong Liang wrote: > Hello everyone, > > I am trying to compile petsc-3.2 on cray. However, when I specify blas > and lapack library > --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, > libacml_mv.a], petsc complains like this: > > > =============================================================================== > Configuring PETSc to compile on your system > > =============================================================================== > > ******************************************************************************* > ERROR in COMMAND LINE ARGUMENT to ./configure > > ------------------------------------------------------------------------------- > The option libacml_mv.a] should probably be libacml-mv.a] > > ******************************************************************************* > If you specify the full path to the library, this should go away. This is just an error check to prevent people from mistyping options. Thanks, Matt > > However, the library is indeed named as libacml_mv.a. I can compile > successfully without specifying the lapack and blas libraries. Will it > affect the performance? Thanks > > Xiangdong > > > > The configure option I use is: > > ./config/configure.py > --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, > libacml_mv.a] \ > --with-cc=cc --with-cxx=CC --with-fc=ftn -with-shared-lib\ > raries=0 --COPTFLAGS='-fastsse -O3 -tp istanbul-64' > --CXXOPTFLAGS='-fastsse -O3\ > -tp istanbul-64' --FOPTFLAGS='-fastsse -O3 -tp istanbul-64' > --with-debugging=0\ > --prefix=$MYINSTALL --PETSC_ARCH=arch-opt --with-clib-autodetect=0 --\ > with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 > --download-blacs=yes --\ > download-scalapack=yes --download-parmetis=yes --download-superlu=yes > --downloa\ > d-superlu_dist=yes --download-scotch=yes --download-pastix=yes > --download-mumps\ > =yes --download-spooles=yes --with-hdf5=yes > --with-hdf5-dir=/opt/cray/hdf5/1.8.\ > 5.0/hdf5-gnu/ --download-ptscotch --download-umfpack=yes > --download-hypre=yes -\ > -download-spai=yes > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Thomas.Witkowski at tu-dresden.de Thu Dec 8 08:07:32 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 08 Dec 2011 15:07:32 +0100 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> <20111208145133.da67vywg04gk88s0@mail.zih.tu-dresden.de> Message-ID: <20111208150732.mik15126g4400oos@mail.zih.tu-dresden.de> Zitat von Matthew Knepley : > > Did you remember to call MatAssemblyBegin/End()? > Yes > Matt > > >> Thomas >> >> Zitat von Thomas Witkowski >> >> >: >> >> >> I've some strange problem: I create a mpiaij matrix, fill it with some >>> values and want to make use of it in kspsolve. But I get the error >>> >>> [15]PETSC ERROR: Object is in wrong state! >>> [15]PETSC ERROR: Matrix is missing diagonal entry 1129! >>> >>> I've checked the following things: >>> - There is a MatSetValues call that contains the entry 1129/1129 with >>> some value >>> - MatView shows that there exists the value 1129/1129 >>> - MatAssemblyBegin/End is called correctly >>> - valgrind says: 0 errors >>> - ddt also cannot find anything unusual >>> >>> I use petsc 3.2-p5, its compiled in debug mode. What could I check to >>> solve this problem? Any hints? >>> >>> Thomas >>> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > From Thomas.Witkowski at tu-dresden.de Thu Dec 8 08:11:35 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 08 Dec 2011 15:11:35 +0100 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: <4935A058-DAF6-41E6-A6D9-C549E854965D@mcs.anl.gov> References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> <4935A058-DAF6-41E6-A6D9-C549E854965D@mcs.anl.gov> Message-ID: <20111208151135.09i8np2rpc444g4k@mail.zih.tu-dresden.de> Okay, if the entry number is the local one, than I can check this again. In this case, I think that I know where the problem is: By the way, the error message is as follows: [15]PETSC ERROR: Object is in wrong state! [15]PETSC ERROR: Matrix is missing diagonal entry 1129! [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 [15]PETSC ERROR: See docs/changes/index.html for recent updates. [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [15]PETSC ERROR: See docs/index.html for manual pages. [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: ./ch2 on a arch-linu named p1q024 by witkowsk Thu Dec 8 13:39:46 2011 [15]PETSC ERROR: Libraries linked from /fastfs/witkowsk/petsc/install/petsc-3.2-p5-dbg/lib [15]PETSC ERROR: Configure run at Fri Dec 2 14:45:50 2011 [15]PETSC ERROR: Configure options --prefix=/fastfs/witkowsk/petsc/install/petsc-3.2-p5-dbg --download-blacs --download-parmetis --download-scalapack --download-umfpac\ k --with-blas-lapack-dir=/work/licsoft/libraries/mkl/9.1.023/lib/em64t --with-c++-support --download-mumps --download-superlu --with-clanguage=cxx [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1636 in src/mat/impls/aij/seq/aijfact.c [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1740 in src/mat/impls/aij/seq/aijfact.c [15]PETSC ERROR: MatILUFactorSymbolic() line 6092 in src/mat/interface/matrix.c [15]PETSC ERROR: PCSetUp_ILU() line 216 in src/ksp/pc/impls/factor/ilu/ilu.c [15]PETSC ERROR: PCSetUp() line 819 in src/ksp/pc/interface/precon.c [15]PETSC ERROR: KSPSetUp() line 260 in src/ksp/ksp/interface/itfunc.c [15]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Multiblock() line 973 in src/ksp/pc/impls/bjacobi/bjacobi.c [15]PETSC ERROR: PCSetUpOnBlocks() line 852 in src/ksp/pc/interface/precon.c [15]PETSC ERROR: KSPSetUpOnBlocks() line 154 in src/ksp/ksp/interface/itfunc.c Zitat von Barry Smith : > > Thomas, > > As always including the ENTIRE error message makes determining > the issue much easier. It tells us what functions are being called > at this point and the line numbers. Without we would only be > speculating. > > In this case the error message is printed on process of rank 15 > at location 1129 ON THAT PROCESS so the you need to determine the > global row of location 1129 on the 15th process. That row is > missing the diagonal. > > Barry > > The reason I know it is the not the global row 1129 but the 1129th > row on process 15 is because the I know the check for diagonal > entries is done in the factorization that is taking place on each > process (each factorization is running at the same time on different > processes). This would be clear with the entire error message. > > > On Dec 8, 2011, at 7:49 AM, Thomas Witkowski wrote: > >> I've some strange problem: I create a mpiaij matrix, fill it with >> some values and want to make use of it in kspsolve. But I get the >> error >> >> [15]PETSC ERROR: Object is in wrong state! >> [15]PETSC ERROR: Matrix is missing diagonal entry 1129! >> >> I've checked the following things: >> - There is a MatSetValues call that contains the entry 1129/1129 >> with some value >> - MatView shows that there exists the value 1129/1129 >> - MatAssemblyBegin/End is called correctly >> - valgrind says: 0 errors >> - ddt also cannot find anything unusual >> >> I use petsc 3.2-p5, its compiled in debug mode. What could I check >> to solve this problem? Any hints? >> >> Thomas > > From bsmith at mcs.anl.gov Thu Dec 8 10:03:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 8 Dec 2011 10:03:20 -0600 Subject: [petsc-users] Problem: matrix is missing diagonal entry In-Reply-To: <20111208151135.09i8np2rpc444g4k@mail.zih.tu-dresden.de> References: <20111208144907.tqlz5agu5co00c44@mail.zih.tu-dresden.de> <4935A058-DAF6-41E6-A6D9-C549E854965D@mcs.anl.gov> <20111208151135.09i8np2rpc444g4k@mail.zih.tu-dresden.de> Message-ID: <1697A0FF-14C5-431A-B718-B3055655BF74@mcs.anl.gov> These lines make it clear the problem is detected on a sequential matrix on process 15. This is why we like the full message. Barry [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1636 in src/mat/impls/aij/seq/aijfact.c [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1740 in src/mat/impls/aij/seq/aijfact.c [15]PETSC ERROR: MatILUFactorSymbolic() line 6092 in src/mat/interface/matrix.c On Dec 8, 2011, at 8:11 AM, Thomas Witkowski wrote: > Okay, if the entry number is the local one, than I can check this again. In this case, I think that I know where the problem is: By the way, the error message is as follows: > > [15]PETSC ERROR: Object is in wrong state! > [15]PETSC ERROR: Matrix is missing diagonal entry 1129! > [15]PETSC ERROR: ------------------------------------------------------------------------ > [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 > [15]PETSC ERROR: See docs/changes/index.html for recent updates. > [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [15]PETSC ERROR: See docs/index.html for manual pages. > [15]PETSC ERROR: ------------------------------------------------------------------------ > [15]PETSC ERROR: ./ch2 on a arch-linu named p1q024 by witkowsk Thu Dec 8 13:39:46 2011 > [15]PETSC ERROR: Libraries linked from /fastfs/witkowsk/petsc/install/petsc-3.2-p5-dbg/lib > [15]PETSC ERROR: Configure run at Fri Dec 2 14:45:50 2011 > [15]PETSC ERROR: Configure options --prefix=/fastfs/witkowsk/petsc/install/petsc-3.2-p5-dbg --download-blacs --download-parmetis --download-scalapack --download-umfpac\ > k --with-blas-lapack-dir=/work/licsoft/libraries/mkl/9.1.023/lib/em64t --with-c++-support --download-mumps --download-superlu --with-clanguage=cxx > [15]PETSC ERROR: ------------------------------------------------------------------------ > [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1636 in src/mat/impls/aij/seq/aijfact.c > [15]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1740 in src/mat/impls/aij/seq/aijfact.c > [15]PETSC ERROR: MatILUFactorSymbolic() line 6092 in src/mat/interface/matrix.c > [15]PETSC ERROR: PCSetUp_ILU() line 216 in src/ksp/pc/impls/factor/ilu/ilu.c > [15]PETSC ERROR: PCSetUp() line 819 in src/ksp/pc/interface/precon.c > [15]PETSC ERROR: KSPSetUp() line 260 in src/ksp/ksp/interface/itfunc.c > [15]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Multiblock() line 973 in src/ksp/pc/impls/bjacobi/bjacobi.c > [15]PETSC ERROR: PCSetUpOnBlocks() line 852 in src/ksp/pc/interface/precon.c > [15]PETSC ERROR: KSPSetUpOnBlocks() line 154 in src/ksp/ksp/interface/itfunc.c > > > Zitat von Barry Smith : > >> >> Thomas, >> >> As always including the ENTIRE error message makes determining the issue much easier. It tells us what functions are being called at this point and the line numbers. Without we would only be speculating. >> >> In this case the error message is printed on process of rank 15 at location 1129 ON THAT PROCESS so the you need to determine the global row of location 1129 on the 15th process. That row is missing the diagonal. >> >> Barry >> >> The reason I know it is the not the global row 1129 but the 1129th row on process 15 is because the I know the check for diagonal entries is done in the factorization that is taking place on each process (each factorization is running at the same time on different processes). This would be clear with the entire error message. >> >> >> On Dec 8, 2011, at 7:49 AM, Thomas Witkowski wrote: >> >>> I've some strange problem: I create a mpiaij matrix, fill it with some values and want to make use of it in kspsolve. But I get the error >>> >>> [15]PETSC ERROR: Object is in wrong state! >>> [15]PETSC ERROR: Matrix is missing diagonal entry 1129! >>> >>> I've checked the following things: >>> - There is a MatSetValues call that contains the entry 1129/1129 with some value >>> - MatView shows that there exists the value 1129/1129 >>> - MatAssemblyBegin/End is called correctly >>> - valgrind says: 0 errors >>> - ddt also cannot find anything unusual >>> >>> I use petsc 3.2-p5, its compiled in debug mode. What could I check to solve this problem? Any hints? >>> >>> Thomas >> >> > > From keita at cray.com Thu Dec 8 12:02:32 2011 From: keita at cray.com (Keita Teranishi) Date: Thu, 8 Dec 2011 12:02:32 -0600 Subject: [petsc-users] petsc 3.2 compiling on cray In-Reply-To: References: Message-ID: <5D6E0DF460ACF34C88644E1EA91DCD0D12E2B2A8BC@CFEXMBX.americas.cray.com> Hi, Although it is dependent on the programming environment you have, I suggest putting "-lsci_(compiler name)_mp" instead of putting ACML. If you are using gnu, it is -lsci_gnu_mp. Thanks, Keita From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Thursday, December 08, 2011 8:06 AM To: PETSc users list Subject: Re: [petsc-users] petsc 3.2 compiling on cray On Thu, Dec 8, 2011 at 1:11 AM, Xiangdong Liang > wrote: Hello everyone, I am trying to compile petsc-3.2 on cray. However, when I specify blas and lapack library --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, libacml_mv.a], petsc complains like this: =============================================================================== Configuring PETSc to compile on your system =============================================================================== ******************************************************************************* ERROR in COMMAND LINE ARGUMENT to ./configure ------------------------------------------------------------------------------- The option libacml_mv.a] should probably be libacml-mv.a] ******************************************************************************* If you specify the full path to the library, this should go away. This is just an error check to prevent people from mistyping options. Thanks, Matt However, the library is indeed named as libacml_mv.a. I can compile successfully without specifying the lapack and blas libraries. Will it affect the performance? Thanks Xiangdong The configure option I use is: ./config/configure.py --with-blas-lapack-lib=[/opt/acml/4.4.0/pgi64/lib/libacml.a, libacml_mv.a] \ --with-cc=cc --with-cxx=CC --with-fc=ftn -with-shared-lib\ raries=0 --COPTFLAGS='-fastsse -O3 -tp istanbul-64' --CXXOPTFLAGS='-fastsse -O3\ -tp istanbul-64' --FOPTFLAGS='-fastsse -O3 -tp istanbul-64' --with-debugging=0\ --prefix=$MYINSTALL --PETSC_ARCH=arch-opt --with-clib-autodetect=0 --\ with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --download-blacs=yes --\ download-scalapack=yes --download-parmetis=yes --download-superlu=yes --downloa\ d-superlu_dist=yes --download-scotch=yes --download-pastix=yes --download-mumps\ =yes --download-spooles=yes --with-hdf5=yes --with-hdf5-dir=/opt/cray/hdf5/1.8.\ 5.0/hdf5-gnu/ --download-ptscotch --download-umfpack=yes --download-hypre=yes -\ -download-spai=yes -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiaohl1986 at gmail.com Thu Dec 8 12:12:14 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Thu, 8 Dec 2011 12:12:14 -0600 Subject: [petsc-users] linear problem solved by snes in 2 iteration. Message-ID: Hi I am using cell centered finite difference method for -div K (grad p) = f with Neumann boundary condtion. I called SNESGetKSP(snes, &ksp); MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &nullsp); KSPSetNullSpace(ksp, nullsp); MatNullSpaceDestroy(&nullsp); to handle the null space. But I got the result like the following 0 SNES Function norm 3.512407365520e+02 Linear solve converged due to CONVERGED_RTOL iterations 16 1 SNES Function norm 6.615335308038e-03 Linear solve converged due to CONVERGED_RTOL iterations 20 2 SNES Function norm 3.979662741066e-08 I checked my numerical solution with analytical solution (p=x,y,z,cos(pi*x),cos(pi*y), cos(pi*z). These are the difference between the petsc result and analytical solution for -da_grid_x 5 -da_grid_y 5 -da_grid_z 100 i j k 1 1 0 petsc 0.499959 true 0.495 petsc-true 0.00495888 i j k 1 1 1 petsc 0.486626 true 0.485 petsc-true 0.00162554 i j k 1 1 2 petsc 0.475959 true 0.475 petsc-true 0.000958875 i j k 1 1 3 petsc 0.465673 true 0.465 petsc-true 0.000673159 I think I got the correct solution. I do not understand why snes took 2 iterations for a linear problem? Is it normal? -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 8 12:33:50 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Dec 2011 10:33:50 -0800 Subject: [petsc-users] linear problem solved by snes in 2 iteration. In-Reply-To: References: Message-ID: It is because of the dafault convergence tolerances. Try -ksp_rtol 1e-10 On Dec 8, 2011 10:12 AM, "Hailong Xiao" wrote: > Hi > > I am using cell centered finite difference method for > -div K (grad p) = f > with Neumann boundary condtion. > > I called > SNESGetKSP(snes, &ksp); > MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &nullsp); > KSPSetNullSpace(ksp, nullsp); > MatNullSpaceDestroy(&nullsp); > > to handle the null space. > > But I got the result like the following > 0 SNES Function norm 3.512407365520e+02 > Linear solve converged due to CONVERGED_RTOL iterations 16 > 1 SNES Function norm 6.615335308038e-03 > Linear solve converged due to CONVERGED_RTOL iterations 20 > 2 SNES Function norm 3.979662741066e-08 > > I checked my numerical solution with analytical solution > (p=x,y,z,cos(pi*x),cos(pi*y), cos(pi*z). > > These are the difference between the petsc result and analytical solution > for > -da_grid_x 5 > -da_grid_y 5 > -da_grid_z 100 > > i j k 1 1 0 > petsc 0.499959 > true 0.495 > petsc-true 0.00495888 > i j k 1 1 1 > petsc 0.486626 > true 0.485 > petsc-true 0.00162554 > i j k 1 1 2 > petsc 0.475959 > true 0.475 > petsc-true 0.000958875 > i j k 1 1 3 > petsc 0.465673 > true 0.465 > petsc-true 0.000673159 > > I think I got the correct solution. > > I do not understand why snes took 2 iterations for a linear problem? Is it > normal? > > > -- > Hailong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiaohl1986 at gmail.com Thu Dec 8 12:45:09 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Thu, 8 Dec 2011 12:45:09 -0600 Subject: [petsc-users] linear problem solved by snes in 2 iteration. In-Reply-To: References: Message-ID: I see. now it's one iteration. Thank you Jed. Hailong On Thu, Dec 8, 2011 at 12:33 PM, Jed Brown wrote: > It is because of the dafault convergence tolerances. > > Try -ksp_rtol 1e-10 > On Dec 8, 2011 10:12 AM, "Hailong Xiao" wrote: > >> Hi >> >> I am using cell centered finite difference method for >> -div K (grad p) = f >> with Neumann boundary condtion. >> >> I called >> SNESGetKSP(snes, &ksp); >> MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL, &nullsp); >> KSPSetNullSpace(ksp, nullsp); >> MatNullSpaceDestroy(&nullsp); >> >> to handle the null space. >> >> But I got the result like the following >> 0 SNES Function norm 3.512407365520e+02 >> Linear solve converged due to CONVERGED_RTOL iterations 16 >> 1 SNES Function norm 6.615335308038e-03 >> Linear solve converged due to CONVERGED_RTOL iterations 20 >> 2 SNES Function norm 3.979662741066e-08 >> >> I checked my numerical solution with analytical solution >> (p=x,y,z,cos(pi*x),cos(pi*y), cos(pi*z). >> >> These are the difference between the petsc result and analytical solution >> for >> -da_grid_x 5 >> -da_grid_y 5 >> -da_grid_z 100 >> >> i j k 1 1 0 >> petsc 0.499959 >> true 0.495 >> petsc-true 0.00495888 >> i j k 1 1 1 >> petsc 0.486626 >> true 0.485 >> petsc-true 0.00162554 >> i j k 1 1 2 >> petsc 0.475959 >> true 0.475 >> petsc-true 0.000958875 >> i j k 1 1 3 >> petsc 0.465673 >> true 0.465 >> petsc-true 0.000673159 >> >> I think I got the correct solution. >> >> I do not understand why snes took 2 iterations for a linear problem? Is >> it normal? >> >> >> -- >> Hailong >> > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Dec 9 04:51:42 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 09 Dec 2011 11:51:42 +0100 Subject: [petsc-users] Metis building Message-ID: <4EE1E83E.2050600@gfz-potsdam.de> Hello, It's pretty weird situation now. A couple of weeks ago I configured petsc-3.2-p5 successfully, today I decided to do it with new options and it didn't work, I tried to do it with old options just to check and got error: lib/petsc-3.2-p5> ./configure --with-petsc-arch=intel-complex-debug-f-ds --with-fortran-interfaces=1 --download-superlu_dist --download-mumps --download-parmetis --download-metis --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== ***** WARNING: CC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: CXX found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: FC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: F77 found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: F90 found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== Compiling & installing Metis; this may take several minutes =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make on Metis: Could not execute "cd /home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": (cd Lib ; make clean ) make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Lib' rm -f *.o make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Lib' (cd Programs ; make clean ) make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Programs' rm -f *.o make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Programs' (cd Test ; make clean ) make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Test' rm -f *.o make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Test' make: *** No rule to make target `library'. Stop. ******************************************************************************* I looked into /home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/makefile: default: (cd Lib ; make ) (cd Programs ; make ) (cd Test ; make ) clean: (cd Lib ; make clean ) (cd Programs ; make clean ) (cd Test ; make clean ) realclean: (cd Lib ; make realclean ) (cd Programs ; make realclean ) (cd Test ; make realclean ) So there is no library. I have no idea what went wrong. Any suggestions would be appreciated. Thanks in advance. Regards, Alexander From bsmith at mcs.anl.gov Fri Dec 9 07:35:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 9 Dec 2011 07:35:27 -0600 Subject: [petsc-users] Metis building In-Reply-To: <4EE1E83E.2050600@gfz-potsdam.de> References: <4EE1E83E.2050600@gfz-potsdam.de> Message-ID: <70CD6723-A7B3-4374-AC14-686AA04D90D9@mcs.anl.gov> PETSc users, Please do NOT send configure/make issues to petsc-users, send them to petsc-maint at mcs.anl.gov and ALWAYS send configure.log and make.log with the message. Barry On Dec 9, 2011, at 4:51 AM, Alexander Grayver wrote: > Hello, > > It's pretty weird situation now. A couple of weeks ago I configured petsc-3.2-p5 successfully, today I decided to do it with new options and it didn't work, I tried to do it with old options just to check and got error: > > lib/petsc-3.2-p5> ./configure --with-petsc-arch=intel-complex-debug-f-ds --with-fortran-interfaces=1 --download-superlu_dist --download-mumps --download-parmetis --download-metis --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > =============================================================================== ***** WARNING: CC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: CXX found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: FC found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: F77 found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== ***** WARNING: F90 found in enviornment variables - ignoring ****** =============================================================================== =============================================================================== Compiling & installing Metis; this may take several minutes =============================================================================== ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error running make on Metis: Could not execute "cd /home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": > (cd Lib ; make clean ) > make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Lib' > rm -f *.o > make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Lib' > (cd Programs ; make clean ) > make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Programs' > rm -f *.o > make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Programs' > (cd Test ; make clean ) > make[1]: Entering directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Test' > rm -f *.o > make[1]: Leaving directory `/home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/Test' > make: *** No rule to make target `library'. Stop. > ******************************************************************************* > > I looked into /home/lib/petsc-3.2-p5/externalpackages/metis-4.0.3/makefile: > > > default: > (cd Lib ; make ) > (cd Programs ; make ) > (cd Test ; make ) > > clean: > (cd Lib ; make clean ) > (cd Programs ; make clean ) > (cd Test ; make clean ) > > realclean: > (cd Lib ; make realclean ) > (cd Programs ; make realclean ) > (cd Test ; make realclean ) > > So there is no library. > I have no idea what went wrong. Any suggestions would be appreciated. > > Thanks in advance. > > Regards, > Alexander From agrayver at gfz-potsdam.de Fri Dec 9 09:44:41 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 09 Dec 2011 16:44:41 +0100 Subject: [petsc-users] ILU in parallel Message-ID: <4EE22CE9.9070201@gfz-potsdam.de> Hello, Sorry for dummy question, but what are options for parallel ILU in petsc? Which packages are best for big sparse matrices (>= 10^6)? Where do I start from? Regards, Alexander From knepley at gmail.com Fri Dec 9 09:58:51 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 9 Dec 2011 09:58:51 -0600 Subject: [petsc-users] ILU in parallel In-Reply-To: <4EE22CE9.9070201@gfz-potsdam.de> References: <4EE22CE9.9070201@gfz-potsdam.de> Message-ID: On Fri, Dec 9, 2011 at 9:44 AM, Alexander Grayver wrote: > Hello, > > Sorry for dummy question, but what are options for parallel ILU in petsc? > Which packages are best for big sparse matrices (>= 10^6)? > Where do I start from? > http://www.mcs.anl.gov/petsc/petsc-dev/docs/linearsolvertable.html I generally recommend against parallel ILU, but Hypre has it. Matt Regards, > Alexander > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Dec 9 10:06:50 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 09 Dec 2011 17:06:50 +0100 Subject: [petsc-users] ILU in parallel In-Reply-To: References: <4EE22CE9.9070201@gfz-potsdam.de> Message-ID: <4EE2321A.1080600@gfz-potsdam.de> On 09.12.2011 16:58, Matthew Knepley wrote: > On Fri, Dec 9, 2011 at 9:44 AM, Alexander Grayver > > wrote: > > Hello, > > Sorry for dummy question, but what are options for parallel ILU in > petsc? Which packages are best for big sparse matrices (>= 10^6)? > Where do I start from? > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/linearsolvertable.html > > I generally recommend against parallel ILU, but Hypre has it. Why? > > Matt > > Regards, > Alexander > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 9 10:11:09 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 9 Dec 2011 08:11:09 -0800 Subject: [petsc-users] ILU in parallel In-Reply-To: <4EE2321A.1080600@gfz-potsdam.de> References: <4EE22CE9.9070201@gfz-potsdam.de> <4EE2321A.1080600@gfz-potsdam.de> Message-ID: On Fri, Dec 9, 2011 at 08:06, Alexander Grayver wrote: > Why? Usually poor parallel scalability and unreliable convergence properties. But sometimes it works well enough. -pc_type hypre -pc_hypre_type euclid OR -pc_type hypre -pc_hypre_type pilut -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 9 10:16:09 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 9 Dec 2011 10:16:09 -0600 Subject: [petsc-users] ILU in parallel In-Reply-To: <4EE2321A.1080600@gfz-potsdam.de> References: <4EE22CE9.9070201@gfz-potsdam.de> <4EE2321A.1080600@gfz-potsdam.de> Message-ID: On Fri, Dec 9, 2011 at 10:06 AM, Alexander Grayver wrote: > ** > On 09.12.2011 16:58, Matthew Knepley wrote: > > On Fri, Dec 9, 2011 at 9:44 AM, Alexander Grayver > wrote: > >> Hello, >> >> Sorry for dummy question, but what are options for parallel ILU in petsc? >> Which packages are best for big sparse matrices (>= 10^6)? >> Where do I start from? >> > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/linearsolvertable.html > > I generally recommend against parallel ILU, but Hypre has it. > > Why? > Its fundamentally not scalable, has unpredictable performance, and in my experience worse than sparse direct. Matt > Matt > > Regards, >> Alexander >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Dec 9 10:17:00 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 09 Dec 2011 17:17:00 +0100 Subject: [petsc-users] ILU in parallel In-Reply-To: References: <4EE22CE9.9070201@gfz-potsdam.de> <4EE2321A.1080600@gfz-potsdam.de> Message-ID: <4EE2347C.2010707@gfz-potsdam.de> Jed, Matt, Ok, I see. Perhaps it was bad idea to play around with ILU for my problems... Regards, Alexander On 09.12.2011 17:16, Matthew Knepley wrote: > On Fri, Dec 9, 2011 at 10:06 AM, Alexander Grayver > > wrote: > > On 09.12.2011 16:58, Matthew Knepley wrote: >> On Fri, Dec 9, 2011 at 9:44 AM, Alexander Grayver >> > wrote: >> >> Hello, >> >> Sorry for dummy question, but what are options for parallel >> ILU in petsc? Which packages are best for big sparse matrices >> (>= 10^6)? >> Where do I start from? >> >> >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/linearsolvertable.html >> >> I generally recommend against parallel ILU, but Hypre has it. > Why? > > > Its fundamentally not scalable, has unpredictable performance, and in > my experience worse than sparse direct. > > Matt > >> Matt >> >> Regards, >> Alexander >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 9 10:21:08 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 9 Dec 2011 08:21:08 -0800 Subject: [petsc-users] ILU in parallel In-Reply-To: <4EE2347C.2010707@gfz-potsdam.de> References: <4EE22CE9.9070201@gfz-potsdam.de> <4EE2321A.1080600@gfz-potsdam.de> <4EE2347C.2010707@gfz-potsdam.de> Message-ID: On Fri, Dec 9, 2011 at 08:17, Alexander Grayver wrote: > Ok, I see. Perhaps it was bad idea to play around with ILU for my > problems... It only takes a moment to try those options with Hypre, we just don't have much confidence that it will end up being a good way to solve the problem -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Fri Dec 9 17:27:41 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 18:27:41 -0500 Subject: [petsc-users] colors in mat_view_draw Message-ID: Hello everyone, When I use mat_view_draw for a matrix, I see different colors ( red , blue, and etc) in the drawing. What does these colors mean? Thanks. Best, Xiangdong From knepley at gmail.com Fri Dec 9 17:42:32 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 9 Dec 2011 17:42:32 -0600 Subject: [petsc-users] colors in mat_view_draw In-Reply-To: References: Message-ID: Positive, negative, and zero. Matt On Fri, Dec 9, 2011 at 5:27 PM, Xiangdong Liang wrote: > Hello everyone, > > When I use mat_view_draw for a matrix, I see different colors ( red , > blue, and etc) in the drawing. What does these colors mean? Thanks. > > Best, > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Fri Dec 9 17:55:28 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 18:55:28 -0500 Subject: [petsc-users] sparse direct solver and number of nonzero elements Message-ID: Hello everyone, I am solving Ax=b with sparse direct solver Pastix. I have two equivalent A's (upto these zero entries): A1 and A2. A1 is generated with ignor_zero_entries and A2 is without this option. For example A1 has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve them with Pastix, I found the time for solving sparser A1 actually is longer (10%--20% worse) than A2. Does anyone have thoughts on this? Thanks. Xiangdong P.S. The time I count is only for Spare LU solving (not including the matrix assembling time). From bsmith at mcs.anl.gov Fri Dec 9 18:02:56 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 9 Dec 2011 18:02:56 -0600 Subject: [petsc-users] sparse direct solver and number of nonzero elements In-Reply-To: References: Message-ID: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> On Dec 9, 2011, at 5:55 PM, Xiangdong Liang wrote: > Hello everyone, > > I am solving Ax=b with sparse direct solver Pastix. I have two > equivalent A's (upto these zero entries): A1 and A2. A1 is generated > with ignor_zero_entries and A2 is without this option. For example A1 > has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve > them with Pastix, I found the time for solving sparser A1 actually is > longer (10%--20% worse) than A2. Does anyone have thoughts on this? Completely possible and not particularly surprising. The amount of work required for sparse LU depends in an incrediably complicated way on the nonzero structure of the matrix, it is only very very minorly related to the number of nonzeros in the matrix. One fun thing to check would be the number of nonzeros in the factor of A1 and the number of nonzeros in the factor of A2. (I'm not sure if Pastix has a way to check this). Barry > Thanks. > > Xiangdong > > P.S. The time I count is only for Spare LU solving (not including the > matrix assembling time). From xdliang at gmail.com Fri Dec 9 18:13:52 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 19:13:52 -0500 Subject: [petsc-users] sparse direct solver and number of nonzero elements In-Reply-To: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> References: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> Message-ID: On Fri, Dec 9, 2011 at 7:02 PM, Barry Smith wrote: > > On Dec 9, 2011, at 5:55 PM, Xiangdong Liang wrote: > >> Hello everyone, >> >> I am solving Ax=b with sparse direct solver Pastix. I have two >> equivalent A's (upto these zero entries): A1 and A2. A1 is generated >> with ignor_zero_entries and A2 is without this option. For example A1 >> has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve >> them with Pastix, I found the time for solving sparser A1 actually is >> longer (10%--20% worse) than A2. Does anyone have thoughts on this? > > ? Completely possible and not particularly surprising. The amount of work required for sparse LU depends in an incrediably complicated way on the nonzero structure of the matrix, it is only very very minorly related to the number of nonzeros in the matrix. > ? One fun thing to check would be the number of nonzeros in the factor of A1 and the number of nonzeros in the factor of A2. (I'm not sure if Pastix has a way to check this). Should the nnz in the factor of A1 be the same as nnz in the factor of A2 since A1 and A2 are the same matrices except some zeros (due to ignore_zero_entries in the matrix assembling)? Xiangdong > > ? Barry > >> Thanks. >> >> Xiangdong >> >> P.S. The time I count is only for Spare LU solving (not including the >> matrix assembling time). > From xdliang at gmail.com Fri Dec 9 18:15:43 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 19:15:43 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: If I use MAT_IGNORE_ZERO_ENTRIES option, is it possible that I still see some zeros in the sparse matrix? In my problem, when I use this ignore_zero options, the number of nonzeros is significantly reduced (from mat_view_info) but still larger than the number of nonzeros counting from matlab. When I use mat_view_draw, I see some light green lines (corresponding to zeros). I am wondering where these zeros come from? Thanks Xiangdong On Wed, Dec 7, 2011 at 12:15 PM, Satish Balay wrote: > On Wed, 7 Dec 2011, Satish Balay wrote: > >> As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) >> - before any calls to MatSetValue/s() >> >> And verify with MatView() > > Ok - here is what you can see with a slightly modified mat/examples/tests/ex38.c > > Notice "C[0,1] = 0.0" and thats not visible anymore with > MatSetOption(MAT_IGNORE_ZERO_ENTRIES) > > Satish > > -------- > asterix:/home/balay/spetsc/src/mat/examples/tests>cat ex38.c > static char help[] = "Tests MatSetValues().\n\n"; > #include > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int argc,char **args) > { > ?Mat ? ? ? ? ? ?C; > ?PetscInt ? ? ? i,n = 5,midx[3],nidx[2]; > ?PetscErrorCode ierr; > ?PetscBool ? ? ?flg; > ?PetscScalar ? ?v[6]; > > ?PetscInitialize(&argc,&args,(char *)0,help); > ?ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > ?ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); > ?ierr = MatSetFromOptions(C);CHKERRQ(ierr); > > ?ierr = PetscOptionsHasName(PETSC_NULL,"-ignore_zero_entries",&flg);CHKERRQ(ierr); > ?if (flg) { ?ierr = MatSetOption(C,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE);CHKERRQ(ierr);} > > ?for (i=0; i<6; i++) v[i] = (PetscReal)i; > ?midx[0] = 0; midx[1] = 2; midx[2] = 3; > ?nidx[0] = 1; nidx[1] = 3; > ?ierr = MatSetValues(C,3,midx,2,nidx,v,ADD_VALUES);CHKERRQ(ierr); > ?ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ?ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ?ierr = MatView(C,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > ?ierr = MatDestroy(&C);CHKERRQ(ierr); > ?ierr = PetscFinalize(); > ?return 0; > } > > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info > Matrix Object: 1 MPI processes > ?type: seqaij > ?rows=5, cols=5 > ?total: nonzeros=6, allocated nonzeros=25 > ?total number of mallocs used during MatSetValues calls =0 > ? ?using I-node routines: found 4 nodes, limit used is 5 > Matrix Object: 1 MPI processes > ?type: seqaij > row 0: (1, 0) ?(3, 1) > row 1: > row 2: (1, 2) ?(3, 3) > row 3: (1, 4) ?(3, 5) > row 4: > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info -ignore_zero_entries > Matrix Object: 1 MPI processes > ?type: seqaij > ?rows=5, cols=5 > ?total: nonzeros=5, allocated nonzeros=25 > ?total number of mallocs used during MatSetValues calls =0 > ? ?using I-node routines: found 4 nodes, limit used is 5 > Matrix Object: 1 MPI processes > ?type: seqaij > row 0: (3, 1) > row 1: > row 2: (1, 2) ?(3, 3) > row 3: (1, 4) ?(3, 5) > row 4: > asterix:/home/balay/spetsc/src/mat/examples/tests> From knepley at gmail.com Fri Dec 9 18:23:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 9 Dec 2011 18:23:52 -0600 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Fri, Dec 9, 2011 at 6:15 PM, Xiangdong Liang wrote: > If I use MAT_IGNORE_ZERO_ENTRIES option, is it possible that I still > see some zeros in the sparse matrix? In my problem, when I use this > Not if they are truly zero. > ignore_zero options, the number of nonzeros is significantly reduced > (from mat_view_info) but still larger than the number of nonzeros > counting from matlab. When I use mat_view_draw, I see some light green > lines (corresponding to zeros). I am wondering where these zeros come > from? Thanks > Print it and see where they are. Matt > Xiangdong > > On Wed, Dec 7, 2011 at 12:15 PM, Satish Balay wrote: > > On Wed, 7 Dec 2011, Satish Balay wrote: > > > >> As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) > >> - before any calls to MatSetValue/s() > >> > >> And verify with MatView() > > > > Ok - here is what you can see with a slightly modified > mat/examples/tests/ex38.c > > > > Notice "C[0,1] = 0.0" and thats not visible anymore with > > MatSetOption(MAT_IGNORE_ZERO_ENTRIES) > > > > Satish > > > > -------- > > asterix:/home/balay/spetsc/src/mat/examples/tests>cat ex38.c > > static char help[] = "Tests MatSetValues().\n\n"; > > #include > > #undef __FUNCT__ > > #define __FUNCT__ "main" > > int main(int argc,char **args) > > { > > Mat C; > > PetscInt i,n = 5,midx[3],nidx[2]; > > PetscErrorCode ierr; > > PetscBool flg; > > PetscScalar v[6]; > > > > PetscInitialize(&argc,&args,(char *)0,help); > > ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > > ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); > > ierr = MatSetFromOptions(C);CHKERRQ(ierr); > > > > ierr = > PetscOptionsHasName(PETSC_NULL,"-ignore_zero_entries",&flg);CHKERRQ(ierr); > > if (flg) { ierr = > MatSetOption(C,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE);CHKERRQ(ierr);} > > > > for (i=0; i<6; i++) v[i] = (PetscReal)i; > > midx[0] = 0; midx[1] = 2; midx[2] = 3; > > nidx[0] = 1; nidx[1] = 3; > > ierr = MatSetValues(C,3,midx,2,nidx,v,ADD_VALUES);CHKERRQ(ierr); > > ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = MatView(C,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > > ierr = MatDestroy(&C);CHKERRQ(ierr); > > ierr = PetscFinalize(); > > return 0; > > } > > > > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info > > Matrix Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > total: nonzeros=6, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 4 nodes, limit used is 5 > > Matrix Object: 1 MPI processes > > type: seqaij > > row 0: (1, 0) (3, 1) > > row 1: > > row 2: (1, 2) (3, 3) > > row 3: (1, 4) (3, 5) > > row 4: > > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info > -ignore_zero_entries > > Matrix Object: 1 MPI processes > > type: seqaij > > rows=5, cols=5 > > total: nonzeros=5, allocated nonzeros=25 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 4 nodes, limit used is 5 > > Matrix Object: 1 MPI processes > > type: seqaij > > row 0: (3, 1) > > row 1: > > row 2: (1, 2) (3, 3) > > row 3: (1, 4) (3, 5) > > row 4: > > asterix:/home/balay/spetsc/src/mat/examples/tests> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 9 18:24:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 9 Dec 2011 18:24:47 -0600 Subject: [petsc-users] sparse direct solver and number of nonzero elements In-Reply-To: References: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> Message-ID: On Fri, Dec 9, 2011 at 6:13 PM, Xiangdong Liang wrote: > On Fri, Dec 9, 2011 at 7:02 PM, Barry Smith wrote: > > > > On Dec 9, 2011, at 5:55 PM, Xiangdong Liang wrote: > > > >> Hello everyone, > >> > >> I am solving Ax=b with sparse direct solver Pastix. I have two > >> equivalent A's (upto these zero entries): A1 and A2. A1 is generated > >> with ignor_zero_entries and A2 is without this option. For example A1 > >> has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve > >> them with Pastix, I found the time for solving sparser A1 actually is > >> longer (10%--20% worse) than A2. Does anyone have thoughts on this? > > > > Completely possible and not particularly surprising. The amount of > work required for sparse LU depends in an incrediably complicated way on > the nonzero structure of the matrix, it is only very very minorly related > to the number of nonzeros in the matrix. > > One fun thing to check would be the number of nonzeros in the factor > of A1 and the number of nonzeros in the factor of A2. (I'm not sure if > Pastix has a way to check this). > > Should the nnz in the factor of A1 be the same as nnz in the factor of > A2 since A1 and A2 are the same matrices except some zeros (due to > ignore_zero_entries in the matrix assembling)? > nnz is structural, and has nothing to do with numeric values. Matt > Xiangdong > > > > > Barry > > > >> Thanks. > >> > >> Xiangdong > >> > >> P.S. The time I count is only for Spare LU solving (not including the > >> matrix assembling time). > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 9 18:30:41 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 9 Dec 2011 16:30:41 -0800 Subject: [petsc-users] sparse direct solver and number of nonzero elements In-Reply-To: References: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> Message-ID: No, a different ordering will be computed, and the ordering with the sparser matrix could even lead to more fill. On Dec 9, 2011 4:13 PM, "Xiangdong Liang" wrote: > On Fri, Dec 9, 2011 at 7:02 PM, Barry Smith wrote: > > > > On Dec 9, 2011, at 5:55 PM, Xiangdong Liang wrote: > > > >> Hello everyone, > >> > >> I am solving Ax=b with sparse direct solver Pastix. I have two > >> equivalent A's (upto these zero entries): A1 and A2. A1 is generated > >> with ignor_zero_entries and A2 is without this option. For example A1 > >> has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve > >> them with Pastix, I found the time for solving sparser A1 actually is > >> longer (10%--20% worse) than A2. Does anyone have thoughts on this? > > > > Completely possible and not particularly surprising. The amount of > work required for sparse LU depends in an incrediably complicated way on > the nonzero structure of the matrix, it is only very very minorly related > to the number of nonzeros in the matrix. > > One fun thing to check would be the number of nonzeros in the factor > of A1 and the number of nonzeros in the factor of A2. (I'm not sure if > Pastix has a way to check this). > > Should the nnz in the factor of A1 be the same as nnz in the factor of > A2 since A1 and A2 are the same matrices except some zeros (due to > ignore_zero_entries in the matrix assembling)? > > Xiangdong > > > > > Barry > > > >> Thanks. > >> > >> Xiangdong > >> > >> P.S. The time I count is only for Spare LU solving (not including the > >> matrix assembling time). > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Fri Dec 9 19:06:38 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 20:06:38 -0500 Subject: [petsc-users] sparse direct solver and number of nonzero elements In-Reply-To: References: <90ED0078-E464-4D54-BF94-EB6DB7C3C819@mcs.anl.gov> Message-ID: I use the option ?-mat_pastix_verbose 1 to print the Pastix factorization information. You are right. The sparser one actually has more Number of nonzeros (with fill-in) and Number of operations (LU), thus takes longer to factorize. Thanks. Xiangdong On Fri, Dec 9, 2011 at 7:30 PM, Jed Brown wrote: > No, a different ordering will be computed, and the ordering with the sparser > matrix could even lead to more fill. > > On Dec 9, 2011 4:13 PM, "Xiangdong Liang" wrote: >> >> On Fri, Dec 9, 2011 at 7:02 PM, Barry Smith wrote: >> > >> > On Dec 9, 2011, at 5:55 PM, Xiangdong Liang wrote: >> > >> >> Hello everyone, >> >> >> >> I am solving Ax=b with sparse direct solver Pastix. I have two >> >> equivalent A's (upto these zero entries): A1 and A2. A1 is generated >> >> with ignor_zero_entries and A2 is without this option. For example A1 >> >> has 9 millions nonzeros, while A2 has 10 millions zeros. When I solve >> >> them with Pastix, I found the time for solving sparser A1 actually is >> >> longer (10%--20% worse) than A2. Does anyone have thoughts on this? >> > >> > ? Completely possible and not particularly surprising. The amount of >> > work required for sparse LU depends in an incrediably complicated way on the >> > nonzero structure of the matrix, it is only very very minorly related to the >> > number of nonzeros in the matrix. >> > ? One fun thing to check would be the number of nonzeros in the factor >> > of A1 and the number of nonzeros in the factor of A2. (I'm not sure if >> > Pastix has a way to check this). >> >> Should the nnz in the factor of A1 be the same as nnz in the factor of >> A2 since A1 and A2 are the same matrices except some zeros (due to >> ignore_zero_entries in the matrix assembling)? >> >> Xiangdong >> >> > >> > ? Barry >> > >> >> Thanks. >> >> >> >> Xiangdong >> >> >> >> P.S. The time I count is only for Spare LU solving (not including the >> >> matrix assembling time). >> > From xdliang at gmail.com Fri Dec 9 19:32:06 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 9 Dec 2011 20:32:06 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Fri, Dec 9, 2011 at 7:23 PM, Matthew Knepley wrote: > On Fri, Dec 9, 2011 at 6:15 PM, Xiangdong Liang wrote: >> >> If I use MAT_IGNORE_ZERO_ENTRIES option, is it possible that I still >> see some zeros in the sparse matrix? In my problem, when I use this > > > Not if they are truly zero. > >> >> ignore_zero options, the number of nonzeros is significantly reduced >> (from mat_view_info) but still larger than the number of nonzeros >> counting from matlab. When I use mat_view_draw, I see some light green >> lines (corresponding to zeros). I am wondering where these zeros come >> from? Thanks I figured out where these extra zeros comes from. At these positions, I first addvalues 1 then addvalues -1. Therefore, even thought the value at this position is zero, it cannot be ignored by the ignore_zero_options. Another quick question, when I use mat_view_draw command line option, how can I make the figure X window larger? I think petsc PetscDrawOpenX can do this job, but just wondering whether any quick command line options? Thanks. Xiangdong > > > Print it and see where they are. > > ? Matt > >> >> Xiangdong >> >> On Wed, Dec 7, 2011 at 12:15 PM, Satish Balay wrote: >> > On Wed, 7 Dec 2011, Satish Balay wrote: >> > >> >> As indicated call "MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE) >> >> - before any calls to MatSetValue/s() >> >> >> >> And verify with MatView() >> > >> > Ok - here is what you can see with a slightly modified >> > mat/examples/tests/ex38.c >> > >> > Notice "C[0,1] = 0.0" and thats not visible anymore with >> > MatSetOption(MAT_IGNORE_ZERO_ENTRIES) >> > >> > Satish >> > >> > -------- >> > asterix:/home/balay/spetsc/src/mat/examples/tests>cat ex38.c >> > static char help[] = "Tests MatSetValues().\n\n"; >> > #include >> > #undef __FUNCT__ >> > #define __FUNCT__ "main" >> > int main(int argc,char **args) >> > { >> > ?Mat ? ? ? ? ? ?C; >> > ?PetscInt ? ? ? i,n = 5,midx[3],nidx[2]; >> > ?PetscErrorCode ierr; >> > ?PetscBool ? ? ?flg; >> > ?PetscScalar ? ?v[6]; >> > >> > ?PetscInitialize(&argc,&args,(char *)0,help); >> > ?ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); >> > ?ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); >> > ?ierr = MatSetFromOptions(C);CHKERRQ(ierr); >> > >> > ?ierr = >> > PetscOptionsHasName(PETSC_NULL,"-ignore_zero_entries",&flg);CHKERRQ(ierr); >> > ?if (flg) { ?ierr = >> > MatSetOption(C,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE);CHKERRQ(ierr);} >> > >> > ?for (i=0; i<6; i++) v[i] = (PetscReal)i; >> > ?midx[0] = 0; midx[1] = 2; midx[2] = 3; >> > ?nidx[0] = 1; nidx[1] = 3; >> > ?ierr = MatSetValues(C,3,midx,2,nidx,v,ADD_VALUES);CHKERRQ(ierr); >> > ?ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> > ?ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> > ?ierr = MatView(C,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> > ?ierr = MatDestroy(&C);CHKERRQ(ierr); >> > ?ierr = PetscFinalize(); >> > ?return 0; >> > } >> > >> > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info >> > Matrix Object: 1 MPI processes >> > ?type: seqaij >> > ?rows=5, cols=5 >> > ?total: nonzeros=6, allocated nonzeros=25 >> > ?total number of mallocs used during MatSetValues calls =0 >> > ? ?using I-node routines: found 4 nodes, limit used is 5 >> > Matrix Object: 1 MPI processes >> > ?type: seqaij >> > row 0: (1, 0) ?(3, 1) >> > row 1: >> > row 2: (1, 2) ?(3, 3) >> > row 3: (1, 4) ?(3, 5) >> > row 4: >> > asterix:/home/balay/spetsc/src/mat/examples/tests>./ex38 -mat_view_info >> > -ignore_zero_entries >> > Matrix Object: 1 MPI processes >> > ?type: seqaij >> > ?rows=5, cols=5 >> > ?total: nonzeros=5, allocated nonzeros=25 >> > ?total number of mallocs used during MatSetValues calls =0 >> > ? ?using I-node routines: found 4 nodes, limit used is 5 >> > Matrix Object: 1 MPI processes >> > ?type: seqaij >> > row 0: (3, 1) >> > row 1: >> > row 2: (1, 2) ?(3, 3) >> > row 3: (1, 4) ?(3, 5) >> > row 4: >> > asterix:/home/balay/spetsc/src/mat/examples/tests> > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From balay at mcs.anl.gov Fri Dec 9 22:02:14 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 9 Dec 2011 22:02:14 -0600 (CST) Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Fri, 9 Dec 2011, Xiangdong Liang wrote: > Another quick question, when I use mat_view_draw command line option, > how can I make the figure X window larger? I think petsc > PetscDrawOpenX can do this job, but just wondering whether any quick > command line options? Thanks. You can use '-mat_view_draw -draw_pause -1' And then manually resize the window to the desired size by draging the corner with the mouse - and the use the combination of zoom-in/zoom-out [mouse-left,mouse-middle] to reposition the view as desired. Satish From xdliang at gmail.com Fri Dec 9 23:21:04 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Sat, 10 Dec 2011 00:21:04 -0500 Subject: [petsc-users] zeros in sparse aij In-Reply-To: References: <5B4CFC69-3CA6-42D8-92FF-C1B17FE8740D@mcs.anl.gov> <85802264-F0EA-4FDF-B42E-31A887B02725@mcs.anl.gov> Message-ID: On Fri, Dec 9, 2011 at 11:02 PM, Satish Balay wrote: > On Fri, 9 Dec 2011, Xiangdong Liang wrote: > >> Another quick question, when I use mat_view_draw command line option, >> how can I make the figure X window larger? I think petsc >> PetscDrawOpenX can do this job, but just wondering whether any quick >> command line options? Thanks. > > You can use '-mat_view_draw -draw_pause -1' > > And then manually resize the window to the desired size by draging the > corner with the mouse - and the use the combination of > zoom-in/zoom-out [mouse-left,mouse-middle] to reposition the view as > desired. Got it, Thanks. I am missing the "mouse-left, mouse-middle, and mouse right" parts before. > > Satish From xdliang at gmail.com Fri Dec 9 23:54:33 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Sat, 10 Dec 2011 00:54:33 -0500 Subject: [petsc-users] solve complex linear system: real or complex formulation Message-ID: Hello everyone, I am solving a complex linear system C x = d, where C= A+ iB (A, B are real), in sparse-direct solver. So far, I use the real formulation by solving the linear system [A, -B; B,A]. The reason we chose this approach is to use the property of that the imaginary part B in our problem is sparser than A. However, I just found that the spare-direct solver cannot benefit from this property. Now, I am thinking to re-implement it in to complex version by solving Cx=d in the complex version. Of course, complex formulation would perform better for iterative solvers since it only has half eigenvalues than the corresponding real formulations. I am wondering the comparison between real and complex formulation for sparse-direct solver. Can anyone share experiences on this? Thanks. Xiangdong From jack.poulson at gmail.com Sat Dec 10 10:03:20 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Sat, 10 Dec 2011 10:03:20 -0600 Subject: [petsc-users] solve complex linear system: real or complex formulation In-Reply-To: References: Message-ID: Xiangdong, Nearly all of the time in a serial sparse-direct factorization goes into performing many different dense "frontal" factorizations. Efficient implementations of dense LU factorization spend almost all of their time within dense matrix-matrix multiplication, and complex matrix-matrix multiplication requires about 4 times as much work as real matrix-matrix multiplication, as half the work is in adds and half the work is in multiplies with the standard approach (complex addition and multiplication are respectively require 2 and 6 real flops). Overall, a complex sparse-direct solve takes about 4 times as many flops (and twice as much memory) as a real sparse-direct solve, but due to having a higher arithmetic intensity (roughly, more flops per piece of data), the complex version will almost always take significantly less than 4 times as long as the real version. Jack On Fri, Dec 9, 2011 at 11:54 PM, Xiangdong Liang wrote: > Hello everyone, > > I am solving a complex linear system C x = d, where C= A+ iB (A, B are > real), in sparse-direct solver. So far, I use the real formulation by > solving the linear system [A, -B; B,A]. The reason we chose this > approach is to use the property of that the imaginary part B in our > problem is sparser than A. However, I just found that the spare-direct > solver cannot benefit from this property. > > Now, I am thinking to re-implement it in to complex version by solving > Cx=d in the complex version. Of course, complex formulation would > perform better for iterative solvers since it only has half > eigenvalues than the corresponding real formulations. I am wondering > the comparison between real and complex formulation for sparse-direct > solver. Can anyone share experiences on this? Thanks. > > Xiangdong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Sat Dec 10 11:58:45 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sat, 10 Dec 2011 11:58:45 -0600 Subject: [petsc-users] solve complex linear system: real or complex formulation In-Reply-To: References: Message-ID: I would definitely use complex precision. All direct sovler packages support complex, superlu, mumps and petsc. Hong On Sat, Dec 10, 2011 at 10:03 AM, Jack Poulson wrote: > Xiangdong, > > Nearly all of the time in a serial sparse-direct factorization goes into > performing many different dense "frontal" factorizations. Efficient > implementations of dense LU factorization spend almost all of their time > within dense matrix-matrix multiplication, and complex matrix-matrix > multiplication requires about 4 times as much work as real matrix-matrix > multiplication, as half the work is in adds and half the work is in > multiplies with the standard approach (complex addition and multiplication > are respectively require 2 and 6 real flops). > > Overall, a complex sparse-direct solve takes about 4 times as many flops > (and twice as much memory) as a real sparse-direct solve, but due to having > a higher arithmetic intensity (roughly, more flops per piece of data), the > complex version will almost always take significantly less than 4 times as > long as the real version. > > Jack > > > On Fri, Dec 9, 2011 at 11:54 PM, Xiangdong Liang wrote: >> >> Hello everyone, >> >> I am solving a complex linear system C x = d, where C= A+ iB (A, B are >> real), in sparse-direct solver. So far, I use the real formulation by >> solving the linear system [A, -B; B,A]. ?The reason we chose this >> approach is to use the property of that the imaginary part B in our >> problem is sparser than A. However, I just found that the spare-direct >> solver cannot benefit from this property. >> >> Now, I am thinking to re-implement it in to complex version by solving >> Cx=d in the complex version. Of course, complex formulation would >> perform better for iterative solvers since it only has half >> eigenvalues than the corresponding real formulations. I am wondering >> the comparison between real and complex formulation for sparse-direct >> solver. Can anyone share experiences on this? Thanks. >> >> Xiangdong > > From xdliang at gmail.com Sun Dec 11 00:05:15 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Sun, 11 Dec 2011 01:05:15 -0500 Subject: [petsc-users] PaStix is slower in pestc-dev Message-ID: Hello PETSc team, I was using PaStiX within petsc 3.1-p8. Today, I am trying PaStiX within petsc-dev. However, For the same code, solving the same linear system takes longer time (120s vs 90s) in petsc-dev. Both are compiled with debugging mode off. Is it possible that the newer PaStiX is slower than old version? or due to some options in compiling? Best, Xiangdong PS. I cannot use PaStiX in 3.2 because in the runtime, PaStiX crashed due to missing of the option -DWITHSCOTCH during compiling. It was reported before by one user and fixed in petsc-dev, but not in petsc-3.2-p5. http://lists.mcs.anl.gov/pipermail/petsc-users/2011-March/008356.html From uwe.schlifkowitz at uibk.ac.at Sun Dec 11 06:45:31 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Sun, 11 Dec 2011 13:45:31 +0100 Subject: [petsc-users] MatGetDiagonal Message-ID: <8B5EADFB-B5A5-4BAB-8EBB-9F2C531094E9@uibk.ac.at> Dear list, as explained in the last thread (see http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009706.html ) about MatGetDiagonal, the fix for a bug was pushed to petsc-dev. My results from running example 52 from ~petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c are attached below. Are these the correct results? I ran the example on Max OS X 10.6.8 as well as on CentOS with both petsc-3.2-p5 and petsc-dev (pulled from hg yesterday) and obtained the same results. Please let me know if you need the full output. (I recently started working on Clemens' project, i think he left in August.) Regards, Uwe $ ./ex52 -use_petsc_ch Matrix Object: 1 MPI processes type: seqaij row 0: (0, 4) (5, -1) (28, -1) (29, -1) row 1: (1, 4) (5, -1) (12, -1) (13, -1) ... Vector Object: 1 MPI processes type: seq 4 4 4 4 4 3.25 4 3.5 4 4 3.75 3.46548 3.1138 2.70534 ... 3.10993 2.55974 2.38793 2.32336 2.34087 2.5002 3.30916 Norm of error < 1.e-12 iterations 1 From knepley at gmail.com Sun Dec 11 08:04:40 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 11 Dec 2011 08:04:40 -0600 Subject: [petsc-users] PaStix is slower in pestc-dev In-Reply-To: References: Message-ID: On Sun, Dec 11, 2011 at 12:05 AM, Xiangdong Liang wrote: > Hello PETSc team, > > I was using PaStiX within petsc 3.1-p8. Today, I am trying PaStiX > within petsc-dev. However, For the same code, solving the same linear > system takes longer time (120s vs 90s) in petsc-dev. Both are compiled > with debugging mode off. Is it possible that the newer PaStiX is > slower than old version? or due to some options in compiling? > It is likely the options are not exactly the same, meaning the ordering is different, etc. Did you check everything wiht -ksp_view and -ksp_monitor? Matt > Best, > Xiangdong > > PS. I cannot use PaStiX in 3.2 because in the runtime, PaStiX crashed > due to missing of the option -DWITHSCOTCH during compiling. It was > reported before by one user and fixed in petsc-dev, but not in > petsc-3.2-p5. > > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-March/008356.html > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From uwe.schlifkowitz at uibk.ac.at Sun Dec 11 08:16:28 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Sun, 11 Dec 2011 15:16:28 +0100 Subject: [petsc-users] MatGetDiagonal In-Reply-To: <8B5EADFB-B5A5-4BAB-8EBB-9F2C531094E9@uibk.ac.at> References: <8B5EADFB-B5A5-4BAB-8EBB-9F2C531094E9@uibk.ac.at> Message-ID: <0B10C455-09DA-4D5B-B1EC-F7E531B8EC60@uibk.ac.at> On 11.12.2011, at 13:45, Uwe Schlifkowitz wrote: > Dear list, > > as explained in the last thread (see http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009706.html ) about MatGetDiagonal, the fix for a bug was pushed to petsc-dev. > > My results from running example 52 from ~petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c are attached below. Are these the correct results? > I ran the example on Max OS X 10.6.8 as well as on CentOS with both petsc-3.2-p5 and petsc-dev (pulled from hg yesterday) and obtained the same results. Please let me know if you need the full output. > > (I recently started working on Clemens' project, i think he left in August.) > > Regards, > Uwe > > $ ./ex52 -use_petsc_ch > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 4) (5, -1) (28, -1) (29, -1) > row 1: (1, 4) (5, -1) (12, -1) (13, -1) > ... > Vector Object: 1 MPI processes > type: seq > 4 > 4 > 4 > 4 > 4 > 3.25 > 4 > 3.5 > 4 > 4 > 3.75 > 3.46548 > 3.1138 > 2.70534 > ... > 3.10993 > 2.55974 > 2.38793 > 2.32336 > 2.34087 > 2.5002 > 3.30916 > Norm of error < 1.e-12 iterations 1 I just realized i somehow had an outdated ex52.c and makefile. Now i obtain the correct results as shown in http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009706.html . However, i'm wondering how come there's such a difference in the results. The output of diff -u $NEWFILE $OLDFILE is the following: $ diff -u ex52.c ex52.c.old --- ex52.c 2011-12-02 07:02:05.000000000 +0100 +++ ex52.c.old 2011-12-10 13:12:22.046776194 +0100 @@ -165,7 +165,7 @@ ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); } ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);CHKERRQ(ierr); - ierr = PCFactorSetUpMatSolverPackage(pc);CHKERRQ(ierr); /* call MatGetFactor() to create F */ + ierr = PCFactorSetUpMatSolverPackage(pc);CHKERRQ(ierr); ierr = PCFactorGetMatrix(pc,&F);CHKERRQ(ierr); icntl=7; ival = 2; ierr = MatMumpsSetIcntl(F,icntl,ival);CHKERRQ(ierr); @@ -262,7 +264,7 @@ print statement from all processes that share a communicator. An alternative is PetscFPrintf(), which prints to a file. */ - ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G iterations %D\n", + ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %A iterations %D\n", norm,its);CHKERRQ(ierr); /* The diff output for the makefiles is here: http://dl.dropbox.com/u/32772/diff-u-makefile-makefile.old.txt So the question is, what other parameters have been changed to produce said results? Regards, Uwe From agrayver at gfz-potsdam.de Sun Dec 11 09:29:44 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Sun, 11 Dec 2011 16:29:44 +0100 Subject: [petsc-users] Using Petsc with multiple RHS Message-ID: <4EE4CC68.2010605@gfz-potsdam.de> Hello, I used to use MUMPS directly with sparse multiple RHS. Now I use MUMPS through PETSc interface and solution for multiple RHS takes 1.5-2 times longer (MatMatSolve). My first question is whether do you use multiple RHS internally or you solve one-by-one? Second guess concerns the option: ICNTL(20) (rhs sparse pattern): 0 It's always switched off in PETSc, however my RHS are very sparse (8-32 nonzeros ans my system is 10^6). I can imagine that ignoring sparsity makes this performance difference. Any ideas are appreciated. Regards, Alexander From xdliang at gmail.com Sun Dec 11 10:08:05 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Sun, 11 Dec 2011 11:08:05 -0500 Subject: [petsc-users] PaStix is slower in pestc-dev In-Reply-To: References: Message-ID: On Sun, Dec 11, 2011 at 9:04 AM, Matthew Knepley wrote: > On Sun, Dec 11, 2011 at 12:05 AM, Xiangdong Liang wrote: >> >> Hello PETSc team, >> >> I was using PaStiX within petsc 3.1-p8. Today, I am trying PaStiX >> within petsc-dev. However, For the same code, solving the same linear >> system takes longer time (120s vs 90s) in petsc-dev. Both are compiled >> with debugging mode off. Is it possible that the newer PaStiX is >> slower than old version? or due to some options in compiling? > > > It is likely the options are not exactly the same, meaning the ordering is > different, > etc. Did you check everything wiht -ksp_view and -ksp_monitor? I use ksp_view, ksp_monitor options and same -pc_factor_zeropivot 1e-12, the dev version is still slower. One difference I see from pastix_verbose is the option of Pastix: In pets-3.1: I have DISTRIBUTED : Not defined FLUIDBOX : Not defined METIS : Not defined While in petsc-dev, I have DISTRIBUTED : Defined METIS : Not defined WITH_SCOTCH : Defined However, I do not understand these options. Could that be the reason? Xiangdong > > ? ?Matt > >> >> Best, >> Xiangdong >> >> PS. I cannot use PaStiX in 3.2 because in the runtime, PaStiX crashed >> due to missing of the option -DWITHSCOTCH during compiling. It was >> reported before by one user and fixed in petsc-dev, but not in >> petsc-3.2-p5. >> >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-March/008356.html > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Sun Dec 11 11:49:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 11 Dec 2011 11:49:27 -0600 Subject: [petsc-users] PaStix is slower in pestc-dev In-Reply-To: References: Message-ID: <1CB88C79-0BFC-4850-9B45-C5936F4265A9@mcs.anl.gov> Looks like Pastix is running with different options hence different performance. Barry On Dec 11, 2011, at 10:08 AM, Xiangdong Liang wrote: > On Sun, Dec 11, 2011 at 9:04 AM, Matthew Knepley wrote: >> On Sun, Dec 11, 2011 at 12:05 AM, Xiangdong Liang wrote: >>> >>> Hello PETSc team, >>> >>> I was using PaStiX within petsc 3.1-p8. Today, I am trying PaStiX >>> within petsc-dev. However, For the same code, solving the same linear >>> system takes longer time (120s vs 90s) in petsc-dev. Both are compiled >>> with debugging mode off. Is it possible that the newer PaStiX is >>> slower than old version? or due to some options in compiling? >> >> >> It is likely the options are not exactly the same, meaning the ordering is >> different, >> etc. Did you check everything wiht -ksp_view and -ksp_monitor? > > I use ksp_view, ksp_monitor options and same -pc_factor_zeropivot > 1e-12, the dev version is still slower. One difference I see from > pastix_verbose is the option of Pastix: > > In pets-3.1: I have > > DISTRIBUTED : Not defined > FLUIDBOX : Not defined > METIS : Not defined > > > While in petsc-dev, I have > > DISTRIBUTED : Defined > METIS : Not defined > WITH_SCOTCH : Defined > > > However, I do not understand these options. Could that be the reason? > > Xiangdong > >> >> Matt >> >>> >>> Best, >>> Xiangdong >>> >>> PS. I cannot use PaStiX in 3.2 because in the runtime, PaStiX crashed >>> due to missing of the option -DWITHSCOTCH during compiling. It was >>> reported before by one user and fixed in petsc-dev, but not in >>> petsc-3.2-p5. >>> >>> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-March/008356.html >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From bsmith at mcs.anl.gov Sun Dec 11 11:50:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 11 Dec 2011 11:50:34 -0600 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <4EE4CC68.2010605@gfz-potsdam.de> References: <4EE4CC68.2010605@gfz-potsdam.de> Message-ID: On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: > Hello, > > I used to use MUMPS directly with sparse multiple RHS. Now I use MUMPS through PETSc interface and solution for multiple RHS takes 1.5-2 times longer (MatMatSolve). > My first question is whether do you use multiple RHS internally or you solve one-by-one? One by one > Second guess concerns the option: > ICNTL(20) (rhs sparse pattern): 0 > > It's always switched off in PETSc, however my RHS are very sparse (8-32 nonzeros ans my system is 10^6). I can imagine that ignoring sparsity makes this performance difference. > Any ideas are appreciated. We do not handle a sparse right hand side. > > Regards, > Alexander > From xdliang at gmail.com Sun Dec 11 12:05:35 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Sun, 11 Dec 2011 13:05:35 -0500 Subject: [petsc-users] PaStix is slower in pestc-dev In-Reply-To: <1CB88C79-0BFC-4850-9B45-C5936F4265A9@mcs.anl.gov> References: <1CB88C79-0BFC-4850-9B45-C5936F4265A9@mcs.anl.gov> Message-ID: Is there an easy way to change these options? From MATSOLVERPASTIX, it seems that no command line options to control this. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATSOLVERPASTIX.html Do I have to modify PaStiX.py and recompile? If yes, would you like to give me some hints on where to modify? Thanks. Xiangdong On Sun, Dec 11, 2011 at 12:49 PM, Barry Smith wrote: > > ?Looks like Pastix is running with different options hence different performance. > > ? Barry > > On Dec 11, 2011, at 10:08 AM, Xiangdong Liang wrote: > >> On Sun, Dec 11, 2011 at 9:04 AM, Matthew Knepley wrote: >>> On Sun, Dec 11, 2011 at 12:05 AM, Xiangdong Liang wrote: >>>> >>>> Hello PETSc team, >>>> >>>> I was using PaStiX within petsc 3.1-p8. Today, I am trying PaStiX >>>> within petsc-dev. However, For the same code, solving the same linear >>>> system takes longer time (120s vs 90s) in petsc-dev. Both are compiled >>>> with debugging mode off. Is it possible that the newer PaStiX is >>>> slower than old version? or due to some options in compiling? >>> >>> >>> It is likely the options are not exactly the same, meaning the ordering is >>> different, >>> etc. Did you check everything wiht -ksp_view and -ksp_monitor? >> >> I ?use ksp_view, ksp_monitor options and same -pc_factor_zeropivot >> 1e-12, the dev version is still slower. One difference I see from >> pastix_verbose is the option of Pastix: >> >> In pets-3.1: I have >> >> ? ? ?DISTRIBUTED ? ? ? ? : ? ? ? ? ? ? ? ? ?Not defined >> ? ? ? FLUIDBOX ? ? ? ? ? ?: ? ? ? ? ? ? ? ? ?Not defined >> ? ? ? METIS ? ? ? ? ? ? ? : ? ? ? ? ? ? ? ? ?Not defined >> >> >> While in petsc-dev, I have >> >> ? ? ?DISTRIBUTED ? ? ? ? : ? ? ? ? ? ? ? ? ? Defined >> ? ? ? ?METIS ? ? ? ? ? ? ? : ? ? ? ? ? ? ? ? ? Not defined >> ? ? ? ?WITH_SCOTCH ? ? ? ? : ? ? ? ? ? ? ? ? ? Defined >> >> >> However, I do not understand these options. Could that be the reason? >> >> Xiangdong >> >>> >>> ? ?Matt >>> >>>> >>>> Best, >>>> Xiangdong >>>> >>>> PS. I cannot use PaStiX in 3.2 because in the runtime, PaStiX crashed >>>> due to missing of the option -DWITHSCOTCH during compiling. It was >>>> reported before by one user and fixed in petsc-dev, but not in >>>> petsc-3.2-p5. >>>> >>>> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-March/008356.html >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener > From behzad.baghapour at gmail.com Mon Dec 12 00:40:24 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 10:10:24 +0330 Subject: [petsc-users] How to add a new strategy for SNES? Message-ID: Dear developers, I need to add and examine a new strategy like "Line-Search" or "Trust-Region" in SNES context. How should follow the correct procedure? Regards, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 12 02:48:23 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 09:48:23 +0100 Subject: [petsc-users] debugging with no X11 available Message-ID: Hi, I am debugging my code on a system that does not allow any X11 connections, therefore the following does not work: mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 Are there alternative ways of using a debugger circumventing X11 connections? Thanks Dominik From behzad.baghapour at gmail.com Mon Dec 12 02:49:11 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 12:19:11 +0330 Subject: [petsc-users] Need to Know about Trust-Region Message-ID: Dear developers, In Petsc user manual, it was mentioned that Trust-Region Method is working upon MINPACK routines. I need to know more details of the procedure like: 1- How to update the TR bound? 2- How solving the minimization subproblem? 3- How to calculate the search direction? Is the "dogleg" direction is used? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Dec 12 03:10:22 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 12 Dec 2011 10:10:22 +0100 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: References: <4EE4CC68.2010605@gfz-potsdam.de> Message-ID: <4EE5C4FE.1040404@gfz-potsdam.de> Hi Barry, Thanks for answer. I should have asked about that from the very beginning actually. I get 2 times decrease in performance with 20 RHS, I can imagine how slow it will be when I will use thousands of them, moreover it will require a lot of memory to store them as a dense matrix. On 11.12.2011 18:50, Barry Smith wrote: > On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: > One by one I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, PaSTiX support multiple RHS. > We do not handle a sparse right hand side. Since I already transferred my code to PETSc anyway, my question now if it's possible to implement sparse multiple RHS and solve them simultaneously? Something like MatMatSolveSparse. I would implement it myself, the question is there a way to integrate it into PETSc as a patch or something like that? Regards, Alexander From knepley at gmail.com Mon Dec 12 06:44:56 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 06:44:56 -0600 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: The LS implementation is fairly well commented, but you may just need http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESLineSearchSet.html Matt On Mon, Dec 12, 2011 at 12:40 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear developers, > > I need to add and examine a new strategy like "Line-Search" or > "Trust-Region" in SNES context. How should follow the correct procedure? > > Regards, > BehZad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 12 06:46:43 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 06:46:43 -0600 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba wrote: > Hi, > I am debugging my code on a system that does not allow any X11 > connections, therefore the following does not work: > > mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 > > Are there alternative ways of using a debugger circumventing X11 > connections? > No. You can usually set the DISPLAY correctly for the backend machine. Consult the admin. Matt > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Mon Dec 12 06:48:59 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 12 Dec 2011 15:48:59 +0300 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Not on a BG/P you can't. A On Mon, Dec 12, 2011 at 3:46 PM, Matthew Knepley wrote: > On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba wrote: > >> Hi, >> I am debugging my code on a system that does not allow any X11 >> connections, therefore the following does not work: >> >> mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 >> >> Are there alternative ways of using a debugger circumventing X11 >> connections? >> > > No. You can usually set the DISPLAY correctly for the backend machine. > Consult the admin. > > Matt > > >> Thanks >> Dominik >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 12 06:50:14 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 06:50:14 -0600 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <4EE5C4FE.1040404@gfz-potsdam.de> References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> Message-ID: On Mon, Dec 12, 2011 at 3:10 AM, Alexander Grayver wrote: > Hi Barry, > > Thanks for answer. I should have asked about that from the very beginning > actually. > I get 2 times decrease in performance with 20 RHS, I can imagine how slow > it will be when I will use thousands of them, > moreover it will require a lot of memory to store them as a dense matrix. > > On 11.12.2011 18:50, Barry Smith wrote: > >> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >> One by one >> > > I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, > PaSTiX support multiple RHS. > We don't need it for the vast majority of problems. > We do not handle a sparse right hand side. > > Since I already transferred my code to PETSc anyway, my question now if > it's possible to implement sparse multiple RHS and solve them > simultaneously? > Something like MatMatSolveSparse. I would implement it myself, the > question is there a way to integrate it into PETSc as a patch or something > like that? > You can just implement MatMatSolve for another matrix type, like AIJ (although you probably want the transpose). Matt > Regards, > Alexander > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Mon Dec 12 06:52:54 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 12 Dec 2011 15:52:54 +0300 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Sorry, to expand my answer, "what Matt said". You don't have the advantage of using the functionality of -start_in_debugger if you cannot make X11 connections from the compute nodes, so you will need to rely on whatever debugging facilities your systems team has set up, and break on PetscError if you want to catch PETSc errors percolating up the stack. A On Mon, Dec 12, 2011 at 3:48 PM, Aron Ahmadia wrote: > Not on a BG/P you can't. > > A > > > On Mon, Dec 12, 2011 at 3:46 PM, Matthew Knepley wrote: > >> On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba wrote: >> >>> Hi, >>> I am debugging my code on a system that does not allow any X11 >>> connections, therefore the following does not work: >>> >>> mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 >>> >>> Are there alternative ways of using a debugger circumventing X11 >>> connections? >>> >> >> No. You can usually set the DISPLAY correctly for the backend machine. >> Consult the admin. >> >> Matt >> >> >>> Thanks >>> Dominik >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 12 07:11:34 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 14:11:34 +0100 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Thanks for your answers. Meanwhile I clarified the situation a bit: I can bring xterm up manually from the command line, but the job is run using a scheduler (slurm). It then gets executed on arbitrary nodes (some stripped down linux) which apparently can not make X11 connections. Of course, they have their own debugging environment, but I just do not want to learn it now. The admins are unable to help me more. I tried running the debugger explicitly: aprun -n 2 -N 1 -d 1 -cc cpu gdb Solver run.xml but so only one gdb instance is invoked in the terminal window, the other is in a parallel universe or something... Still any chance to get the second window somehow? Thanks Dominik On Mon, Dec 12, 2011 at 1:52 PM, Aron Ahmadia wrote: > Sorry, to expand my answer, "what Matt said". ?You don't have the advantage > of using the functionality of -start_in_debugger if you cannot make X11 > connections from the compute nodes, so you will need to rely on whatever > debugging facilities your systems team has set up, and break on PetscError > if you want to catch PETSc errors percolating up the stack. > > A > > > On Mon, Dec 12, 2011 at 3:48 PM, Aron Ahmadia > wrote: >> >> Not on a BG/P you can't. >> >> A >> >> >> On Mon, Dec 12, 2011 at 3:46 PM, Matthew Knepley >> wrote: >>> >>> On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba >>> wrote: >>>> >>>> Hi, >>>> I am debugging my code on a system that does not allow any X11 >>>> connections, therefore the following does not work: >>>> >>>> mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 >>>> >>>> Are there alternative ways of using a debugger circumventing X11 >>>> connections? >>> >>> >>> No. You can usually set the DISPLAY correctly for the backend machine. >>> Consult the admin. >>> >>> ? ?Matt >>> >>>> >>>> Thanks >>>> Dominik >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >> >> > From knepley at gmail.com Mon Dec 12 07:15:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 07:15:28 -0600 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 7:11 AM, Dominik Szczerba wrote: > Thanks for your answers. Meanwhile I clarified the situation a bit: > > I can bring xterm up manually from the command line, but the job is > run using a scheduler (slurm). It then gets executed on arbitrary > nodes (some stripped down linux) which apparently can not make X11 > connections. > Sometimes you can set the env on the compute nodes, and get DISPLAY right. > Of course, they have their own debugging environment, but I just do > not want to learn it now. The admins are unable to help me more. I > They should definitely be fired. > tried running the debugger explicitly: > > aprun -n 2 -N 1 -d 1 -cc cpu gdb Solver run.xml > > but so only one gdb instance is invoked in the terminal window, the > other is in a parallel universe or something... Still any chance to > get the second window somehow? > Why not run 2 procs locally, instead of through the scheduler? Matt > Thanks > Dominik > > On Mon, Dec 12, 2011 at 1:52 PM, Aron Ahmadia > wrote: > > Sorry, to expand my answer, "what Matt said". You don't have the > advantage > > of using the functionality of -start_in_debugger if you cannot make X11 > > connections from the compute nodes, so you will need to rely on whatever > > debugging facilities your systems team has set up, and break on > PetscError > > if you want to catch PETSc errors percolating up the stack. > > > > A > > > > > > On Mon, Dec 12, 2011 at 3:48 PM, Aron Ahmadia > > > wrote: > >> > >> Not on a BG/P you can't. > >> > >> A > >> > >> > >> On Mon, Dec 12, 2011 at 3:46 PM, Matthew Knepley > >> wrote: > >>> > >>> On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba < > dominik at itis.ethz.ch> > >>> wrote: > >>>> > >>>> Hi, > >>>> I am debugging my code on a system that does not allow any X11 > >>>> connections, therefore the following does not work: > >>>> > >>>> mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 > >>>> > >>>> Are there alternative ways of using a debugger circumventing X11 > >>>> connections? > >>> > >>> > >>> No. You can usually set the DISPLAY correctly for the backend machine. > >>> Consult the admin. > >>> > >>> Matt > >>> > >>>> > >>>> Thanks > >>>> Dominik > >>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > their > >>> experiments lead. > >>> -- Norbert Wiener > >> > >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Dec 12 07:18:26 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 12 Dec 2011 07:18:26 -0600 (CST) Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: If you just need any one node with the debugger - you can do: mpiexec -n 4 ./ex2 -start_in_debugger noxterm -debugger_node 0 [or 1 or 2 or 3] If you need to use a debugger on multiple nodes - you'll have to figure out if you can start your parallel job with multiple commands [from multiple terminals]. Satish On Mon, 12 Dec 2011, Dominik Szczerba wrote: > Thanks for your answers. Meanwhile I clarified the situation a bit: > > I can bring xterm up manually from the command line, but the job is > run using a scheduler (slurm). It then gets executed on arbitrary > nodes (some stripped down linux) which apparently can not make X11 > connections. > Of course, they have their own debugging environment, but I just do > not want to learn it now. The admins are unable to help me more. I > tried running the debugger explicitly: > > aprun -n 2 -N 1 -d 1 -cc cpu gdb Solver run.xml > > but so only one gdb instance is invoked in the terminal window, the > other is in a parallel universe or something... Still any chance to > get the second window somehow? > > Thanks > Dominik > > On Mon, Dec 12, 2011 at 1:52 PM, Aron Ahmadia wrote: > > Sorry, to expand my answer, "what Matt said". ?You don't have the advantage > > of using the functionality of -start_in_debugger if you cannot make X11 > > connections from the compute nodes, so you will need to rely on whatever > > debugging facilities your systems team has set up, and break on PetscError > > if you want to catch PETSc errors percolating up the stack. > > > > A > > > > > > On Mon, Dec 12, 2011 at 3:48 PM, Aron Ahmadia > > wrote: > >> > >> Not on a BG/P you can't. > >> > >> A > >> > >> > >> On Mon, Dec 12, 2011 at 3:46 PM, Matthew Knepley > >> wrote: > >>> > >>> On Mon, Dec 12, 2011 at 2:48 AM, Dominik Szczerba > >>> wrote: > >>>> > >>>> Hi, > >>>> I am debugging my code on a system that does not allow any X11 > >>>> connections, therefore the following does not work: > >>>> > >>>> mpiexec -n 2 solver run.xml -start_in_debugger -display :0.0 > >>>> > >>>> Are there alternative ways of using a debugger circumventing X11 > >>>> connections? > >>> > >>> > >>> No. You can usually set the DISPLAY correctly for the backend machine. > >>> Consult the admin. > >>> > >>> ? ?Matt > >>> > >>>> > >>>> Thanks > >>>> Dominik > >>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which their > >>> experiments lead. > >>> -- Norbert Wiener > >> > >> > > > From balay at mcs.anl.gov Mon Dec 12 07:21:44 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 12 Dec 2011 07:21:44 -0600 (CST) Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: On Mon, 12 Dec 2011, Matthew Knepley wrote: > On Mon, Dec 12, 2011 at 7:11 AM, Dominik Szczerba wrote: > > > Thanks for your answers. Meanwhile I clarified the situation a bit: > > > > I can bring xterm up manually from the command line, but the job is > > run using a scheduler (slurm). It then gets executed on arbitrary > > nodes (some stripped down linux) which apparently can not make X11 > > connections. > > > > Sometimes you can set the env on the compute nodes, and get DISPLAY right. With compute nodes -its not easy. Even if they have X11 installed - you might have to do multiple things: 1. allocate nodes 2. create ssh-x11 tunnels to each node thats allocated [and hope its the same localhost:10 value] 3. Now start up the parallel job with this display mpiexec -n 4 ./ex2 -start_in_debugger -display localhost:10 Satish From dominik at itis.ethz.ch Mon Dec 12 07:27:58 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 14:27:58 +0100 Subject: [petsc-users] example to use GetVecType Message-ID: Is there a C example how to use GetVecType? I am trying to do something like // VecType* type; char type[1024]; ierr = VecGetType(b, type); ierr = PetscSynchronizedFPrintf(PETSC_COMM_WORLD, stderr, "%s\n", type); CHKERRQ(ierr); but always get this or another error like cannot convert 'char*' to 'const char**' or similar. Many thanks Dominik From dominik at itis.ethz.ch Mon Dec 12 07:35:27 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 14:35:27 +0100 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Dear all, Many thanks for your suggestions, but there is one fundamental obstacle: there is no mpiexec available at all, everything must go through a scheduler. I will see into what Satish proposed, i.e. to first get hold of known nodes. But I am afraid this may fail, because these run some very limited system. Regards, Dominik On Mon, Dec 12, 2011 at 2:21 PM, Satish Balay wrote: > On Mon, 12 Dec 2011, Matthew Knepley wrote: > >> On Mon, Dec 12, 2011 at 7:11 AM, Dominik Szczerba wrote: >> >> > Thanks for your answers. Meanwhile I clarified the situation a bit: >> > >> > I can bring xterm up manually from the command line, but the job is >> > run using a scheduler (slurm). It then gets executed on arbitrary >> > nodes (some stripped down linux) which apparently can not make X11 >> > connections. >> > >> >> Sometimes you can set the env on the compute nodes, and get DISPLAY right. > > > With compute nodes -its not easy. Even if they have X11 installed - you > might have to do multiple things: > > 1. allocate nodes > 2. create ssh-x11 tunnels to each node thats allocated [and hope its the same localhost:10 value] > 3. Now start up the parallel job with this display > > mpiexec -n 4 ./ex2 -start_in_debugger -display localhost:10 > > Satish > From knepley at gmail.com Mon Dec 12 07:36:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 07:36:03 -0600 Subject: [petsc-users] example to use GetVecType In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 7:27 AM, Dominik Szczerba wrote: > Is there a C example how to use GetVecType? > I am trying to do something like > > // VecType* type; > char type[1024]; > ierr = VecGetType(b, type); > VecType t; VecGetType(b, &t); Printf("%s\n", t); Matt > ierr = PetscSynchronizedFPrintf(PETSC_COMM_WORLD, stderr, "%s\n", > type); CHKERRQ(ierr); > > but always get this or another error like cannot convert 'char*' to > 'const char**' or similar. > > Many thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 12 07:39:43 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 14:39:43 +0100 Subject: [petsc-users] example to use GetVecType In-Reply-To: References: Message-ID: > VecType t; > > VecGetType(b, &t); > Printf("%s\n", t); > > ? Matt Thanks, but I am getting the same error: error: invalid conversion from 'char**' to 'const char**' Using gcc 4.6.2 Dominik From aron.ahmadia at kaust.edu.sa Mon Dec 12 07:44:28 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 12 Dec 2011 16:44:28 +0300 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Dear Dominik, One trick for getting around this that works on LoadLeveler (and I suspect SLURM) is running: xterm Instead of the usual "mpirun" when your batch script gets executed. As long as the scheduler's batch script inherits your X11 environment and is running on the login node, you'll then have access to an xterm with full access to mpirun, ssh, etc... Good luck, Aron On Mon, Dec 12, 2011 at 4:35 PM, Dominik Szczerba wrote: > Dear all, > > Many thanks for your suggestions, but there is one fundamental > obstacle: there is no mpiexec available at all, everything must go > through a scheduler. > > I will see into what Satish proposed, i.e. to first get hold of known > nodes. But I am afraid this may fail, because these run some very > limited system. > > Regards, > Dominik > > On Mon, Dec 12, 2011 at 2:21 PM, Satish Balay wrote: > > On Mon, 12 Dec 2011, Matthew Knepley wrote: > > > >> On Mon, Dec 12, 2011 at 7:11 AM, Dominik Szczerba >wrote: > >> > >> > Thanks for your answers. Meanwhile I clarified the situation a bit: > >> > > >> > I can bring xterm up manually from the command line, but the job is > >> > run using a scheduler (slurm). It then gets executed on arbitrary > >> > nodes (some stripped down linux) which apparently can not make X11 > >> > connections. > >> > > >> > >> Sometimes you can set the env on the compute nodes, and get DISPLAY > right. > > > > > > With compute nodes -its not easy. Even if they have X11 installed - you > > might have to do multiple things: > > > > 1. allocate nodes > > 2. create ssh-x11 tunnels to each node thats allocated [and hope its the > same localhost:10 value] > > 3. Now start up the parallel job with this display > > > > mpiexec -n 4 ./ex2 -start_in_debugger -display localhost:10 > > > > Satish > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 12 07:48:54 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 07:48:54 -0600 Subject: [petsc-users] example to use GetVecType In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 7:39 AM, Dominik Szczerba wrote: > > VecType t; > > > > VecGetType(b, &t); > > Printf("%s\n", t); > > > > Matt > > Thanks, but I am getting the same error: > > error: invalid conversion from 'char**' to 'const char**' > Cast it to (const VecType *). > Using gcc 4.6.2 A C compiler would not give you that error. Matt > > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 12 08:04:58 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 15:04:58 +0100 Subject: [petsc-users] example to use GetVecType In-Reply-To: References: Message-ID: >> > VecType t; >> > >> > VecGetType(b, &t); >> > Printf("%s\n", t); >> > >> > ? Matt >> >> Thanks, but I am getting the same error: >> >> error: invalid conversion from 'char**' to 'const char**' > > Cast it to (const VecType *). This works, thanks! >> Using gcc 4.6.2 > A C compiler would not give you that error. Correct, I have meant g++ 4.6.2, sorry. Dominik From dominik at itis.ethz.ch Mon Dec 12 08:13:17 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 15:13:17 +0100 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Thanks for a hint, but xterm does not seem to work with the scheduler: xterm Xt error: Can't open display: localhost:54.0 It works, however, when invoked explicitly from the shell. I experimented a lot with the DISPLAY variable to no avail... It would be very helpful to be able to run 2 processes in a debugger without relying on X11 at all... Thanks a lot anyway, Dominik On Mon, Dec 12, 2011 at 2:44 PM, Aron Ahmadia wrote: > Dear Dominik, > > One trick for getting around this that works on LoadLeveler (and I suspect > SLURM) is running: > > xterm > > Instead of the usual "mpirun" when your batch script gets executed. ?As long > as the scheduler's batch script inherits your X11 environment and is running > on the login node, you'll then have access to an xterm with full access to > mpirun, ssh, etc... > > Good luck, > Aron > > > On Mon, Dec 12, 2011 at 4:35 PM, Dominik Szczerba > wrote: >> >> Dear all, >> >> Many thanks for your suggestions, but there is one fundamental >> obstacle: there is no mpiexec available at all, everything must go >> through a scheduler. >> >> I will see into what Satish proposed, i.e. to first get hold of known >> nodes. But I am afraid this may fail, because these run some very >> limited system. >> >> Regards, >> Dominik >> >> On Mon, Dec 12, 2011 at 2:21 PM, Satish Balay wrote: >> > On Mon, 12 Dec 2011, Matthew Knepley wrote: >> > >> >> On Mon, Dec 12, 2011 at 7:11 AM, Dominik Szczerba >> >> wrote: >> >> >> >> > Thanks for your answers. Meanwhile I clarified the situation a bit: >> >> > >> >> > I can bring xterm up manually from the command line, but the job is >> >> > run using a scheduler (slurm). It then gets executed on arbitrary >> >> > nodes (some stripped down linux) which apparently can not make X11 >> >> > connections. >> >> > >> >> >> >> Sometimes you can set the env on the compute nodes, and get DISPLAY >> >> right. >> > >> > >> > With compute nodes -its not easy. Even if they have X11 installed - you >> > might have to do multiple things: >> > >> > 1. allocate nodes >> > 2. create ssh-x11 tunnels to each node thats allocated [and hope its the >> > same localhost:10 value] >> > 3. Now start up the parallel job with this display >> > >> > mpiexec -n 4 ./ex2 -start_in_debugger -display localhost:10 >> > >> > Satish >> > > > From aron.ahmadia at kaust.edu.sa Mon Dec 12 08:19:43 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 12 Dec 2011 17:19:43 +0300 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: > > It would be very helpful to be able to run 2 processes in a debugger > without relying on X11 at all... > As several others suggested, you can probably do this on your own laptop or workstation if you can reproduce the problem locally. This is how I debug all of my parallel code. A -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 12 08:23:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Dec 2011 08:23:20 -0600 Subject: [petsc-users] Need to Know about Trust-Region In-Reply-To: References: Message-ID: <1413F959-F3A6-4965-91D2-4EAFA10B37F7@mcs.anl.gov> BehZad, All the source code for the trust region method can be found in src/snes/impls/tr You can find the details there. Barry > BehZad On Dec 12, 2011, at 2:49 AM, behzad baghapour wrote: > Dear developers, > > In Petsc user manual, it was mentioned that Trust-Region Method is working upon MINPACK routines. I need to know more details of the procedure like: > > 1- How to update the TR bound? > 2- How solving the minimization subproblem? > 3- How to calculate the search direction? Is the "dogleg" direction is used? > > Thanks a lot, > BehZad From bsmith at mcs.anl.gov Mon Dec 12 08:26:26 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Dec 2011 08:26:26 -0600 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <4EE5C4FE.1040404@gfz-potsdam.de> References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> Message-ID: <28183CC9-7A6B-4F8A-973F-6C4774DDD6E0@mcs.anl.gov> Alexander, Do you expect that solving with a sparse right hand side will result in an answer that is also sparse? Normally once you've done the lower triangular solve and then the upper triangular solve won't the result be dense? Barry On Dec 12, 2011, at 3:10 AM, Alexander Grayver wrote: > Hi Barry, > > Thanks for answer. I should have asked about that from the very beginning actually. > I get 2 times decrease in performance with 20 RHS, I can imagine how slow it will be when I will use thousands of them, > moreover it will require a lot of memory to store them as a dense matrix. > > On 11.12.2011 18:50, Barry Smith wrote: >> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >> One by one > > I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, PaSTiX support multiple RHS. > >> We do not handle a sparse right hand side. > > Since I already transferred my code to PETSc anyway, my question now if it's possible to implement sparse multiple RHS and solve them simultaneously? > Something like MatMatSolveSparse. I would implement it myself, the question is there a way to integrate it into PETSc as a patch or something like that? > > Regards, > Alexander From hzhang at mcs.anl.gov Mon Dec 12 08:37:47 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 12 Dec 2011 08:37:47 -0600 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <28183CC9-7A6B-4F8A-973F-6C4774DDD6E0@mcs.anl.gov> References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> <28183CC9-7A6B-4F8A-973F-6C4774DDD6E0@mcs.anl.gov> Message-ID: Reading MUMPS manual, mumps only supports centralized dense and sparse rhs, while solution can be centralized or distributed, but only in the dense format (because manual does not mention sparse solu). It would be easy to add MatMatSolve() in Petsc-mumps interface for dense rhs, but will require vecscatter of petsc distributed dense matrix for rhs to mumps centralized dense rhs - I doubt it is feasible for thousands of rhs. Hong On Mon, Dec 12, 2011 at 8:26 AM, Barry Smith wrote: > > ? Alexander, > > ? ?Do you expect that solving with a sparse right hand side will result in an answer that is also sparse? Normally once you've done the lower triangular solve and then the upper triangular solve won't the result be dense? > > ? ?Barry > > On Dec 12, 2011, at 3:10 AM, Alexander Grayver wrote: > >> Hi Barry, >> >> Thanks for answer. I should have asked about that from the very beginning actually. >> I get 2 times decrease in performance with 20 RHS, I can imagine how slow it will be when I will use thousands of them, >> moreover it will require a lot of memory to store them as a dense matrix. >> >> On 11.12.2011 18:50, Barry Smith wrote: >>> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >>> ? ?One by one >> >> I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, PaSTiX support multiple RHS. >> >>> ? ?We do not handle a sparse right hand side. >> >> Since I already transferred my code to PETSc anyway, my question now if it's possible to implement sparse multiple RHS and solve them simultaneously? >> Something like MatMatSolveSparse. I would implement it myself, the question is there a way to integrate it into PETSc as a patch or something like that? >> >> Regards, >> Alexander > From dominik at itis.ethz.ch Mon Dec 12 09:15:51 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 16:15:51 +0100 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: > As several others suggested, you can probably do this on your own laptop or > workstation if you can reproduce the problem locally. ?This is how I debug > all of my parallel code. The problem is there is no problem on my own machine... only there :`( OK, giving up, probably I need to either learn their debugger or debug by foot, commenting out single code lines and adding verbosity.... Many thanks and best regards, Dominik From agrayver at gfz-potsdam.de Mon Dec 12 09:22:49 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 12 Dec 2011 16:22:49 +0100 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> <28183CC9-7A6B-4F8A-973F-6C4774DDD6E0@mcs.anl.gov> Message-ID: <4EE61C49.60406@gfz-potsdam.de> Matt, Barry & Hong, Thanks a lot for response. I may be wrong, but I think that people chose direct solvers for their robustness and ability to solve for many RHS very quickly. Surely I don't expect my solution to be sparse and it is not, but at least in electromagnetics it is pretty common to have sparse RHS. I am wondering how about other disciplines? I don't know about other solvers, but when I use multiple RHS in mumps it works several times faster then solving for them sequentially. It is just my experience. And I also can imagine that using sparsity is another obvious advantage if you have sparse RHS. Now for 1000 rhs and a system of order 10^6 I need 32 gb to use MatMatSolve with double complex arithmetic. For instance, LDLt factorization of matrix 10^6 takes ~80 gb. One option would be to use MatMatSolve(A,X,X) thus reducing memory consumption twice. >> I doubt it is feasible for thousands of rhs. You're right, it is not. It is feasible only if you have sparse RHS. Regards, Alexander On 12.12.2011 15:37, Hong Zhang wrote: > Reading MUMPS manual, mumps only supports centralized dense and sparse rhs, > while solution can be centralized or distributed, but only in the > dense format (because manual > does not mention sparse solu). > > It would be easy to add MatMatSolve() in Petsc-mumps interface for dense rhs, > but will require vecscatter of petsc distributed dense matrix for rhs > to mumps centralized > dense rhs - I doubt it is feasible for thousands of rhs. > > Hong > > > On Mon, Dec 12, 2011 at 8:26 AM, Barry Smith wrote: >> Alexander, >> >> Do you expect that solving with a sparse right hand side will result in an answer that is also sparse? Normally once you've done the lower triangular solve and then the upper triangular solve won't the result be dense? >> >> Barry >> >> On Dec 12, 2011, at 3:10 AM, Alexander Grayver wrote: >> >>> Hi Barry, >>> >>> Thanks for answer. I should have asked about that from the very beginning actually. >>> I get 2 times decrease in performance with 20 RHS, I can imagine how slow it will be when I will use thousands of them, >>> moreover it will require a lot of memory to store them as a dense matrix. >>> >>> On 11.12.2011 18:50, Barry Smith wrote: >>>> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >>>> One by one >>> I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, PaSTiX support multiple RHS. >>> >>>> We do not handle a sparse right hand side. >>> Since I already transferred my code to PETSc anyway, my question now if it's possible to implement sparse multiple RHS and solve them simultaneously? >>> Something like MatMatSolveSparse. I would implement it myself, the question is there a way to integrate it into PETSc as a patch or something like that? >>> >>> Regards, >>> Alexander From behzad.baghapour at gmail.com Mon Dec 12 09:49:51 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 19:19:51 +0330 Subject: [petsc-users] Need to Know about Trust-Region In-Reply-To: <1413F959-F3A6-4965-91D2-4EAFA10B37F7@mcs.anl.gov> References: <1413F959-F3A6-4965-91D2-4EAFA10B37F7@mcs.anl.gov> Message-ID: Thanks, I will check... -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 12 10:18:45 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 19:48:45 +0330 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: Thanks. I should try it. Is there any option for Trust Region too? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 12 10:24:19 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 10:24:19 -0600 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 10:18 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Thanks. I should try it. Is there any option for Trust Region too? > No. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 12 10:29:06 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 19:59:06 +0330 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: OK. Then How I can access to Jacobian matrix in SNESLineSearchSet() in order to implement some steps in my procedure? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 12 10:37:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Dec 2011 10:37:48 -0600 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 10:29 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > OK. Then How I can access to Jacobian matrix in SNESLineSearchSet() in > order to implement some steps in my procedure? > See the default method: http://www.mcs.anl.gov/petsc/petsc-current/src/snes/impls/ls/ls.c.html#SNESLineSearchCubic Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 12 11:14:46 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 12 Dec 2011 20:44:46 +0330 Subject: [petsc-users] How to add a new strategy for SNES? In-Reply-To: References: Message-ID: OK. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From uwe.schlifkowitz at uibk.ac.at Mon Dec 12 11:35:46 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Mon, 12 Dec 2011 18:35:46 +0100 Subject: [petsc-users] Return value of PCFactorGetMatrix Message-ID: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> Dear list, following advice from this list and petsc documentation (especially http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex52.c.html ), i have the following code: //--------- ierr = KSPCreate( coml,&kspBA);CHKERRQ(ierr); ierr = KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetType(kspBA,KSPPREONLY);CHKERRQ(ierr); ierr = KSPGetPC(kspBA, &pcba); ierr = PCSetType(pcba, PCCHOLESKY);CHKERRQ(ierr); ierr = PCFactorSetMatSolverPackage(pcba, MATSOLVERPETSC);CHKERRQ(ierr); ierr = PCFactorSetUpMatSolverPackage(pcba);CHKERRQ(ierr); ierr = MatGetFactor(Kt,"petsc",MAT_FACTOR_CHOLESKY,&DiagM);CHKERRQ(ierr); ierr = PCFactorGetMatrix(pcba, &DiagM);CHKERRQ(ierr); ierr = KSPSetUp(kspBA);CHKERRQ(ierr); //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = VecDuplicate(fin,&diag);CHKERRQ(ierr); ierr = MatGetDiagonal(DiagM,diag);CHKERRQ(ierr); //----------- where Kt is a stiffness matrix. I was hoping that PCFactorGetMatrix() makes DiagM become D from a LDL^T factorization, but to me it seems as if this is not true. Can anyone clarify? Regards, Uwe From bsmith at mcs.anl.gov Mon Dec 12 11:40:18 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Dec 2011 11:40:18 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> Message-ID: A "factored" matrix in PETSc whether with a PETSc implementation or an external implementation such as MUMPS, or SuperLU is a weird beast. It is likely in some strange convoluted data structure from which extracting information is not easy. What information, exactly do you want to obtain from the factored matrix? If we know that we might be able to help you get it. Barry On Dec 12, 2011, at 11:35 AM, Uwe Schlifkowitz wrote: > Dear list, > > following advice from this list and petsc documentation (especially http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex52.c.html ), i have the following code: > > //--------- > ierr = KSPCreate( coml,&kspBA);CHKERRQ(ierr); > ierr = KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > > ierr = KSPSetType(kspBA,KSPPREONLY);CHKERRQ(ierr); > > ierr = KSPGetPC(kspBA, &pcba); > > ierr = PCSetType(pcba, PCCHOLESKY);CHKERRQ(ierr); > > ierr = PCFactorSetMatSolverPackage(pcba, MATSOLVERPETSC);CHKERRQ(ierr); > ierr = PCFactorSetUpMatSolverPackage(pcba);CHKERRQ(ierr); > ierr = MatGetFactor(Kt,"petsc",MAT_FACTOR_CHOLESKY,&DiagM);CHKERRQ(ierr); > ierr = PCFactorGetMatrix(pcba, &DiagM);CHKERRQ(ierr); > > ierr = KSPSetUp(kspBA);CHKERRQ(ierr); > //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > ierr = VecDuplicate(fin,&diag);CHKERRQ(ierr); > ierr = MatGetDiagonal(DiagM,diag);CHKERRQ(ierr); > //----------- > > where Kt is a stiffness matrix. I was hoping that PCFactorGetMatrix() makes DiagM become D from a LDL^T factorization, but to me it seems as if this is not true. > > Can anyone clarify? > > Regards, > Uwe From abhyshr at mcs.anl.gov Mon Dec 12 11:44:34 2011 From: abhyshr at mcs.anl.gov (Shri) Date: Mon, 12 Dec 2011 11:44:34 -0600 (CST) Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> Message-ID: <248733558.56683.1323711874447.JavaMail.root@zimbra.anl.gov> MatGetFactor() only sets up the matrix for factorization but does not do any factorization. You need to call MatCholeskyFactorSymbolic() http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCholeskyFactorSymbolic.html and MatCholeskyFactorNumeric() http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCholeskyFactorNumeric.html#MatCholeskyFactorNumeric after MatGetFactor(). Shri ----- Original Message ----- > Dear list, > > following advice from this list and petsc documentation (especially > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex52.c.html > ), i have the following code: > > //--------- > ierr = KSPCreate( coml,&kspBA);CHKERRQ(ierr); > ierr = KSPSetOperators( kspBA, Kt, Kt, > DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > > ierr = KSPSetType(kspBA,KSPPREONLY);CHKERRQ(ierr); > > ierr = KSPGetPC(kspBA, &pcba); > > ierr = PCSetType(pcba, PCCHOLESKY);CHKERRQ(ierr); > > ierr = PCFactorSetMatSolverPackage(pcba, > MATSOLVERPETSC);CHKERRQ(ierr); > ierr = PCFactorSetUpMatSolverPackage(pcba);CHKERRQ(ierr); > ierr = > MatGetFactor(Kt,"petsc",MAT_FACTOR_CHOLESKY,&DiagM);CHKERRQ(ierr); > ierr = PCFactorGetMatrix(pcba, &DiagM);CHKERRQ(ierr); > > ierr = KSPSetUp(kspBA);CHKERRQ(ierr); > //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); > ierr = VecDuplicate(fin,&diag);CHKERRQ(ierr); > ierr = MatGetDiagonal(DiagM,diag);CHKERRQ(ierr); > //----------- > > where Kt is a stiffness matrix. I was hoping that PCFactorGetMatrix() > makes DiagM become D from a LDL^T factorization, but to me it seems as > if this is not true. > > Can anyone clarify? > > Regards, > Uwe From uwe.schlifkowitz at uibk.ac.at Mon Dec 12 11:49:45 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Mon, 12 Dec 2011 18:49:45 +0100 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> Message-ID: <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> I'm continuing Clemens Domanig's work. Currently I'm stuck at his problem as described here: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html As far as I understand, getting the diagonal matrix from MUMPS is not possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. Uwe On 12.12.2011, at 18:40, Barry Smith wrote: > > A "factored" matrix in PETSc whether with a PETSc implementation or an external implementation such as MUMPS, or SuperLU is a weird beast. It is likely in some strange convoluted data structure from which extracting information is not easy. > > What information, exactly do you want to obtain from the factored matrix? If we know that we might be able to help you get it. > > Barry > > On Dec 12, 2011, at 11:35 AM, Uwe Schlifkowitz wrote: > >> Dear list, >> >> following advice from this list and petsc documentation (especially http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex52.c.html ), i have the following code: >> >> //--------- >> ierr = KSPCreate( coml,&kspBA);CHKERRQ(ierr); >> ierr = KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); >> >> ierr = KSPSetType(kspBA,KSPPREONLY);CHKERRQ(ierr); >> >> ierr = KSPGetPC(kspBA, &pcba); >> >> ierr = PCSetType(pcba, PCCHOLESKY);CHKERRQ(ierr); >> >> ierr = PCFactorSetMatSolverPackage(pcba, MATSOLVERPETSC);CHKERRQ(ierr); >> ierr = PCFactorSetUpMatSolverPackage(pcba);CHKERRQ(ierr); >> ierr = MatGetFactor(Kt,"petsc",MAT_FACTOR_CHOLESKY,&DiagM);CHKERRQ(ierr); >> ierr = PCFactorGetMatrix(pcba, &DiagM);CHKERRQ(ierr); >> >> ierr = KSPSetUp(kspBA);CHKERRQ(ierr); >> //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >> ierr = VecDuplicate(fin,&diag);CHKERRQ(ierr); >> ierr = MatGetDiagonal(DiagM,diag);CHKERRQ(ierr); >> //----------- >> >> where Kt is a stiffness matrix. I was hoping that PCFactorGetMatrix() makes DiagM become D from a LDL^T factorization, but to me it seems as if this is not true. >> >> Can anyone clarify? >> >> Regards, >> Uwe From xdliang at gmail.com Mon Dec 12 11:56:33 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Mon, 12 Dec 2011 12:56:33 -0500 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <4EE61C49.60406@gfz-potsdam.de> References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> <28183CC9-7A6B-4F8A-973F-6C4774DDD6E0@mcs.anl.gov> <4EE61C49.60406@gfz-potsdam.de> Message-ID: Hi Alexander, I am just curious how other sparse direct solver (eg. PaStiX) works on your problem. Right now, I only have single RHS, but I will need multi-RHS soon. For my single RHS problem, PaStiX works better for my sparse matrix pattern (helmholtz like). If you tried PaStiX, I would be happy to know the comparison between these solvers. Thanks. Xiangdong On Mon, Dec 12, 2011 at 10:22 AM, Alexander Grayver wrote: > Matt, Barry & Hong, > > Thanks a lot for response. > I may be wrong, but I think that people chose direct solvers for their > robustness and ability to solve for many RHS very quickly. > Surely I don't expect my solution to be sparse and it is not, but at least > in electromagnetics it is pretty common to have sparse RHS. > I am wondering how about other disciplines? > > I don't know about other solvers, but when I use multiple RHS in mumps it > works several times faster then solving for them sequentially. It is just my > experience. > > And I also can imagine that using sparsity is another obvious advantage if > you have sparse RHS. > Now for 1000 rhs and a system of order 10^6 I need 32 gb to use MatMatSolve > with double complex arithmetic. For instance, LDLt factorization of matrix > 10^6 takes ~80 gb. > One option would be to use MatMatSolve(A,X,X) thus reducing memory > consumption twice. > > >>> ?I doubt it is feasible for thousands of rhs. > > > You're right, it is not. It is feasible only if you have sparse RHS. > > Regards, > Alexander > > > On 12.12.2011 15:37, Hong Zhang wrote: >> >> Reading MUMPS manual, mumps only supports centralized dense and sparse >> rhs, >> while solution can be centralized or distributed, but only in the >> dense format (because manual >> does not mention sparse solu). >> >> It would be easy to add MatMatSolve() in Petsc-mumps interface for dense >> rhs, >> but will require vecscatter of petsc distributed dense matrix for rhs >> to mumps centralized >> dense rhs - I doubt it is feasible for thousands of rhs. >> >> Hong >> >> >> On Mon, Dec 12, 2011 at 8:26 AM, Barry Smith ?wrote: >>> >>> ? Alexander, >>> >>> ? ?Do you expect that solving with a sparse right hand side will result >>> in an answer that is also sparse? Normally once you've done the lower >>> triangular solve and then the upper triangular solve won't the result be >>> dense? >>> >>> ? ?Barry >>> >>> On Dec 12, 2011, at 3:10 AM, Alexander Grayver wrote: >>> >>>> Hi Barry, >>>> >>>> Thanks for answer. I should have asked about that from the very >>>> beginning actually. >>>> I get 2 times decrease in performance with 20 RHS, I can imagine how >>>> slow it will be when I will use thousands of them, >>>> moreover it will require a lot of memory to store them as a dense >>>> matrix. >>>> >>>> On 11.12.2011 18:50, Barry Smith wrote: >>>>> >>>>> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >>>>> ? ?One by one >>>> >>>> I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, >>>> PaSTiX support multiple RHS. >>>> >>>>> ? ?We do not handle a sparse right hand side. >>>> >>>> Since I already transferred my code to PETSc anyway, my question now if >>>> it's possible to implement sparse multiple RHS and solve them >>>> simultaneously? >>>> Something like MatMatSolveSparse. I would implement it myself, the >>>> question is there a way to integrate it into PETSc as a patch or something >>>> like that? >>>> >>>> Regards, >>>> Alexander > > From john.fettig at gmail.com Mon Dec 12 12:23:21 2011 From: john.fettig at gmail.com (John Fettig) Date: Mon, 12 Dec 2011 13:23:21 -0500 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: On Mon, Dec 12, 2011 at 10:15 AM, Dominik Szczerba wrote: > > As several others suggested, you can probably do this on your own laptop > or > > workstation if you can reproduce the problem locally. This is how I > debug > > all of my parallel code. > > The problem is there is no problem on my own machine... only there :`( > > OK, giving up, probably I need to either learn their debugger or debug > by foot, commenting out single code lines and adding verbosity.... > > Many thanks and best regards, > Dominik > What about using GNU screen? You add -start_in_debugger -debug_terminal "screen -X -S debug screen" while having a screen open on the node with "screen -S debug". When the processes launch you will get new windows open in the screen session. Of course, you'll need to be running on a single node unless there is some way to attach to a remote screen. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 12 13:05:39 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Dec 2011 13:05:39 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> Message-ID: On Dec 12, 2011, at 11:49 AM, Uwe Schlifkowitz wrote: > I'm continuing Clemens Domanig's work. Currently I'm stuck at his problem as described here: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html > > As far as I understand, getting the diagonal matrix from MUMPS is not possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. We could add support for MatGetDiagonal() for PETSc factored matrices if you really need it, but what are you using it for? Do you want the inertia? We could give that for PETSc factored matrices. Barry > > > Uwe > > On 12.12.2011, at 18:40, Barry Smith wrote: >> >> A "factored" matrix in PETSc whether with a PETSc implementation or an external implementation such as MUMPS, or SuperLU is a weird beast. It is likely in some strange convoluted data structure from which extracting information is not easy. >> >> What information, exactly do you want to obtain from the factored matrix? If we know that we might be able to help you get it. >> >> Barry >> >> On Dec 12, 2011, at 11:35 AM, Uwe Schlifkowitz wrote: >> >>> Dear list, >>> >>> following advice from this list and petsc documentation (especially http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex52.c.html ), i have the following code: >>> >>> //--------- >>> ierr = KSPCreate( coml,&kspBA);CHKERRQ(ierr); >>> ierr = KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); >>> >>> ierr = KSPSetType(kspBA,KSPPREONLY);CHKERRQ(ierr); >>> >>> ierr = KSPGetPC(kspBA, &pcba); >>> >>> ierr = PCSetType(pcba, PCCHOLESKY);CHKERRQ(ierr); >>> >>> ierr = PCFactorSetMatSolverPackage(pcba, MATSOLVERPETSC);CHKERRQ(ierr); >>> ierr = PCFactorSetUpMatSolverPackage(pcba);CHKERRQ(ierr); >>> ierr = MatGetFactor(Kt,"petsc",MAT_FACTOR_CHOLESKY,&DiagM);CHKERRQ(ierr); >>> ierr = PCFactorGetMatrix(pcba, &DiagM);CHKERRQ(ierr); >>> >>> ierr = KSPSetUp(kspBA);CHKERRQ(ierr); >>> //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>> //ierr = MatView(DiagM,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); >>> ierr = VecDuplicate(fin,&diag);CHKERRQ(ierr); >>> ierr = MatGetDiagonal(DiagM,diag);CHKERRQ(ierr); >>> //----------- >>> >>> where Kt is a stiffness matrix. I was hoping that PCFactorGetMatrix() makes DiagM become D from a LDL^T factorization, but to me it seems as if this is not true. >>> >>> Can anyone clarify? >>> >>> Regards, >>> Uwe > > From dominik at itis.ethz.ch Mon Dec 12 13:17:43 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 12 Dec 2011 20:17:43 +0100 Subject: [petsc-users] debugging with no X11 available In-Reply-To: References: Message-ID: Thats a very interesting hint. However, I must be using at least two nodes... Thanks Dominik On Mon, Dec 12, 2011 at 7:23 PM, John Fettig wrote: > On Mon, Dec 12, 2011 at 10:15 AM, Dominik Szczerba > wrote: >> >> > As several others suggested, you can probably do this on your own laptop >> > or >> > workstation if you can reproduce the problem locally. ?This is how I >> > debug >> > all of my parallel code. >> >> The problem is there is no problem on my own machine... only there :`( >> >> OK, giving up, probably I need to either learn their debugger or debug >> by foot, commenting out single code lines and adding verbosity.... >> >> Many thanks and best regards, >> Dominik > > > What about using GNU screen?? You add > > ?-start_in_debugger -debug_terminal "screen -X -S debug screen" > > while having a screen open on the node with "screen -S debug".? When the > processes launch you will get new windows open in the screen session.? Of > course, you'll need to be running on a single node unless there is some way > to attach to a remote screen. > > John > From sebastian.skatulla at googlemail.com Mon Dec 12 18:51:30 2011 From: sebastian.skatulla at googlemail.com (Sebastian Skatulla) Date: Tue, 13 Dec 2011 02:51:30 +0200 Subject: [petsc-users] Additive Schwarz sub-domain definition Message-ID: <1323737490.2044.32.camel@sebastian-skatulla-6.stf.uct.ac.za> Hello I am struggling to solve an incompressible solid mechanics problem using a meshfree code based on Moving Least Square approximations. My knowledge of solvers and preconditioners is quite limited. But it seems that Additive Schwarz or Gram Schmidt preconditioners together with GMRES is a good choice - yet still not good enough in my case. So, I was experimenting with exact (LU) solves in each subdomain which does the trick but the memory need is significant. Now, my question is what purpose serves the function PCASMSetOverlap and could it be used to improve this setup in any way? Thanks. Regards, Sebastian -- Dr Sebastian Skatulla Department of Civil Engineering University of Cape Town Cape Town South Africa From jedbrown at mcs.anl.gov Mon Dec 12 20:18:24 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 12 Dec 2011 18:18:24 -0800 Subject: [petsc-users] Additive Schwarz sub-domain definition In-Reply-To: <1323737490.2044.32.camel@sebastian-skatulla-6.stf.uct.ac.za> References: <1323737490.2044.32.camel@sebastian-skatulla-6.stf.uct.ac.za> Message-ID: On Mon, Dec 12, 2011 at 16:51, Sebastian Skatulla < sebastian.skatulla at googlemail.com> wrote: > I am struggling to solve an incompressible solid mechanics problem using > a meshfree code based on Moving Least Square approximations. My > knowledge of solvers and preconditioners is quite limited. But it seems > that Additive Schwarz or Gram Schmidt preconditioners together with > GMRES is a good choice - yet still not good enough in my case. So, I was > experimenting with exact (LU) solves in each subdomain which does the > trick but the memory need is significant. Now, my question is what > purpose serves the function PCASMSetOverlap and could it be used to > improve this setup in any way? Thanks. > Incompressible problems generally lead to indefinite or saddle-point matrices. These cause two problems for Schwarz preconditioners: 1. Local problems must be stable. A general way to compute stable subdomains is to take a set of dual variables (e.g. all owned pressure dofs) and add all primal variables that are connected to those dual variables (e.g. all displacements that influence those pressures). 2. Local solves are more delicate. Incomplete factorization is unreliable and very order-dependent. Direct solves work, but are more expensive. You can use "block preconditioners" as the local solve (this could be as simple as -sub_pc_type fieldsplit -sub_pc_fieldsplit_type schur -sub_pc_fieldsplit_detect_saddle_point, but you will likely have to learn a bit about PCFieldSplit and how to build block preconditioners for your problem). -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 12 20:18:43 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Dec 2011 20:18:43 -0600 Subject: [petsc-users] Additive Schwarz sub-domain definition In-Reply-To: <1323737490.2044.32.camel@sebastian-skatulla-6.stf.uct.ac.za> References: <1323737490.2044.32.camel@sebastian-skatulla-6.stf.uct.ac.za> Message-ID: <16C1F46F-2167-430A-8D0E-6B657275EBAB@mcs.anl.gov> set overlap determines how much each subdomain overlaps the other domains. Generally the larger the overlap the faster the convergence but then each subproblem is larger so the the end result may be slower code. You should try -pc_asm_overlap 2 to see if that helps convergence. Based on your determination that accurate subdomain solves are needed, it is unlikely using more overlap with a poorer subdomain solve is likely to work well. Incompressible solvers generally have a particular structure that you must leverage in the linear solver or you get slow convergence. You should research how this can be done for your problem. Barry On Dec 12, 2011, at 6:51 PM, Sebastian Skatulla wrote: > Hello > > I am struggling to solve an incompressible solid mechanics problem using > a meshfree code based on Moving Least Square approximations. My > knowledge of solvers and preconditioners is quite limited. But it seems > that Additive Schwarz or Gram Schmidt preconditioners together with > GMRES is a good choice - yet still not good enough in my case. So, I was > experimenting with exact (LU) solves in each subdomain which does the > trick but the memory need is significant. Now, my question is what > purpose serves the function PCASMSetOverlap and could it be used to > improve this setup in any way? Thanks. > > Regards, > Sebastian > -- > Dr Sebastian Skatulla > Department of Civil Engineering > University of Cape Town > Cape Town > South Africa > From thomas.leissing at cstb.fr Tue Dec 13 08:43:07 2011 From: thomas.leissing at cstb.fr (Thomas Leissing) Date: Tue, 13 Dec 2011 15:43:07 +0100 Subject: [petsc-users] PetscInitalize() and console output Message-ID: <1323787387.32812.9.camel@gre-019882> Dear all, Somewhere in my boundary element solver I have a console progress bar which prints to screen the progress of an operation. It looks like this: INFO : 50% [------------> ] The function is very simple: //i is the current iteration number //n is the total number of iterations void progressBar(int x, int n) { if ( x % (n/100) != 0 ) return; // Calculate the ratio of complete-to-incomplete. float ratio = x/(float)n; int c = ratio * 80 // Show the percentage complete. printf("INFO :\t%3d%% [", (int)(ratio*100) ); // Show the load bar. for (int x=0; x"); for (int x=c; x From knepley at gmail.com Tue Dec 13 08:45:59 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 08:45:59 -0600 Subject: [petsc-users] PetscInitalize() and console output In-Reply-To: <1323787387.32812.9.camel@gre-019882> References: <1323787387.32812.9.camel@gre-019882> Message-ID: On Tue, Dec 13, 2011 at 8:43 AM, Thomas Leissing wrote: > ** > Dear all, > > Somewhere in my boundary element solver I have a console progress bar > which prints to screen the progress of an operation. > It looks like this: > INFO : 50% [------------> ] > > The function is very simple: > //i is the current iteration number > //n is the total number of iterations > void progressBar(int x, int n) > { > > if ( x % (n/100) != 0 ) return; > > // Calculate the ratio of complete-to-incomplete. > float ratio = x/(float)n; > int c = ratio * 80 > > // Show the percentage complete. > printf("INFO :\t%3d%% [", (int)(ratio*100) ); > > // Show the load bar. > for (int x=0; x printf("-"); > > printf(">"); > > for (int x=c; x printf(" "); > > // ANSI Control codes to go back to the > // previous line and clear it. > printf("]\n\033[F\033[J"); > } > > > My problem is that the progress bar doesn't print to screen as soon as I > call the PetscInitialize() function (it works fine otherwise). > I tried to play with fflush(stdout) commands without any success. > > Any ideas ? > This sounds like MPI rather than PETSc, since it plays with output. Are you running with mpiexec? Matt > Thanks, > Thomas Leissing > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.leissing at cstb.fr Tue Dec 13 08:55:06 2011 From: thomas.leissing at cstb.fr (Thomas Leissing) Date: Tue, 13 Dec 2011 15:55:06 +0100 Subject: [petsc-users] petsc-users Digest, Vol 36, Issue 46 In-Reply-To: References: Message-ID: <1323788106.32812.10.camel@gre-019882> > This sounds like MPI rather than PETSc, since it plays with output. Are you > running with mpiexec? > Matt Yes I do. I have the same results with and without mpiexec. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Tue Dec 13 08:59:12 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 13 Dec 2011 17:59:12 +0300 Subject: [petsc-users] petsc-users Digest, Vol 36, Issue 46 In-Reply-To: <1323788106.32812.10.camel@gre-019882> References: <1323788106.32812.10.camel@gre-019882> Message-ID: Hi Thomas, That wasn't quite Matt's question, since MPI_Init gets called regardless of whether you use mpiexec. He wants to know if your code breaks if MPI_Init() is called, in which case we can blame this on your MPI implementation and not PETSc :) -A On Tue, Dec 13, 2011 at 5:55 PM, Thomas Leissing wrote: > ** > > > This sounds like MPI rather than PETSc, since it plays with output. Are > you > > running with mpiexec? > > Matt > > Yes I do. I have the same results with and without mpiexec. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 13 09:03:22 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 09:03:22 -0600 Subject: [petsc-users] petsc-users Digest, Vol 36, Issue 46 In-Reply-To: References: <1323788106.32812.10.camel@gre-019882> Message-ID: On Tue, Dec 13, 2011 at 8:59 AM, Aron Ahmadia wrote: > Hi Thomas, > > That wasn't quite Matt's question, since MPI_Init gets called regardless > of whether you use mpiexec. He wants to know if your code breaks if > > MPI_Init() is called, in which case we can blame this on your MPI > implementation and not PETSc :) > Alternatively, configure PETSc using --with-mpi=0 and check. PETSc really has no way of intercepting output or changing buffering, but MPI does. Matt > -A > > > On Tue, Dec 13, 2011 at 5:55 PM, Thomas Leissing wrote: > >> ** >> >> > This sounds like MPI rather than PETSc, since it plays with output. >> Are you >> > running with mpiexec? >> > Matt >> >> Yes I do. I have the same results with and without mpiexec. >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.leissing at cstb.fr Tue Dec 13 09:14:24 2011 From: thomas.leissing at cstb.fr (Thomas Leissing) Date: Tue, 13 Dec 2011 16:14:24 +0100 Subject: [petsc-users] PetscInitalize() and console output In-Reply-To: Message-ID: <1323789264.32812.16.camel@gre-019882> Indeed, a call to MPI_Init() breaks the printf()... I'll look on the MPI side. Thanks for your help. (Sorry for my last answer, it had the wrong subject...) Thomas Leissing -------------- next part -------------- An HTML attachment was scrubbed... URL: From uwe.schlifkowitz at uibk.ac.at Tue Dec 13 09:36:11 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Tue, 13 Dec 2011 16:36:11 +0100 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> Message-ID: <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> On 12.12.2011, at 20:05, Barry Smith wrote: > > On Dec 12, 2011, at 11:49 AM, Uwe Schlifkowitz wrote: > >> I'm continuing Clemens Domanig's work. Currently I'm stuck at his problem as described here: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html >> >> As far as I understand, getting the diagonal matrix from MUMPS is not possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. > > We could add support for MatGetDiagonal() for PETSc factored matrices if you really need it, but what are you using it for? Do you want the inertia? We could give that for PETSc factored matrices. > > > Barry > K is an element stiffness matrix. K is related to displacement d and force f by K d = f. Since K is not invertible, a solution can be found by means of LDL^T decomposition where K = LDL^T . Here L is the lower triangular matrix and D is the diagonal matrix, which is what I am looking for. The number of negative values in D correspond to the number of negative eigenvalues of K and serve as an indicator of the occurrence of branching points. I am not sure if this has to do with inertia, but I am fairly new to all of this so suggestions are very welcome. Uwe From knepley at gmail.com Tue Dec 13 09:43:17 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 09:43:17 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> Message-ID: On Tue, Dec 13, 2011 at 9:36 AM, Uwe Schlifkowitz < uwe.schlifkowitz at uibk.ac.at> wrote: > On 12.12.2011, at 20:05, Barry Smith wrote: > > > > On Dec 12, 2011, at 11:49 AM, Uwe Schlifkowitz wrote: > > > >> I'm continuing Clemens Domanig's work. Currently I'm stuck at his > problem as described here: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html > >> > >> As far as I understand, getting the diagonal matrix from MUMPS is not > possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. > > > > We could add support for MatGetDiagonal() for PETSc factored matrices > if you really need it, but what are you using it for? Do you want the > inertia? We could give that for PETSc factored matrices. > > > > > > Barry > > > > K is an element stiffness matrix. K is related to displacement d and force > f by K d = f. Since K is not invertible, a solution can be found by means > of LDL^T decomposition where K = LDL^T . > > Here L is the lower triangular matrix and D is the diagonal matrix, which > is what I am looking for. The number of negative values in D correspond to > the number of negative eigenvalues of K and serve as an indicator of the > occurrence of branching points. > > I am not sure if this has to do with inertia, but I am fairly new to all > of this so suggestions are very welcome. Yes, it looks like what you want http://en.wikipedia.org/wiki/Sylvester's_law_of_inertia Matt > > Uwe -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Tue Dec 13 10:13:00 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 13 Dec 2011 11:13:00 -0500 Subject: [petsc-users] PetscPrintf Message-ID: Hello everyone, I am using PetscPrintf(PETSC_COMM_WORLD,.....) to print out some information. When I run the program on multiple processors, some of these calls are printed out multiple times, while other calls are printed out once. Should it be printed only once if I use PETSC_COMM_WORLD? Thanks. Xiangdong PS. I am using petsc-dev. My code works fine on real petsc-3.1-p8. I just switched to complex petsc-dev and see this problem. From knepley at gmail.com Tue Dec 13 10:14:36 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 10:14:36 -0600 Subject: [petsc-users] PetscPrintf In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 10:13 AM, Xiangdong Liang wrote: > Hello everyone, > > I am using PetscPrintf(PETSC_COMM_WORLD,.....) to print out some > information. When I run the program on multiple processors, some of > these calls are printed out multiple times, while other calls are > printed out once. Should it be printed only once if I use > PETSC_COMM_WORLD? Thanks. The print is executed only on rank 0 of the communicator provided. Matt > > Xiangdong > > PS. I am using petsc-dev. My code works fine on real petsc-3.1-p8. I > just switched to complex petsc-dev and see this problem. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Tue Dec 13 10:19:29 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 13 Dec 2011 11:19:29 -0500 Subject: [petsc-users] PetscPrintf In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 11:14 AM, Matthew Knepley wrote: > On Tue, Dec 13, 2011 at 10:13 AM, Xiangdong Liang wrote: >> >> Hello everyone, >> >> I am using PetscPrintf(PETSC_COMM_WORLD,.....) to print out some >> information. When I run the program on multiple processors, some of >> these calls are printed out multiple times, while other calls are >> printed out once. Should it be printed only once if I use >> PETSC_COMM_WORLD? Thanks. > > > The print is executed only on rank 0 of the communicator provided. I agree. This implies that the message can be printed out at most once. Do you have any hints why I saw the message printed out multiple times? Thanks. Xiangdong > > ? ?Matt > >> >> >> Xiangdong >> >> PS. I am using petsc-dev. My code works fine on real petsc-3.1-p8. I >> just switched to complex petsc-dev and see this problem. > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Dec 13 10:24:37 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 10:24:37 -0600 Subject: [petsc-users] PetscPrintf In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 10:19 AM, Xiangdong Liang wrote: > On Tue, Dec 13, 2011 at 11:14 AM, Matthew Knepley > wrote: > > On Tue, Dec 13, 2011 at 10:13 AM, Xiangdong Liang > wrote: > >> > >> Hello everyone, > >> > >> I am using PetscPrintf(PETSC_COMM_WORLD,.....) to print out some > >> information. When I run the program on multiple processors, some of > >> these calls are printed out multiple times, while other calls are > >> printed out once. Should it be printed only once if I use > >> PETSC_COMM_WORLD? Thanks. > > > > > > The print is executed only on rank 0 of the communicator provided. > > I agree. This implies that the message can be printed out at most > once. Do you have any hints why I saw the message printed out multiple > times? Thanks. > Maybe 1) You have a bug in your code 2) You ran with the wrong mpiexec, so it just ran 2 copies Matt > Xiangdong > > > > > > Matt > > > >> > >> > >> Xiangdong > >> > >> PS. I am using petsc-dev. My code works fine on real petsc-3.1-p8. I > >> just switched to complex petsc-dev and see this problem. > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Dec 13 10:24:59 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 13 Dec 2011 10:24:59 -0600 (CST) Subject: [petsc-users] PetscPrintf In-Reply-To: References: Message-ID: On Tue, 13 Dec 2011, Xiangdong Liang wrote: > > The print is executed only on rank 0 of the communicator provided. > > I agree. This implies that the message can be printed out at most > once. Do you have any hints why I saw the message printed out multiple > times? Perhaps bug with usage of PetscPrintf()? If not - send us a test code [that we can compile/run - not a snippet] that demonstrates this problem - to reproduce. Satish From uwe.schlifkowitz at uibk.ac.at Tue Dec 13 10:48:15 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Tue, 13 Dec 2011 17:48:15 +0100 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> Message-ID: <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> On 13.12.2011, at 16:43, Matthew Knepley wrote: > On Tue, Dec 13, 2011 at 9:36 AM, Uwe Schlifkowitz < > uwe.schlifkowitz at uibk.ac.at> wrote: > >> On 12.12.2011, at 20:05, Barry Smith wrote: >> problem as described here: >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html >> possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. >> if you really need it, but what are you using it for? Do you want the >> inertia? We could give that for PETSc factored matrices. >> >> K is an element stiffness matrix. K is related to displacement d and force >> f by K d = f. Since K is not invertible, a solution can be found by means >> of LDL^T decomposition where K = LDL^T . >> >> Here L is the lower triangular matrix and D is the diagonal matrix, which >> is what I am looking for. The number of negative values in D correspond to >> the number of negative eigenvalues of K and serve as an indicator of the >> occurrence of branching points. >> >> I am not sure if this has to do with inertia, but I am fairly new to all >> of this so suggestions are very welcome. > > > Yes, it looks like what you want > > http://en.wikipedia.org/wiki/Sylvester's_law_of_inertia > > Matt > I found MatGetInertia in petsc's documentation and i am not sure how to use it. I found an old example from petsc-2.3.1 (http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html ), but things seem to have changed since that version. Can you give an example? > >> >> Uwe From knepley at gmail.com Tue Dec 13 10:56:42 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 10:56:42 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> Message-ID: On Tue, Dec 13, 2011 at 10:48 AM, Uwe Schlifkowitz < uwe.schlifkowitz at uibk.ac.at> wrote: > On 13.12.2011, at 16:43, Matthew Knepley wrote: > > On Tue, Dec 13, 2011 at 9:36 AM, Uwe Schlifkowitz < > > uwe.schlifkowitz at uibk.ac.at> wrote: > > > >> On 12.12.2011, at 20:05, Barry Smith wrote: > >> problem as described here: > >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-August/009571.html > >> possible, so Clemens (and I) resort to MATSOLVERPETSC to obtain D. > >> if you really need it, but what are you using it for? Do you want the > >> inertia? We could give that for PETSc factored matrices. > >> > >> K is an element stiffness matrix. K is related to displacement d and > force > >> f by K d = f. Since K is not invertible, a solution can be found by > means > >> of LDL^T decomposition where K = LDL^T . > >> > >> Here L is the lower triangular matrix and D is the diagonal matrix, > which > >> is what I am looking for. The number of negative values in D correspond > to > >> the number of negative eigenvalues of K and serve as an indicator of the > >> occurrence of branching points. > >> > >> I am not sure if this has to do with inertia, but I am fairly new to all > >> of this so suggestions are very welcome. > > > > > > Yes, it looks like what you want > > > > http://en.wikipedia.org/wiki/Sylvester's_law_of_inertia > > > > Matt > > > > I found MatGetInertia in petsc's documentation and i am not sure how to > use it. I found an old example from petsc-2.3.1 ( > http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html), but things seem to have changed since that version. Can you give an > example? > It works exactly as in this example. Matt > > > >> > >> Uwe > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Tue Dec 13 11:21:49 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 13 Dec 2011 12:21:49 -0500 Subject: [petsc-users] PetscPrintf In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 11:24 AM, Matthew Knepley wrote: > On Tue, Dec 13, 2011 at 10:19 AM, Xiangdong Liang wrote: >> >> On Tue, Dec 13, 2011 at 11:14 AM, Matthew Knepley >> wrote: >> > On Tue, Dec 13, 2011 at 10:13 AM, Xiangdong Liang >> > wrote: >> >> >> >> Hello everyone, >> >> >> >> I am using PetscPrintf(PETSC_COMM_WORLD,.....) to print out some >> >> information. When I run the program on multiple processors, some of >> >> these calls are printed out multiple times, while other calls are >> >> printed out once. Should it be printed only once if I use >> >> PETSC_COMM_WORLD? Thanks. >> > >> > >> > The print is executed only on rank 0 of the communicator provided. >> >> I agree. This implies that the message can be printed out at most >> once. Do you have any hints why I saw the message printed out multiple >> times? Thanks. > > > Maybe > > 1) You have a bug in your code > > 2) You ran with the wrong mpiexec, so it just ran 2 copies You're right. I use the wrong mpiexec. When I compile petsc-dev, I use download-mpich while the default mpiexec in our system is openmpi. If I use $PETSC_DIR/bin/petscmpiexec, the problem is fixed. Thanks, Matt and Satish. Xiangdong > > ? Matt > >> >> Xiangdong >> >> >> > >> > ? ?Matt >> > >> >> >> >> >> >> Xiangdong >> >> >> >> PS. I am using petsc-dev. My code works fine on real petsc-3.1-p8. I >> >> just switched to complex petsc-dev and see this problem. >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From hzhang at mcs.anl.gov Tue Dec 13 12:03:47 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 13 Dec 2011 12:03:47 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> Message-ID: > I found MatGetInertia in petsc's documentation and i am not sure how to use it. I found an old example from petsc-2.3.1 (http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html ), but things seem to have changed since that version. Can you give an example? petsc-3.2/src/ksp/ksp/examples/tests/ex33.c and ex36.c Hong From uwe.schlifkowitz at uibk.ac.at Tue Dec 13 12:51:24 2011 From: uwe.schlifkowitz at uibk.ac.at (Uwe Schlifkowitz) Date: Tue, 13 Dec 2011 19:51:24 +0100 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> Message-ID: <893220C7-B6D6-470F-B416-F222F845EA64@uibk.ac.at> On 13.12.2011, at 19:03, Hong Zhang wrote: >> I found MatGetInertia in petsc's documentation and i am not sure how to use it. I found an old example from petsc-2.3.1 (http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html ), but things seem to have changed since that version. Can you give an example? > > petsc-3.2/src/ksp/ksp/examples/tests/ex33.c and ex36.c > > Hong Thank you. I already looked into ex33. My input matrix however is not in sbaij format: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Mat type seqaij! How do i convert the matrix to sbaij? I tried MatConvert as in ex36.c, but after that the matrix is not symmetric anymore. Uwe From knepley at gmail.com Tue Dec 13 12:54:34 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 12:54:34 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: <893220C7-B6D6-470F-B416-F222F845EA64@uibk.ac.at> References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> <893220C7-B6D6-470F-B416-F222F845EA64@uibk.ac.at> Message-ID: On Tue, Dec 13, 2011 at 12:51 PM, Uwe Schlifkowitz < uwe.schlifkowitz at uibk.ac.at> wrote: > On 13.12.2011, at 19:03, Hong Zhang wrote: > >> I found MatGetInertia in petsc's documentation and i am not sure how to > use it. I found an old example from petsc-2.3.1 ( > http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html), but things seem to have changed since that version. Can you give an > example? > > > > petsc-3.2/src/ksp/ksp/examples/tests/ex33.c and ex36.c > > > > Hong > > Thank you. I already looked into ex33. My input matrix however is not in > sbaij format: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type seqaij! > > How do i convert the matrix to sbaij? I tried MatConvert as in ex36.c, but > after that the matrix is not symmetric anymore. It is time to read the manual. If you convert to SBAIJ, your matrix is symmetric by definition. Matt > > Uwe -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bogdan at lmn.pub.ro Tue Dec 13 15:16:00 2011 From: bogdan at lmn.pub.ro (Bogdan Dita) Date: Tue, 13 Dec 2011 23:16:00 +0200 Subject: [petsc-users] Optimised vs Debug Message-ID: <1742cc3834e2f262b9bec4b29cf72b62.squirrel@wm.lmn.pub.ro> Hello again, I'm sorry for the delay but due to some technical difficulties I was unable to perform the tests. On Wed, Dec 7, 2011 at 06:31, Bogdan Dita wrote: >> A few weeks ago i posted a question regarding a problem with the >> optimised version of PETSc, what i mean is that I obtained a higher >> solving time with the optimised version then with the debug >> version(~10-15% higher). >> In response to my last post Mr. Jed Brown wrote: > >What do you get from 'mpicc -show'? > >MPICH2 has a tendency to put optimization flags into the wrappers. It's >possible that the "optimized" build of PETSc is effectively using -O2 (from >MPICH2) and -g3 (from PETSc with-debugging=1), which might be faster than >-O2 (from MPICH2) -O (from PETSc with-debugging=0, taking precedence) that >you get when you. You can use PETSc's --COPTFLAGS to set your own >optimization flags for PETSc. > >Let us know if this works. So I reconfigured PETSc as advised, the result was that the time obtained from the new version of PETSc Optimised version(with COPTFLAGS) was about the same as the time from the Debug version(+/- 2 sec). Although is not what I expected this is still an intriguing result for me: on one hand there was definitely something about the flags that the optimised PETSc didn't like, exactly how Mr. J. Brown predicted, but on the other hand this raises new questions. This is the output from 'mpicc -show'. for the cluster: gcc -m64 -O2 -fPIC -Wl,-z,noexecstack -I/usr/include/mpich2-x86_64 -L/usr/lib64/mpich2/lib -lmpich and from my laptop: gcc -I/usr/lib64/mpich2/include -L/usr/lib64/mpich2/lib -Wl,-rpath,/usr/lib64/mpich2/lib -lmpich -lopa -lmpl -lrt -lpthread -------------------------------------- Cosmin-Bogdan DITA From jedbrown at mcs.anl.gov Tue Dec 13 15:19:42 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Dec 2011 13:19:42 -0800 Subject: [petsc-users] Optimised vs Debug In-Reply-To: <1742cc3834e2f262b9bec4b29cf72b62.squirrel@wm.lmn.pub.ro> References: <1742cc3834e2f262b9bec4b29cf72b62.squirrel@wm.lmn.pub.ro> Message-ID: On Tue, Dec 13, 2011 at 13:16, Bogdan Dita wrote: > So I reconfigured PETSc as advised, the result was that the time obtained > from the new version of PETSc Optimised version(with COPTFLAGS) was about > the same as the time from the Debug version(+/- 2 sec). > You have to send output from -log_summary in both cases if you want us to help explain observed performance. > Although is not > what I expected this is still an intriguing result for me: on one hand > there was definitely something about the flags that the optimised PETSc > didn't like, exactly how Mr. J. Brown predicted, but on the other hand > this raises new questions. > > This is the output from 'mpicc -show'. > for the cluster: > gcc -m64 -O2 -fPIC -Wl,-z,noexecstack -I/usr/include/mpich2-x86_64 > -L/usr/lib64/mpich2/lib -lmpich > This one is optimized by default. > > and from my laptop: > gcc -I/usr/lib64/mpich2/include -L/usr/lib64/mpich2/lib > -Wl,-rpath,/usr/lib64/mpich2/lib -lmpich -lopa -lmpl -lrt -lpthread > This one is not. PETSc folks, should we fight MPICH2 by explicitly adding -O0 for debug builds in which CFLAGS are not passed by the user? -------------- next part -------------- An HTML attachment was scrubbed... URL: From keita at cray.com Tue Dec 13 16:07:05 2011 From: keita at cray.com (Keita Teranishi) Date: Tue, 13 Dec 2011 16:07:05 -0600 Subject: [petsc-users] How to create .dat files Message-ID: <5D6E0DF460ACF34C88644E1EA91DCD0D12E2B2B658@CFEXMBX.americas.cray.com> Hi, I am looking for an example program that creates .dat files (matrix&vector data files) for the other PETSc example programs. Could you point the location of the program? Do you have any plan to include any .dat files in the future PETSc distribution? Thanks, ==================================== Keita Teranishi Scientific Library Group Cray Inc. keita at cray.com ====================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 13 16:13:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 13 Dec 2011 16:13:24 -0600 Subject: [petsc-users] How to create .dat files In-Reply-To: <5D6E0DF460ACF34C88644E1EA91DCD0D12E2B2B658@CFEXMBX.americas.cray.com> References: <5D6E0DF460ACF34C88644E1EA91DCD0D12E2B2B658@CFEXMBX.americas.cray.com> Message-ID: On Dec 13, 2011, at 4:07 PM, Keita Teranishi wrote: > Hi, > > I am looking for an example program that creates .dat files (matrix&vector data files) for the other PETSc example programs. Could you point the location of the program? > > Do you have any plan to include any .dat files in the future PETSc distribution? in share/petsc/datafiles/matrices these are generated with bin/matlab/generatePetscTestFiles.m > > Thanks, > ==================================== > Keita Teranishi > Scientific Library Group > Cray Inc. > keita at cray.com > ====================================== > From hzhang at mcs.anl.gov Tue Dec 13 21:26:55 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 13 Dec 2011 21:26:55 -0600 Subject: [petsc-users] Return value of PCFactorGetMatrix In-Reply-To: References: <2E2C93A9-BEEC-4775-B3EE-6D1985DCDB59@uibk.ac.at> <37A49B82-C399-4740-A64B-3C4A20132F91@uibk.ac.at> <34D24FA7-63F7-43EA-9010-EBE3A50D519B@uibk.ac.at> <34D7566E-1D82-4013-8306-162D0D9D66A7@uibk.ac.at> <893220C7-B6D6-470F-B416-F222F845EA64@uibk.ac.at> Message-ID: Uwe, Petsc supports Cholesky factorization for seqaij matrix format. The factored matrix is in seqsbaij, thus MatGetInertia() works for it. ex33.c is written for sbaij matrix, but also works for aij matrix. You can comment out 2 lines //ierr = MatSetOption(B,MAT_IGNORE_LOWER_TRIANGULAR,PETSC_TRUE);CHKERRQ(ierr); then run it with option '-mat_type aij' and get its inertia. e.g., I get ./ex33 -mat_type seqaij -sigma 2.0 MatInertia: nneg: 17, nzero: 0, npos: 83 '-mat_view_info' display more detailed info about the matrices being used in the example. Hong On Tue, Dec 13, 2011 at 12:54 PM, Matthew Knepley wrote: > On Tue, Dec 13, 2011 at 12:51 PM, Uwe Schlifkowitz > wrote: >> >> On 13.12.2011, at 19:03, Hong Zhang wrote: >> >> I found MatGetInertia in petsc's documentation and i am not sure how to >> >> use it. I found an old example from petsc-2.3.1 >> >> (http://www.mcs.anl.gov/petsc/petsc-2.3.1/src/ksp/ksp/examples/tutorials/ex10.c.html >> >> ), but things seem to have changed since that version. Can you give an >> >> example? >> > >> > petsc-3.2/src/ksp/ksp/examples/tests/ex33.c and ex36.c >> > >> > Hong >> >> Thank you. I already looked into ex33. My input matrix however is not in >> sbaij format: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: No support for this operation for this object type! >> [0]PETSC ERROR: Mat type seqaij! >> >> How do i convert the matrix to sbaij? I tried MatConvert as in ex36.c, but >> after that the matrix is not symmetric anymore. > > > It is time to read the manual. If you convert to SBAIJ, your matrix is > symmetric by definition. > > ? ?Matt > >> >> >> Uwe > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From xdliang at gmail.com Tue Dec 13 21:49:40 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 13 Dec 2011 22:49:40 -0500 Subject: [petsc-users] run successfully on 20 processors but failed on 24 processors Message-ID: Hello everyone, I am solving complex Ax=b with PaStix on 20 processors successfully but failed on 24 processors. The relatively error indicated by mat_pastix_verbose becomes "nan" for 24 processors. Where could be wrong? Can someone give me some hints on how I can debug? Thanks. Xiangdong From knepley at gmail.com Tue Dec 13 22:59:09 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Dec 2011 22:59:09 -0600 Subject: [petsc-users] run successfully on 20 processors but failed on 24 processors In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 9:49 PM, Xiangdong Liang wrote: > Hello everyone, > > I am solving complex Ax=b with PaStix on 20 processors successfully > but failed on 24 processors. The relatively error indicated by > mat_pastix_verbose becomes "nan" for 24 processors. Where could be > wrong? Can someone give me some hints on how I can debug? Thanks. First, make sure you did not put any NaNs in your matrix or rhs. Matt > > Xiangdong -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Wed Dec 14 02:02:48 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Wed, 14 Dec 2011 11:32:48 +0330 Subject: [petsc-users] Two Questions about Matrix Preallocation Message-ID: Dear Developers, I am facing with two questions about Block-Sparse Preallocation: 1- I set array "nnz[]" and did: MatSetType ( JAC, MATSEQBAIJ ); MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); MatSeqAIJSetPreallocation( JAC, 0, nnz ); but when I saw log file with -info -mat_view_info, I found this message: [0] MatSetUpPreallocation(): Warning not preallocating matrix storage What this means? Did I correctly preallocate the matrix? 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused with descriptions of the command in man-page: what are "nz" and "nnz[]" in this case? what about number of blocks per row? Thanks bothering too much :-) BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.younas at rug.nl Wed Dec 14 07:07:10 2011 From: m.younas at rug.nl (M.Younas) Date: Wed, 14 Dec 2011 14:07:10 +0100 Subject: [petsc-users] Multigrid as preconditioner In-Reply-To: <7430cf77cfd30.4ee89f56@rug.nl> References: <7610fccbceafa.4ee89cbe@rug.nl> <7400b350c80db.4ee89cfa@rug.nl> <7430aadfcf44a.4ee89d37@rug.nl> <76e0ed8ecb337.4ee89d73@rug.nl> <74008489ccc28.4ee89daf@rug.nl> <7590970ecf516.4ee89dec@rug.nl> <73b09d6dc8cd6.4ee89e28@rug.nl> <7430ee0ecacf8.4ee89e64@rug.nl> <7620f9ddc9f8d.4ee89ea1@rug.nl> <76e087b8cd510.4ee89edd@rug.nl> <7610b6c9cefa1.4ee89f19@rug.nl> <7430cf77cfd30.4ee89f56@rug.nl> Message-ID: <73f0f8e5cc491.4ee8ad8e@rug.nl> ?Hi, PETSc, Team, I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? Kind regards, Younas RUG, Groningen, The Netherlands. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.bodner at unil.ch Wed Dec 14 08:00:03 2011 From: robert.bodner at unil.ch (robert) Date: Wed, 14 Dec 2011 15:00:03 +0100 Subject: [petsc-users] save vector to bin-file at each time step Message-ID: <4EE8ABE3.5050108@unil.ch> Hello, I am quite new to petsc and therefore my question might be quite trivial. I have a 1D finite difference code and want to store several vectors in a binary: ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); ierr = VecView(u,viewer); CHKERRQ(ierr); ierr = VecView(rhs,viewer);CHKERRQ(ierr); However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. In further application I would like to use something similar but putting out the solution vector every timestep. Could someone just provide me some sample code? Thanks, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.younas at rug.nl Wed Dec 14 08:04:47 2011 From: m.younas at rug.nl (M.Younas) Date: Wed, 14 Dec 2011 15:04:47 +0100 Subject: [petsc-users] AMG as preconditioner Message-ID: <7420f1bdca4d2.4ee8bb0f@rug.nl> ?Hi, PETSc, Team, I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? Kind regards, Younas RUG, Groningen, The Netherlands. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 14 08:22:33 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 14 Dec 2011 08:22:33 -0600 Subject: [petsc-users] Two Questions about Matrix Preallocation In-Reply-To: References: Message-ID: On Dec 14, 2011, at 2:02 AM, behzad baghapour wrote: > Dear Developers, > > I am facing with two questions about Block-Sparse Preallocation: > > 1- I set array "nnz[]" and did: > > MatSetType ( JAC, MATSEQBAIJ ); > MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); > MatSeqAIJSetPreallocation( JAC, 0, nnz ); > > but when I saw log file with -info -mat_view_info, I found this message: > > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage Since you are using BAIJ matrix type you need to also call MatSeqBAIJSetPreallocation() > > What this means? Did I correctly preallocate the matrix? > > 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused with descriptions of the command in man-page: > > what are "nz" and "nnz[]" in this case? > > what about number of blocks per row? nz is the maximum number of blocks per row, just like with other matrix formats or you can set nnz[] which for each row is the number of blocks from the diagonal block to the end of the row. For example with a block size of one the matrix 1 0 2 0 3 4 2 4 5 would have a nnz[] = 2,2,1 Barry > > Thanks bothering too much :-) > BehZad > From bsmith at mcs.anl.gov Wed Dec 14 08:26:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 14 Dec 2011 08:26:27 -0600 Subject: [petsc-users] Multigrid as preconditioner In-Reply-To: <73f0f8e5cc491.4ee8ad8e@rug.nl> References: <7610fccbceafa.4ee89cbe@rug.nl> <7400b350c80db.4ee89cfa@rug.nl> <7430aadfcf44a.4ee89d37@rug.nl> <76e0ed8ecb337.4ee89d73@rug.nl> <74008489ccc28.4ee89daf@rug.nl> <7590970ecf516.4ee89dec@rug.nl> <73b09d6dc8cd6.4ee89e28@rug.nl> <7430ee0ecacf8.4ee89e64@rug.nl> <7620f9ddc9f8d.4ee89ea1@rug.nl> <76e087b8cd510.4ee89edd@rug.nl> <7610b6c9cefa1.4ee89f19@rug.nl> <7430cf77cfd30.4ee89f56@rug.nl> <73f0f8e5cc491.4ee8ad8e@rug.nl> Message-ID: <2770E217-0CC1-4E85-AB40-35B00F705D53@mcs.anl.gov> On Dec 14, 2011, at 7:07 AM, M.Younas wrote: > Hi, PETSc, Team, > > I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? By default PETSc uses ONE v-cycle of the BoomerAMG as a preconditioner for GMRES. With the option -ksp_type richardson then PETSc is using a bunch of BoomerAMG v-cycles until the convergence criteria is satisfied without using a GMRES Barry Note that a preconditioner can be any stationary linear iterative scheme. That is any linear operation that approximately solves A x = b, no matter how complicated the stationary linear iterative scheme is it can be used as a preconditioner. > > Kind regards, > > Younas > > RUG, Groningen, The Netherlands. From bsmith at mcs.anl.gov Wed Dec 14 08:32:58 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 14 Dec 2011 08:32:58 -0600 Subject: [petsc-users] save vector to bin-file at each time step In-Reply-To: <4EE8ABE3.5050108@unil.ch> References: <4EE8ABE3.5050108@unil.ch> Message-ID: <76717B20-13EF-4FD2-BF50-CC12E9E87842@mcs.anl.gov> On Dec 14, 2011, at 8:00 AM, robert wrote: > Hello, > > I am quite new to petsc and therefore my question might be quite trivial. > > I have a 1D finite difference code and want to store several vectors in a binary: > > ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); > ierr = VecView(u,viewer); CHKERRQ(ierr); > ierr = VecView(rhs,viewer);CHKERRQ(ierr); > > However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. 1) Did you call PetscViewerDestroy() before the end of the program? If not called all the data may not be flushed into the file. 2) How are you reading the vectors into Matlab? For example [u,rhs] = PetscBinaryRead('filename') will read in two vectors. If you want to call PetscBinaryRead() repeatedly to read more information than read the help for PetscBinaryRead() carefully, you cannot simply call u = PetscBinaryRead('filename'); rhs = PetscBinaryRead('filename') and hope to get the second vector For example: fd = PetscOpenFile('filename'); u = PetscBinaryRead(fd); % get first vector rhs = PetscBinaryRead(fd); % get second vector Barry > > In further application I would like to use something similar but putting out the solution vector every timestep. > > Could someone just provide me some sample code? > > Thanks, > Robert > > From robert.bodner at unil.ch Wed Dec 14 08:44:39 2011 From: robert.bodner at unil.ch (robert) Date: Wed, 14 Dec 2011 15:44:39 +0100 Subject: [petsc-users] save vector to bin-file at each time step In-Reply-To: <76717B20-13EF-4FD2-BF50-CC12E9E87842@mcs.anl.gov> References: <4EE8ABE3.5050108@unil.ch> <76717B20-13EF-4FD2-BF50-CC12E9E87842@mcs.anl.gov> Message-ID: <4EE8B657.4090506@unil.ch> On 12/14/2011 03:32 PM, Barry Smith wrote: > On Dec 14, 2011, at 8:00 AM, robert wrote: > >> Hello, >> >> I am quite new to petsc and therefore my question might be quite trivial. >> >> I have a 1D finite difference code and want to store several vectors in a binary: >> >> ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); >> ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); >> ierr = VecView(u,viewer); CHKERRQ(ierr); >> ierr = VecView(rhs,viewer);CHKERRQ(ierr); >> >> However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. >> >> >> 2) How are you reading the vectors into Matlab? For example [u,rhs] = PetscBinaryRead('filename') will read in two vectors. If you want to call PetscBinaryRead() repeatedly to read more information than ... Hi, thank you for your fast reply. It's as easy as that and I spent hours ... :-) >> >> In further application I would like to use something similar but putting out the solution vector every timestep. >> >> Could someone just provide me some sample code? >> >> Thanks, >> Robert >> >> > From m.younas at rug.nl Wed Dec 14 09:45:32 2011 From: m.younas at rug.nl (M.Younas) Date: Wed, 14 Dec 2011 16:45:32 +0100 Subject: [petsc-users] multigrid as preconditioner--message 8 In-Reply-To: <7660c2e2cf011.4ee8c479@rug.nl> References: <7670c115cb65f.4ee8c30e@rug.nl> <7680a31cca802.4ee8c34a@rug.nl> <7650c7bfcd9a2.4ee8c387@rug.nl> <7610a256cb514.4ee8c3c3@rug.nl> <75c093cbca0b7.4ee8c400@rug.nl> <7690abadcc205.4ee8c43c@rug.nl> <7660c2e2cf011.4ee8c479@rug.nl> Message-ID: <7660d523c8ef3.4ee8d2ac@rug.nl> Dear Barry, Does BoomeAMG precondition (one v-cycle) the system after every iteration of GMRES(m) OR it precondition the linear system once only and then GMRES(m) iterations until convergence is reached? Kind regards, Younas ? On 14-12-11, petsc-users-request at mcs.anl.gov wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > ?? 1.? run successfully on 20 processors but failed on 24 > ????? processors (Xiangdong Liang) > ?? 2. Re:? run successfully on 20 processors but failed on 24 > ????? processors (Matthew Knepley) > ?? 3.? Two Questions about Matrix Preallocation (behzad baghapour) > ?? 4.? Multigrid as preconditioner (M.Younas) > ?? 5.? save vector to bin-file at each time step (robert) > ?? 6.? AMG as preconditioner (M.Younas) > ?? 7. Re:? Two Questions about Matrix Preallocation (Barry Smith) > ?? 8. Re:? Multigrid as preconditioner (Barry Smith) > ?? 9. Re:? save vector to bin-file at each time step (Barry Smith) > ? 10. Re:? save vector to bin-file at each time step (robert) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Dec 2011 22:49:40 -0500 > From: Xiangdong Liang > Subject: [petsc-users] run successfully on 20 processors but failed on > 24 processors > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Hello everyone, > > I am solving complex Ax=b with PaStix on 20 processors successfully > but failed on 24 processors. The relatively error indicated by > mat_pastix_verbose becomes "nan"? for 24 processors. Where could be > wrong? Can someone give me some hints on how I can debug? Thanks. > > Xiangdong > > > ------------------------------ > > Message: 2 > Date: Tue, 13 Dec 2011 22:59:09 -0600 > From: Matthew Knepley > Subject: Re: [petsc-users] run successfully on 20 processors but > failed on 24 processors > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Tue, Dec 13, 2011 at 9:49 PM, Xiangdong Liang wrote: > > > Hello everyone, > > > > I am solving complex Ax=b with PaStix on 20 processors successfully > > but failed on 24 processors. The relatively error indicated by > > mat_pastix_verbose becomes "nan"? for 24 processors. Where could be > > wrong? Can someone give me some hints on how I can debug? Thanks. > > > First, make sure you did not put any NaNs in your matrix or rhs. > > ?? Matt > > > > > > Xiangdong > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 3 > Date: Wed, 14 Dec 2011 11:32:48 +0330 > From: behzad baghapour > Subject: [petsc-users] Two Questions about Matrix Preallocation > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Dear Developers, > > I am facing with two questions about Block-Sparse Preallocation: > > 1- I set array "nnz[]" and did: > > ??? MatSetType ( JAC, MATSEQBAIJ ); > ??? MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); > ??? MatSeqAIJSetPreallocation( JAC, 0, nnz ); > > but when I saw log file with -info -mat_view_info, I found this message: > > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > > What this means? Did I correctly preallocate the matrix? > > 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused > with descriptions of the command in man-page: > > what are "nz" and "nnz[]" in this case? > > what about number of blocks per row? > > Thanks bothering too much :-) > BehZad > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 4 > Date: Wed, 14 Dec 2011 14:07:10 +0100 > From: "M.Younas" > Subject: [petsc-users] Multigrid as preconditioner > To: petsc-users at mcs.anl.gov > Message-ID: <73f0f8e5cc491.4ee8ad8e at rug.nl> > Content-Type: text/plain; charset="iso-8859-1" > > ?Hi, PETSc, Team, > > I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? > > Kind regards, > > Younas > > RUG, Groningen, The Netherlands. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 5 > Date: Wed, 14 Dec 2011 15:00:03 +0100 > From: robert > Subject: [petsc-users] save vector to bin-file at each time step > To: petsc-users at mcs.anl.gov > Message-ID: <4EE8ABE3.5050108 at unil.ch> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > Hello, > > I am quite new to petsc and therefore my question might be quite trivial. > > I have a 1D finite difference code and want to store several vectors in > a binary: > > ???? ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); > ???? ierr = > PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); > CHKERRQ(ierr); > ???? ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); > ???? ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); > ???? ierr = VecView(u,viewer); CHKERRQ(ierr); > ???? ierr = VecView(rhs,viewer);CHKERRQ(ierr); > > However, when I read the file with matlab (PetscBinaryRead.m) I only get > the first vector. > > In further application I would like to use something similar but putting > out the solution vector every timestep. > > Could someone just provide me some sample code? > > Thanks, > Robert > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 6 > Date: Wed, 14 Dec 2011 15:04:47 +0100 > From: "M.Younas" > Subject: [petsc-users] AMG as preconditioner > To: petsc-users at mcs.anl.gov > Message-ID: <7420f1bdca4d2.4ee8bb0f at rug.nl> > Content-Type: text/plain; charset="iso-8859-1" > > ?Hi, PETSc, Team, > > I have > solved a non-symmetric Poisson problem using GMRES(m) as solver and > BoomerAMG (from Hypre) as preconditioner. But I don't understand how the > ?BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the > ?procedure? Does it work like BoomerAMG precondition the liner system > first (once only) and then GMRES(m) is being used for stopping > criterion? > > Kind regards, > > Younas > > RUG, Groningen, The Netherlands. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 7 > Date: Wed, 14 Dec 2011 08:22:33 -0600 > From: Barry Smith > Subject: Re: [petsc-users] Two Questions about Matrix Preallocation > To: PETSc users list , behzad baghapour > > Message-ID: > Content-Type: text/plain; charset=us-ascii > > > On Dec 14, 2011, at 2:02 AM, behzad baghapour wrote: > > > Dear Developers, > > > > I am facing with two questions about Block-Sparse Preallocation: > > > > 1- I set array "nnz[]" and did: > > > >???? MatSetType ( JAC, MATSEQBAIJ ); > >???? MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt );??? > >???? MatSeqAIJSetPreallocation( JAC, 0, nnz ); > > > > but when I saw log file with -info -mat_view_info, I found this message: > > > > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > > ?? Since you are using BAIJ matrix type you need to also call MatSeqBAIJSetPreallocation() > > > > > What this means? Did I correctly preallocate the matrix? > > > > 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused with descriptions of the command in man-page: > > > > what are "nz" and "nnz[]" in this case? > > > > what about number of blocks per row? > > ?? nz is the maximum number of blocks per row, just like with other matrix formats or you can set nnz[] which for each row is the number of blocks from the diagonal block to the end of the row.? For example with a block size of one the matrix > > ? 1?? 0? 2 > ?? 0? 3 4 > ?? 2? 4? 5 > > would have a nnz[] = 2,2,1 > > ?Barry > > > > > Thanks bothering too much :-) > > BehZad > > > > > > ------------------------------ > > Message: 8 > Date: Wed, 14 Dec 2011 08:26:27 -0600 > From: Barry Smith > Subject: Re: [petsc-users] Multigrid as preconditioner > To: PETSc users list > Message-ID: <2770E217-0CC1-4E85-AB40-35B00F705D53 at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > On Dec 14, 2011, at 7:07 AM, M.Younas wrote: > > >? Hi, PETSc, Team, > > > > I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? > > ?? By default PETSc uses ONE v-cycle of the BoomerAMG as a preconditioner for GMRES. > > ?? With the option -ksp_type richardson then PETSc is using a bunch of BoomerAMG v-cycles until the convergence criteria is satisfied without using a GMRES > > ?? Barry > > Note that a preconditioner can be any stationary linear iterative scheme. That is any linear operation that approximately solves A x = b, no matter how complicated the stationary linear iterative scheme is it can be used as a preconditioner. > > > > > > Kind regards, > > > > Younas > > > > RUG, Groningen, The Netherlands. > > > > ------------------------------ > > Message: 9 > Date: Wed, 14 Dec 2011 08:32:58 -0600 > From: Barry Smith > Subject: Re: [petsc-users] save vector to bin-file at each time step > To: PETSc users list > Message-ID: <76717B20-13EF-4FD2-BF50-CC12E9E87842 at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > On Dec 14, 2011, at 8:00 AM, robert wrote: > > > Hello, > > > > I am quite new to petsc and therefore my question might be quite trivial. > > > > I have a 1D finite difference code and want to store several vectors in a binary: > > > >???? ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); > >???? ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); > >???? ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); > >???? ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); > >???? ierr = VecView(u,viewer); CHKERRQ(ierr); > >???? ierr = VecView(rhs,viewer);CHKERRQ(ierr); > > > > However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. > > ?? 1) Did you call PetscViewerDestroy() before the end of the program? If not called all the data may not be flushed into the file. > > ?? 2) How are you reading the vectors into Matlab? For example?? [u,rhs] = PetscBinaryRead('filename') will read in two vectors.? If you want to call PetscBinaryRead() repeatedly to read more information than read the help for PetscBinaryRead() carefully, you cannot simply call u = PetscBinaryRead('filename'); rhs = PetscBinaryRead('filename') and hope to get the second vector > > ? For example: fd = PetscOpenFile('filename'); > ??????????????? u = PetscBinaryRead(fd);?? % get first vector > ??????????????? rhs = PetscBinaryRead(fd);? % get second vector > > ? Barry > > > > > > In further application I would like to use something similar but putting out the solution vector every timestep. > > > > Could someone just provide me some sample code? > > > > Thanks, > > Robert > > > > > > > > ------------------------------ > > Message: 10 > Date: Wed, 14 Dec 2011 15:44:39 +0100 > From: robert > Subject: Re: [petsc-users] save vector to bin-file at each time step > To: PETSc users list > Message-ID: <4EE8B657.4090506 at unil.ch> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 12/14/2011 03:32 PM, Barry Smith wrote: > > On Dec 14, 2011, at 8:00 AM, robert wrote: > > > >> Hello, > >> > >> I am quite new to petsc and therefore my question might be quite trivial. > >> > >> I have a 1D finite difference code and want to store several vectors in a binary: > >> > >>????? ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); > >>????? ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); > >>????? ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); > >>????? ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); > >>????? ierr = VecView(u,viewer); CHKERRQ(ierr); > >>????? ierr = VecView(rhs,viewer);CHKERRQ(ierr); > >> > >> However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. > >> > >> > >>???? 2) How are you reading the vectors into Matlab? For example?? [u,rhs] = PetscBinaryRead('filename') will read in two vectors.? If you want to call PetscBinaryRead() repeatedly to read more information than ... > > Hi, > > thank you for your fast reply. It's as easy as that and I spent hours > ... :-) > >> > > > >> In further application I would like to use something similar but putting out the solution vector every timestep. > >> > >> Could someone just provide me some sample code? > >> > >> Thanks, > >> Robert > >> > >> > > > > > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 36, Issue 50 > ******************************************* > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 14 10:39:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 14 Dec 2011 10:39:41 -0600 Subject: [petsc-users] multigrid as preconditioner--message 8 In-Reply-To: <7660d523c8ef3.4ee8d2ac@rug.nl> References: <7670c115cb65f.4ee8c30e@rug.nl> <7680a31cca802.4ee8c34a@rug.nl> <7650c7bfcd9a2.4ee8c387@rug.nl> <7610a256cb514.4ee8c3c3@rug.nl> <75c093cbca0b7.4ee8c400@rug.nl> <7690abadcc205.4ee8c43c@rug.nl> <7660c2e2cf011.4ee8c479@rug.nl> <7660d523c8ef3.4ee8d2ac@rug.nl> Message-ID: <8A11274A-AE8E-41AE-B9AA-A75590A29072@mcs.anl.gov> On Dec 14, 2011, at 9:45 AM, M.Younas wrote: > Dear Barry, > > Does BoomeAMG precondition (one v-cycle) the system after every iteration of GMRES(m) It is not "after" every iteration of GMRES it is "as part of each iteration" of preconditioned GMRES. > OR > it precondition the linear system once only and then GMRES(m) iterations until convergence is reached? > No. You may want to read up on Krylov methods in a numerical analysis book. Barry > Kind regards, Younas > > > On 14-12-11, petsc-users-request at mcs.anl.gov wrote: >> Send petsc-users mailing list submissions to >> petsc-users at mcs.anl.gov >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users >> or, via email, send a message with subject or body 'help' to >> petsc-users-request at mcs.anl.gov >> >> You can reach the person managing the list at >> petsc-users-owner at mcs.anl.gov >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of petsc-users digest..." >> >> >> Today's Topics: >> >> 1. run successfully on 20 processors but failed on 24 >> processors (Xiangdong Liang) >> 2. Re: run successfully on 20 processors but failed on 24 >> processors (Matthew Knepley) >> 3. Two Questions about Matrix Preallocation (behzad baghapour) >> 4. Multigrid as preconditioner (M.Younas) >> 5. save vector to bin-file at each time step (robert) >> 6. AMG as preconditioner (M.Younas) >> 7. Re: Two Questions about Matrix Preallocation (Barry Smith) >> 8. Re: Multigrid as preconditioner (Barry Smith) >> 9. Re: save vector to bin-file at each time step (Barry Smith) >> 10. Re: save vector to bin-file at each time step (robert) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 13 Dec 2011 22:49:40 -0500 >> From: Xiangdong Liang >> Subject: [petsc-users] run successfully on 20 processors but failed on >> 24 processors >> To: PETSc users list >> Message-ID: >> >> Content-Type: text/plain; charset=ISO-8859-1 >> >> Hello everyone, >> >> I am solving complex Ax=b with PaStix on 20 processors successfully >> but failed on 24 processors. The relatively error indicated by >> mat_pastix_verbose becomes "nan" for 24 processors. Where could be >> wrong? Can someone give me some hints on how I can debug? Thanks. >> >> Xiangdong >> >> >> ------------------------------ >> >> Message: 2 >> Date: Tue, 13 Dec 2011 22:59:09 -0600 >> From: Matthew Knepley >> Subject: Re: [petsc-users] run successfully on 20 processors but >> failed on 24 processors >> To: PETSc users list >> Message-ID: >> >> Content-Type: text/plain; charset="iso-8859-1" >> >> On Tue, Dec 13, 2011 at 9:49 PM, Xiangdong Liang wrote: >> >> > Hello everyone, >> > >> > I am solving complex Ax=b with PaStix on 20 processors successfully >> > but failed on 24 processors. The relatively error indicated by >> > mat_pastix_verbose becomes "nan" for 24 processors. Where could be >> > wrong? Can someone give me some hints on how I can debug? Thanks. >> >> >> First, make sure you did not put any NaNs in your matrix or rhs. >> >> Matt >> >> >> > >> > Xiangdong >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 3 >> Date: Wed, 14 Dec 2011 11:32:48 +0330 >> From: behzad baghapour >> Subject: [petsc-users] Two Questions about Matrix Preallocation >> To: PETSc users list >> Message-ID: >> >> Content-Type: text/plain; charset="iso-8859-1" >> >> Dear Developers, >> >> I am facing with two questions about Block-Sparse Preallocation: >> >> 1- I set array "nnz[]" and did: >> >> MatSetType ( JAC, MATSEQBAIJ ); >> MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); >> MatSeqAIJSetPreallocation( JAC, 0, nnz ); >> >> but when I saw log file with -info -mat_view_info, I found this message: >> >> [0] MatSetUpPreallocation(): Warning not preallocating matrix storage >> >> What this means? Did I correctly preallocate the matrix? >> >> 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused >> with descriptions of the command in man-page: >> >> what are "nz" and "nnz[]" in this case? >> >> what about number of blocks per row? >> >> Thanks bothering too much :-) >> BehZad >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 4 >> Date: Wed, 14 Dec 2011 14:07:10 +0100 >> From: "M.Younas" >> Subject: [petsc-users] Multigrid as preconditioner >> To: petsc-users at mcs.anl.gov >> Message-ID: <73f0f8e5cc491.4ee8ad8e at rug.nl> >> Content-Type: text/plain; charset="iso-8859-1" >> >> ?Hi, PETSc, Team, >> >> I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? >> >> Kind regards, >> >> Younas >> >> RUG, Groningen, The Netherlands. >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 5 >> Date: Wed, 14 Dec 2011 15:00:03 +0100 >> From: robert >> Subject: [petsc-users] save vector to bin-file at each time step >> To: petsc-users at mcs.anl.gov >> Message-ID: <4EE8ABE3.5050108 at unil.ch> >> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" >> >> Hello, >> >> I am quite new to petsc and therefore my question might be quite trivial. >> >> I have a 1D finite difference code and want to store several vectors in >> a binary: >> >> ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); >> ierr = >> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); >> CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); >> ierr = VecView(u,viewer); CHKERRQ(ierr); >> ierr = VecView(rhs,viewer);CHKERRQ(ierr); >> >> However, when I read the file with matlab (PetscBinaryRead.m) I only get >> the first vector. >> >> In further application I would like to use something similar but putting >> out the solution vector every timestep. >> >> Could someone just provide me some sample code? >> >> Thanks, >> Robert >> >> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 6 >> Date: Wed, 14 Dec 2011 15:04:47 +0100 >> From: "M.Younas" >> Subject: [petsc-users] AMG as preconditioner >> To: petsc-users at mcs.anl.gov >> Message-ID: <7420f1bdca4d2.4ee8bb0f at rug.nl> >> Content-Type: text/plain; charset="iso-8859-1" >> >> ?Hi, PETSc, Team, >> >> I have >> solved a non-symmetric Poisson problem using GMRES(m) as solver and >> BoomerAMG (from Hypre) as preconditioner. But I don't understand how the >> BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the >> procedure? Does it work like BoomerAMG precondition the liner system >> first (once only) and then GMRES(m) is being used for stopping >> criterion? >> >> Kind regards, >> >> Younas >> >> RUG, Groningen, The Netherlands. >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 7 >> Date: Wed, 14 Dec 2011 08:22:33 -0600 >> From: Barry Smith >> Subject: Re: [petsc-users] Two Questions about Matrix Preallocation >> To: PETSc users list , behzad baghapour >> >> Message-ID: >> Content-Type: text/plain; charset=us-ascii >> >> >> On Dec 14, 2011, at 2:02 AM, behzad baghapour wrote: >> >> > Dear Developers, >> > >> > I am facing with two questions about Block-Sparse Preallocation: >> > >> > 1- I set array "nnz[]" and did: >> > >> > MatSetType ( JAC, MATSEQBAIJ ); >> > MatSetSizes( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); >> > MatSeqAIJSetPreallocation( JAC, 0, nnz ); >> > >> > but when I saw log file with -info -mat_view_info, I found this message: >> > >> > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage >> >> Since you are using BAIJ matrix type you need to also call MatSeqBAIJSetPreallocation() >> >> > >> > What this means? Did I correctly preallocate the matrix? >> > >> > 2- I need to test MatSeqBAIJSetPreallocation() too but I little confused with descriptions of the command in man-page: >> > >> > what are "nz" and "nnz[]" in this case? >> > >> > what about number of blocks per row? >> >> nz is the maximum number of blocks per row, just like with other matrix formats or you can set nnz[] which for each row is the number of blocks from the diagonal block to the end of the row. For example with a block size of one the matrix >> >> 1 0 2 >> 0 3 4 >> 2 4 5 >> >> would have a nnz[] = 2,2,1 >> >> Barry >> >> > >> > Thanks bothering too much :-) >> > BehZad >> > >> >> >> >> ------------------------------ >> >> Message: 8 >> Date: Wed, 14 Dec 2011 08:26:27 -0600 >> From: Barry Smith >> Subject: Re: [petsc-users] Multigrid as preconditioner >> To: PETSc users list >> Message-ID: <2770E217-0CC1-4E85-AB40-35B00F705D53 at mcs.anl.gov> >> Content-Type: text/plain; charset=us-ascii >> >> >> On Dec 14, 2011, at 7:07 AM, M.Younas wrote: >> >> > Hi, PETSc, Team, >> > >> > I have solved a non-symmetric Poisson problem using GMRES(m) as solver and BoomerAMG (from Hypre) as preconditioner. But I don't understand how the BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the procedure? Does it work like BoomerAMG precondition the liner system first (once only) and then GMRES(m) is being used for stopping criterion? >> >> By default PETSc uses ONE v-cycle of the BoomerAMG as a preconditioner for GMRES. >> >> With the option -ksp_type richardson then PETSc is using a bunch of BoomerAMG v-cycles until the convergence criteria is satisfied without using a GMRES >> >> Barry >> >> Note that a preconditioner can be any stationary linear iterative scheme. That is any linear operation that approximately solves A x = b, no matter how complicated the stationary linear iterative scheme is it can be used as a preconditioner. >> >> >> > >> > Kind regards, >> > >> > Younas >> > >> > RUG, Groningen, The Netherlands. >> >> >> >> ------------------------------ >> >> Message: 9 >> Date: Wed, 14 Dec 2011 08:32:58 -0600 >> From: Barry Smith >> Subject: Re: [petsc-users] save vector to bin-file at each time step >> To: PETSc users list >> Message-ID: <76717B20-13EF-4FD2-BF50-CC12E9E87842 at mcs.anl.gov> >> Content-Type: text/plain; charset=us-ascii >> >> >> On Dec 14, 2011, at 8:00 AM, robert wrote: >> >> > Hello, >> > >> > I am quite new to petsc and therefore my question might be quite trivial. >> > >> > I have a 1D finite difference code and want to store several vectors in a binary: >> > >> > ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); >> > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); >> > ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); >> > ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); >> > ierr = VecView(u,viewer); CHKERRQ(ierr); >> > ierr = VecView(rhs,viewer);CHKERRQ(ierr); >> > >> > However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. >> >> 1) Did you call PetscViewerDestroy() before the end of the program? If not called all the data may not be flushed into the file. >> >> 2) How are you reading the vectors into Matlab? For example [u,rhs] = PetscBinaryRead('filename') will read in two vectors. If you want to call PetscBinaryRead() repeatedly to read more information than read the help for PetscBinaryRead() carefully, you cannot simply call u = PetscBinaryRead('filename'); rhs = PetscBinaryRead('filename') and hope to get the second vector >> >> For example: fd = PetscOpenFile('filename'); >> u = PetscBinaryRead(fd); % get first vector >> rhs = PetscBinaryRead(fd); % get second vector >> >> Barry >> >> >> > >> > In further application I would like to use something similar but putting out the solution vector every timestep. >> > >> > Could someone just provide me some sample code? >> > >> > Thanks, >> > Robert >> > >> > >> >> >> >> ------------------------------ >> >> Message: 10 >> Date: Wed, 14 Dec 2011 15:44:39 +0100 >> From: robert >> Subject: Re: [petsc-users] save vector to bin-file at each time step >> To: PETSc users list >> Message-ID: <4EE8B657.4090506 at unil.ch> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> On 12/14/2011 03:32 PM, Barry Smith wrote: >> > On Dec 14, 2011, at 8:00 AM, robert wrote: >> > >> >> Hello, >> >> >> >> I am quite new to petsc and therefore my question might be quite trivial. >> >> >> >> I have a 1D finite difference code and want to store several vectors in a binary: >> >> >> >> ierr = PetscViewerCreate(PETSC_COMM_WORLD,&viewer);CHKERRQ(ierr); >> >> ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_APPEND,&viewer); CHKERRQ(ierr); >> >> ierr = PetscObjectSetName((PetscObject)u,"u");CHKERRQ(ierr); >> >> ierr = PetscObjectSetName((PetscObject)rhs,"rhs");CHKERRQ(ierr); >> >> ierr = VecView(u,viewer); CHKERRQ(ierr); >> >> ierr = VecView(rhs,viewer);CHKERRQ(ierr); >> >> >> >> However, when I read the file with matlab (PetscBinaryRead.m) I only get the first vector. >> >> >> >> >> >> 2) How are you reading the vectors into Matlab? For example [u,rhs] = PetscBinaryRead('filename') will read in two vectors. If you want to call PetscBinaryRead() repeatedly to read more information than ... >> >> Hi, >> >> thank you for your fast reply. It's as easy as that and I spent hours >> ... :-) >> >> >> >> >> >> In further application I would like to use something similar but putting out the solution vector every timestep. >> >> >> >> Could someone just provide me some sample code? >> >> >> >> Thanks, >> >> Robert >> >> >> >> >> > >> >> >> >> ------------------------------ >> >> _______________________________________________ >> petsc-users mailing list >> petsc-users at mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users >> >> >> End of petsc-users Digest, Vol 36, Issue 50 >> ******************************************* From knepley at gmail.com Wed Dec 14 11:36:39 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 14 Dec 2011 11:36:39 -0600 Subject: [petsc-users] AMG as preconditioner In-Reply-To: <7420f1bdca4d2.4ee8bb0f@rug.nl> References: <7420f1bdca4d2.4ee8bb0f@rug.nl> Message-ID: On Wed, Dec 14, 2011 at 8:04 AM, M.Younas wrote: > Hi, PETSc, Team, > > I have solved a non-symmetric Poisson problem using GMRES(m) as solver and > BoomerAMG (from Hypre) as preconditioner. But I don't understand how the > BoomerAMG works as preconditioner with GMRES(m). Can anyone explain the > procedure? Does it work like BoomerAMG precondition the liner system first > (once only) and then GMRES(m) is being used for stopping criterion? > Please see Yousef Saad's book on Iterative Solution of Linear Systems. Its available on his homepage. Matt > Kind regards, > > Younas > > RUG, Groningen, The Netherlands. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Wed Dec 14 16:07:12 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 14 Dec 2011 17:07:12 -0500 Subject: [petsc-users] run successfully on 20 processors but failed on 24 processors In-Reply-To: References: Message-ID: I use MatNorm and VecNorm on A and b before I call kspsolve, and both of them are finite. Then I use fp_trap to catch where the nan comes from. However, it traces down to unlikely place pthread_join. The gdb where information is given below. Can you give me some help? Thanks. 0x00007faeb6d8cbe5 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007faeb8ab7d6b in sopalin_launch_thread (procnum=18, procnbr=24, ptr=0xffffffff, calc_thrdnbr=1, calc_routine=0x7faeb8a7c9f3 , calc_data=0xfe78f0, comm_thrdnbr=0, comm_routine=0x7faeb8a82e4c , comm_data=0xfe78f0, ooc_thrdnbr=0, ooc_routine=0, ooc_data=0xfe78f0) at sopalin/src/sopalin_thread.c:235 #2 0x00007faeb8a81e52 in Z_Ugmres_thread (datacode=0x11c8390, sopaparam=0x11c8520) at sopalin/src/raff.c:1174 #3 0x00007faeb8a53993 in Z_pastix_task_raff (pastix_data=0x11c8390, pastix_comm=0x1e02690, n=210000, b=0x2528ec0, rhsnbr=1, loc2glob=0x0) at sopalin/src/pastix.c:3581 #4 0x00007faeb8a54a0e in z_pastix (pastix_data=0x20defc0, pastix_comm=0x1e02690, n=210000, colptr=0x245b610, row=0x7faea68b7760, avals=0x7faea4005760, perm=0x238dd60, invp=0x20e1080, b=0x2528ec0, rhs=1, iparm=0x20df004, dparm=0x20df108) at sopalin/src/pastix.c:4262 #5 0x00007faeb83b7fd5 in MatSolve_PaStiX (A=0x203e280, b=0x12140d0, x=0x1f1b430) at /home/xdliang/MyLocal/petsc-dev/src/mat/impls/aij/mpi/pastix/pastix.c:328 #6 0x00007faeb7b51d7c in MatSolve (mat=0x203e280, b=0x12140d0, x=0x1f1b430) at /home/xdliang/MyLocal/petsc-dev/src/mat/interface/matrix.c:3106 #7 0x00007faeb8540f0e in PCApply_LU (pc=0x1de4d20, x=0x12140d0, y=0x1f1b430) ---Type to continue, or q to quit--- at /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c:204 #8 0x00007faeb85df50b in PCApply (pc=0x1de4d20, x=0x12140d0, y=0x1f1b430) at /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/interface/precon.c:383 #9 0x00007faeb863c660 in KSPSolve_PREONLY (ksp=0x1e37590) at /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/impls/preonly/preonly.c:26 #10 0x00007faeb86707fe in KSPSolve (ksp=0x1e37590, b=0x12140d0, x=0x1f1b430) at /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/interface/itfunc.c:429 #11 0x000000000040a213 in EigenSolver_cmplx (data=0x7fffba7529b0, Linear=1, Eig=0, maxeigit=10) at EigenSolver_cmplx.c:66 #12 0x0000000000408ddf in main (argc=77, argv=0x7fffba754df8) at mldos_cmplx.c:322 Here is the error information: [18]PETSC ERROR: *** unknown floating point error occurred *** [18]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [18]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [18]PETSC ERROR: where the result is a bitwise OR of the following flags: [18]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [18]PETSC ERROR: Try option -start_in_debugger [18]PETSC ERROR: likely location of problem given in stack below [18]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [18]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [18]PETSC ERROR: INSTEAD the line number of the start of the function [18]PETSC ERROR: is given. [18]PETSC ERROR: [18] PetscDefaultFPTrap line 342 /home/xdliang/MyLocal/petsc-dev/src/sys/error/fp.c [18]PETSC ERROR: [18] MatSolve_PaStiX line 303 /home/xdliang/MyLocal/petsc-dev/src/mat/impls/aij/mpi/pastix/pastix.c [18]PETSC ERROR: [18] MatSolve line 3089 /home/xdliang/MyLocal/petsc-dev/src/mat/interface/matrix.c [18]PETSC ERROR: [18] PCApply_LU line 202 /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c [18]PETSC ERROR: [18] PCApply line 373 /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/interface/precon.c [18]PETSC ERROR: [18] KSPSolve_PREONLY line 19 /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/impls/preonly/preonly.c [18]PETSC ERROR: [18] KSPSolve line 334 /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/interface/itfunc.c [18]PETSC ERROR: User provided function() line 0 in Unknown directoryUnknown file trapped floating point error On Tue, Dec 13, 2011 at 11:59 PM, Matthew Knepley wrote: > On Tue, Dec 13, 2011 at 9:49 PM, Xiangdong Liang wrote: >> >> Hello everyone, >> >> I am solving complex Ax=b with PaStix on 20 processors successfully >> but failed on 24 processors. The relatively error indicated by >> mat_pastix_verbose becomes "nan" ?for 24 processors. Where could be >> wrong? Can someone give me some hints on how I can debug? Thanks. > > > First, make sure you did not put any NaNs in your matrix or rhs. > > ? ?Matt > >> >> >> Xiangdong > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Wed Dec 14 16:12:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 14 Dec 2011 16:12:03 -0600 Subject: [petsc-users] run successfully on 20 processors but failed on 24 processors In-Reply-To: References: Message-ID: On Wed, Dec 14, 2011 at 4:07 PM, Xiangdong Liang wrote: > I use MatNorm and VecNorm on A and b before I call kspsolve, and both > of them are finite. Then I use fp_trap to catch where the nan comes > from. However, it traces down to unlikely place pthread_join. The gdb > where information is given below. Can you give me some help? Thanks. > Great. This is very helpful. It seems quite clear that this is a Pastix problem. I would submit this is to the Pastix development list. Thanks, Matt > 0x00007faeb6d8cbe5 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00007faeb8ab7d6b in sopalin_launch_thread (procnum=18, procnbr=24, > ptr=0xffffffff, calc_thrdnbr=1, > calc_routine=0x7faeb8a7c9f3 , calc_data=0xfe78f0, > comm_thrdnbr=0, comm_routine=0x7faeb8a82e4c , > comm_data=0xfe78f0, ooc_thrdnbr=0, ooc_routine=0, ooc_data=0xfe78f0) > at sopalin/src/sopalin_thread.c:235 > #2 0x00007faeb8a81e52 in Z_Ugmres_thread (datacode=0x11c8390, > sopaparam=0x11c8520) at sopalin/src/raff.c:1174 > #3 0x00007faeb8a53993 in Z_pastix_task_raff (pastix_data=0x11c8390, > pastix_comm=0x1e02690, n=210000, b=0x2528ec0, rhsnbr=1, loc2glob=0x0) > at sopalin/src/pastix.c:3581 > #4 0x00007faeb8a54a0e in z_pastix (pastix_data=0x20defc0, > pastix_comm=0x1e02690, n=210000, colptr=0x245b610, row=0x7faea68b7760, > avals=0x7faea4005760, perm=0x238dd60, invp=0x20e1080, b=0x2528ec0, > rhs=1, > iparm=0x20df004, dparm=0x20df108) at sopalin/src/pastix.c:4262 > #5 0x00007faeb83b7fd5 in MatSolve_PaStiX (A=0x203e280, b=0x12140d0, > x=0x1f1b430) > at > /home/xdliang/MyLocal/petsc-dev/src/mat/impls/aij/mpi/pastix/pastix.c:328 > #6 0x00007faeb7b51d7c in MatSolve (mat=0x203e280, b=0x12140d0, > x=0x1f1b430) > at /home/xdliang/MyLocal/petsc-dev/src/mat/interface/matrix.c:3106 > #7 0x00007faeb8540f0e in PCApply_LU (pc=0x1de4d20, x=0x12140d0, > y=0x1f1b430) > ---Type to continue, or q to quit--- > at /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c:204 > #8 0x00007faeb85df50b in PCApply (pc=0x1de4d20, x=0x12140d0, y=0x1f1b430) > at /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/interface/precon.c:383 > #9 0x00007faeb863c660 in KSPSolve_PREONLY (ksp=0x1e37590) > at > /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/impls/preonly/preonly.c:26 > #10 0x00007faeb86707fe in KSPSolve (ksp=0x1e37590, b=0x12140d0, > x=0x1f1b430) > at /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/interface/itfunc.c:429 > #11 0x000000000040a213 in EigenSolver_cmplx (data=0x7fffba7529b0, Linear=1, > Eig=0, maxeigit=10) at EigenSolver_cmplx.c:66 > #12 0x0000000000408ddf in main (argc=77, argv=0x7fffba754df8) > at mldos_cmplx.c:322 > > Here is the error information: > > [18]PETSC ERROR: *** unknown floating point error occurred *** > [18]PETSC ERROR: The specific exception can be determined by running > in a debugger. When the > [18]PETSC ERROR: debugger traps the signal, the exception can be found > with fetestexcept(0x3d) > [18]PETSC ERROR: where the result is a bitwise OR of the following flags: > [18]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [18]PETSC ERROR: Try option -start_in_debugger > [18]PETSC ERROR: likely location of problem given in stack below > [18]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [18]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [18]PETSC ERROR: INSTEAD the line number of the start of the function > [18]PETSC ERROR: is given. > [18]PETSC ERROR: [18] PetscDefaultFPTrap line 342 > /home/xdliang/MyLocal/petsc-dev/src/sys/error/fp.c > [18]PETSC ERROR: [18] MatSolve_PaStiX line 303 > /home/xdliang/MyLocal/petsc-dev/src/mat/impls/aij/mpi/pastix/pastix.c > [18]PETSC ERROR: [18] MatSolve line 3089 > /home/xdliang/MyLocal/petsc-dev/src/mat/interface/matrix.c > [18]PETSC ERROR: [18] PCApply_LU line 202 > /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/impls/factor/lu/lu.c > [18]PETSC ERROR: [18] PCApply line 373 > /home/xdliang/MyLocal/petsc-dev/src/ksp/pc/interface/precon.c > [18]PETSC ERROR: [18] KSPSolve_PREONLY line 19 > /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/impls/preonly/preonly.c > [18]PETSC ERROR: [18] KSPSolve line 334 > /home/xdliang/MyLocal/petsc-dev/src/ksp/ksp/interface/itfunc.c > [18]PETSC ERROR: User provided function() line 0 in Unknown > directoryUnknown file trapped floating point error > > > > > > On Tue, Dec 13, 2011 at 11:59 PM, Matthew Knepley > wrote: > > On Tue, Dec 13, 2011 at 9:49 PM, Xiangdong Liang > wrote: > >> > >> Hello everyone, > >> > >> I am solving complex Ax=b with PaStix on 20 processors successfully > >> but failed on 24 processors. The relatively error indicated by > >> mat_pastix_verbose becomes "nan" for 24 processors. Where could be > >> wrong? Can someone give me some hints on how I can debug? Thanks. > > > > > > First, make sure you did not put any NaNs in your matrix or rhs. > > > > Matt > > > >> > >> > >> Xiangdong > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Dec 14 21:14:55 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Dec 2011 19:14:55 -0800 Subject: [petsc-users] Optimised vs Debug In-Reply-To: <50d8467ef890c172ecbfdc53ed199059.squirrel@wm.lmn.pub.ro> References: <1742cc3834e2f262b9bec4b29cf72b62.squirrel@wm.lmn.pub.ro> <50d8467ef890c172ecbfdc53ed199059.squirrel@wm.lmn.pub.ro> Message-ID: On Wed, Dec 14, 2011 at 02:32, Bogdan Dita wrote: > So if on the cluster the mpich2 version is optimised whatever > implementation of Petsc I choose, I can't get a better time result? > If this is true then this are really bad news. > What are your sugestions? So far I used superlu_dist, I tried mumps a few > times but I always got some strange errors and if I recall correctly from > the petsc-users emails it is best to use mumps with petsc-dev? > You can pass the specific optimization flags that you want. The confusion is that MPICH2 is turning optimization on by default, so even the "debug" build is using optimized compiler flags. As for "strange errors", you have to send more information: configure.log, what you expected to happen, what actually happened, and the entire error message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rxk at cfdrc.com Thu Dec 15 10:23:58 2011 From: rxk at cfdrc.com (Ravi Kannan) Date: Thu, 15 Dec 2011 10:23:58 -0600 Subject: [petsc-users] boomerAmg scalability In-Reply-To: <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> Message-ID: <003701ccbb45$f2bb23f0$d8316bd0$@com> Dear All, This is Ravi Kannan from CFD Research Corporation. Recently, we are experimenting with the BoomerAMG preconditioner for some "stiff" CFD problems. In that regard, all the other standard solver-preconditioner combinations failed for the current CFD problem. The boomer is the only one which is able to provide with "converged" solutions. We noticed that the scalability of this boomer preconditioner is really poor. For instance, even with a cell size of 2 million, we cannot scale to even 16 partitions (in contrast, the other solver-preconditioner combinations like the BI-CGS/BJacobi gave good enough scalability). Are we missing something? Do we need to use a more latest version of boomer? Thanks, Ravi. From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Bobby Philip Sent: Thursday, December 15, 2011 9:22 AM To: For users of the development version of PETSc Subject: Re: [petsc-dev] controlling vector values while doing matrix free operations Hmmmm..sorry guys - the entire thread got put away in a folder by my mail reader and I just discovered all your emails :-) SNESVI looks interesting but as someone pointed out I am using matrix free. And as I understand from Barry a matrix free version of SNESVI is not implemented. The idea of flipping the sign seems to be a poor mans approach that might work for me though I would still need some mechanisms in SNES to put the hooks in. Bobby On Dec 14, 2011, at 10:45 PM, Jed Brown wrote: On Wed, Dec 14, 2011 at 19:40, Dmitry Karpeev wrote: The trouble is that the constraints can get "eliminated" only when they become active. That's not what I meant. I meant to transform the algebraic system so that those extra variables were eliminated. The point is that we tend to put a lot of effort into designing effective preconditioners for a standard formulation (e.g. conservative variables), and that is partly lost of we have this other system in which the evaluations of constitutive relations are added explicitly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Fri Dec 16 06:59:41 2011 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 16 Dec 2011 12:59:41 +0000 Subject: [petsc-users] openmpi 1.4.4 Message-ID: OpenMPI 1.4.4 is the stable release since Oct 2011 but petsc-3.2-p5 still downloads 1.4.3: $ grep -m 1 download_openmpi petsc-3.2-p5/config/BuildSystem/config/packages/MPI.py self.download_openmpi = ['http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.3.tar.gz'] Is there a reason not to use 1.4.4? I'm considering OpenMPI 1.4.4 with intel compilers 11.1 on a CentOS 6.1 cluster, if anyone has experience with such a setup please let me know. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From balay at mcs.anl.gov Fri Dec 16 07:21:12 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 16 Dec 2011 07:21:12 -0600 (CST) Subject: [petsc-users] openmpi 1.4.4 In-Reply-To: References: Message-ID: At the relase time of petsc-3.2 openmpi-1.4.3 was the latest. And we generally don't update the default versions of externalpackages for a release version of petsc. However if the installation of 1.4.4 hasn't changed - the following might work for you: --download-openmpi=http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.4.tar.gz Satish On Fri, 16 Dec 2011, Klaij, Christiaan wrote: > OpenMPI 1.4.4 is the stable release since Oct 2011 but petsc-3.2-p5 still downloads 1.4.3: > > $ grep -m 1 download_openmpi petsc-3.2-p5/config/BuildSystem/config/packages/MPI.py > self.download_openmpi = ['http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.3.tar.gz'] > > Is there a reason not to use 1.4.4? I'm considering OpenMPI 1.4.4 with intel compilers 11.1 on a CentOS 6.1 cluster, if anyone has experience with such a setup please let me know. > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > From ckontzialis at lycos.com Fri Dec 16 13:04:37 2011 From: ckontzialis at lycos.com (Konstantinos Kontzialis) Date: Fri, 16 Dec 2011 21:04:37 +0200 Subject: [petsc-users] Get solution at a stage for ts_type arkimex Message-ID: <4EEB9645.9020307@lycos.com> Dear all, I use ts_type arkimex, and I want at the end of every RK stage to apply a limiter function. How can I set TS so that to apply the function at the end of a RK stage? Thank you, Kostas From sean at mcs.anl.gov Fri Dec 16 13:18:46 2011 From: sean at mcs.anl.gov (Sean Farley) Date: Fri, 16 Dec 2011 13:18:46 -0600 Subject: [petsc-users] Get solution at a stage for ts_type arkimex In-Reply-To: <4EEB9645.9020307@lycos.com> References: <4EEB9645.9020307@lycos.com> Message-ID: > > I use ts_type arkimex, and I want at the end of every RK stage to apply a > limiter function. How can I set TS so that to apply the function at the end > of a RK stage? Jed added the new TSAdapt code (in petsc-dev), I would suggest looking at TS*SetAdapt. Here is the one for TSAlpha: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/TSAlphaSetAdapt.html If you looks in $PETSC_DIR/src/ts/examples/tutorials, ex17.c and ex18.c both fiddle with this. In theory, I believe it would be possible to add a custom arkimex adapter that would do this limiter (as a hack). Before Matt or anyone else offers (and *depending on your problem!*) you could try to write this as a constrained inequality and use some of the new SNESVI capabilities: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESVISetVariableBounds.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sat Dec 17 00:33:36 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sat, 17 Dec 2011 10:03:36 +0330 Subject: [petsc-users] Error in Petsc configuration Message-ID: Dear Developers, I want to configure Petsc with given path for the external packages: ./configure --with-f2cblaslapack-dir=/home/behzad/softs/petsc/externalpackages/f2cblaslapack-3.1.1.q --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 --with-hypre-dir=/home/behzad/softs/petsc/externalpackages/hypre-2.7.0b --with-parms-dir=/home/behzad/softs/petsc/externalpackages/pARMS_3.2 --with-debugging=no --with-cxx=g++ -with-clanguage=cxx but I received the error: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 did not work ******************************************************************************* and the procedure was terminated. I know I may in wrong way, please show me how should be the configuring in this matter?? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Dec 17 08:13:07 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 17 Dec 2011 08:13:07 -0600 (CST) Subject: [petsc-users] Error in Petsc configuration In-Reply-To: References: Message-ID: Are you sure these packages are installed at these locations? Did you install them manually? Note: If this is a location from previous --dowload-mpich --download-hypre --download-parms builds - then these dirs just have sources - and not installs. One should continue to use the --download option [even though the sources are already downloaded] If you still have issues - correspond with petsc-maint with the appropriate configure.log Satish On Sat, 17 Dec 2011, behzad baghapour wrote: > Dear Developers, > > I want to configure Petsc with given path for the external packages: > > ./configure > --with-f2cblaslapack-dir=/home/behzad/softs/petsc/externalpackages/f2cblaslapack-3.1.1.q > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 > --with-hypre-dir=/home/behzad/softs/petsc/externalpackages/hypre-2.7.0b > --with-parms-dir=/home/behzad/softs/petsc/externalpackages/pARMS_3.2 > --with-debugging=no --with-cxx=g++ -with-clanguage=cxx > > but I received the error: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 did > not work > ******************************************************************************* > > and the procedure was terminated. I know I may in wrong way, please show me > how should be the configuring in this matter?? > > Thanks a lot, > BehZad > From juhaj at iki.fi Sat Dec 17 08:55:48 2011 From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=) Date: Sat, 17 Dec 2011 14:55:48 +0000 Subject: [petsc-users] Vec entries altered incorrectly Message-ID: <201112171455.57152.juhaj@iki.fi> Hi all! I am at a loss: my vector entries get zeroed when I am not even using the vector. I have the vectors: ierr = DACreateGlobalVector(lattice.da, &Fields); CHKERRQ(ierr); ierr = VecDuplicate(Fields, &lattice.prev_state); CHKERRQ(ierr); ... TSSolve(lattice.ts, Fields); Later on, in a my RHS function PETSc has kindly provided me with my "lattice.da" as part of my context (it is now called simply "da"), it has also given me Fields, so I can do: DALocalInfo l_da; ahcs_field ***fields, ***rhs, ***pfields; Vec lfields, lpfields; begin_using_petsc_DA_step_1(da, &l_da, Fields, &lfields, &fields); where PetscErrorCode begin_using_petsc_DA_step_1(DA da, DALocalInfo *l_da, Vec Fields, Vec *lfields, void *fields) { PetscErrorCode ierr; ierr = DAGetLocalInfo(da, l_da); CHKERRQ(ierr); ierr = DAGetLocalVector(da, lfields); CHKERRQ(ierr); ierr = DAGlobalToLocalBegin(da, Fields, INSERT_VALUES, *lfields); At this point, the other vector in lattice.prev_state gets (some of) its entries zeroed! What am I doing wrong? Obviously, DAGlobalToLocalBegin working on Fields should not alter anything in a duplicated vector, let alone INTERNAL values of that vector (i.e. non-ghost points, which is what happens here). Cheers, Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Sat Dec 17 09:50:07 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 17 Dec 2011 09:50:07 -0600 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: <201112171455.57152.juhaj@iki.fi> References: <201112171455.57152.juhaj@iki.fi> Message-ID: On Sat, Dec 17, 2011 at 8:55 AM, Juha J?ykk? wrote: > Hi all! > > I am at a loss: my vector entries get zeroed when I am not even using the > vector. > This is memory overwriting. Use valgrind to find it. Matt > I have the vectors: > > ierr = DACreateGlobalVector(lattice.da, &Fields); CHKERRQ(ierr); > ierr = VecDuplicate(Fields, &lattice.prev_state); CHKERRQ(ierr); > ... > TSSolve(lattice.ts, Fields); > > Later on, in a my RHS function PETSc has kindly provided me with my > "lattice.da" as part of my context (it is now called simply "da"), it has > also > given me Fields, so I can do: > > DALocalInfo l_da; > ahcs_field ***fields, ***rhs, ***pfields; > Vec lfields, lpfields; > begin_using_petsc_DA_step_1(da, &l_da, Fields, &lfields, &fields); > > where > > PetscErrorCode begin_using_petsc_DA_step_1(DA da, DALocalInfo *l_da, Vec > Fields, Vec *lfields, void *fields) { > > PetscErrorCode ierr; > ierr = DAGetLocalInfo(da, l_da); CHKERRQ(ierr); > ierr = DAGetLocalVector(da, lfields); CHKERRQ(ierr); > ierr = DAGlobalToLocalBegin(da, Fields, INSERT_VALUES, *lfields); > > At this point, the other vector in lattice.prev_state gets (some of) its > entries zeroed! > > What am I doing wrong? Obviously, DAGlobalToLocalBegin working on Fields > should not alter anything in a duplicated vector, let alone INTERNAL > values of > that vector (i.e. non-ghost points, which is what happens here). > > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 17 10:53:22 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 17 Dec 2011 10:53:22 -0600 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: <201112171455.57152.juhaj@iki.fi> References: <201112171455.57152.juhaj@iki.fi> Message-ID: <569F9908-06B8-462A-BA7C-A353BAC080FB@mcs.anl.gov> VecDuplicate does NOT copy the vector values over to the newly created vector. You must call VecCopy() after the VecDuplicate() if you want to copy the values over. Barry On Dec 17, 2011, at 8:55 AM, Juha J?ykk? wrote: > Hi all! > > I am at a loss: my vector entries get zeroed when I am not even using the > vector. > > I have the vectors: > > ierr = DACreateGlobalVector(lattice.da, &Fields); CHKERRQ(ierr); > ierr = VecDuplicate(Fields, &lattice.prev_state); CHKERRQ(ierr); > ... > TSSolve(lattice.ts, Fields); > > Later on, in a my RHS function PETSc has kindly provided me with my > "lattice.da" as part of my context (it is now called simply "da"), it has also > given me Fields, so I can do: > > DALocalInfo l_da; > ahcs_field ***fields, ***rhs, ***pfields; > Vec lfields, lpfields; > begin_using_petsc_DA_step_1(da, &l_da, Fields, &lfields, &fields); > > where > > PetscErrorCode begin_using_petsc_DA_step_1(DA da, DALocalInfo *l_da, Vec > Fields, Vec *lfields, void *fields) { > > PetscErrorCode ierr; > ierr = DAGetLocalInfo(da, l_da); CHKERRQ(ierr); > ierr = DAGetLocalVector(da, lfields); CHKERRQ(ierr); > ierr = DAGlobalToLocalBegin(da, Fields, INSERT_VALUES, *lfields); > > At this point, the other vector in lattice.prev_state gets (some of) its > entries zeroed! > > What am I doing wrong? Obviously, DAGlobalToLocalBegin working on Fields > should not alter anything in a duplicated vector, let alone INTERNAL values of > that vector (i.e. non-ghost points, which is what happens here). > > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- From juhaj at iki.fi Sat Dec 17 16:49:14 2011 From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=) Date: Sat, 17 Dec 2011 22:49:14 +0000 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: <569F9908-06B8-462A-BA7C-A353BAC080FB@mcs.anl.gov> References: <201112171455.57152.juhaj@iki.fi> <569F9908-06B8-462A-BA7C-A353BAC080FB@mcs.anl.gov> Message-ID: <201112172249.17416.juhaj@iki.fi> > VecDuplicate does NOT copy the vector values over to the newly created > vector. You must call VecCopy() after the VecDuplicate() if you want to > copy the values over. No, what I do *not* want to happen is DAGlobalToLocalBegin(da, vec1...) to overwrite values in vec2. I think this is actually a bug in PETSc, but cannot be sure because I do not understand the internals of Vec and DA operations well enough. The question really is, why would DAGlobalToLocalBegin overwrite values in an unrelated vector? (Well, almost unrelated: it was created by VecDuplicate from the other vector.) Cheers, -Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From juhaj at iki.fi Sat Dec 17 16:55:50 2011 From: juhaj at iki.fi (Juha =?utf-8?q?J=C3=A4ykk=C3=A4?=) Date: Sat, 17 Dec 2011 22:55:50 +0000 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: References: <201112171455.57152.juhaj@iki.fi> Message-ID: <201112172255.50679.juhaj@iki.fi> > This is memory overwriting. Use valgrind to find it. Valgrind gives it a clean bill of health: ==30174== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 153 from 1) ==30174== malloc/free: in use at exit: 0 bytes in 0 blocks. ==30174== malloc/free: 0 allocs, 0 frees, 0 bytes allocated. ==30174== For counts of detected errors, rerun with: -v ==30174== All heap blocks were freed -- no leaks are possible. What am I actually even looking for? Myself overwriting stuff or a PETSc bug? I do not quite understand how PETSc even could manage to overwrite unrelated vectors. Or do you suspect I overwrite the vector structure at somepoint and that confuses PETSc later on? Cheers, Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Sat Dec 17 17:18:45 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 17 Dec 2011 17:18:45 -0600 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: <201112172255.50679.juhaj@iki.fi> References: <201112171455.57152.juhaj@iki.fi> <201112172255.50679.juhaj@iki.fi> Message-ID: On Sat, Dec 17, 2011 at 4:55 PM, Juha J?ykk? wrote: > > This is memory overwriting. Use valgrind to find it. > > Valgrind gives it a clean bill of health: > > ==30174== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 153 from 1) > ==30174== malloc/free: in use at exit: 0 bytes in 0 blocks. > ==30174== malloc/free: 0 allocs, 0 frees, 0 bytes allocated. > ==30174== For counts of detected errors, rerun with: -v > ==30174== All heap blocks were freed -- no leaks are possible. > > What am I actually even looking for? Myself overwriting stuff or a PETSc > bug? > I do not quite understand how PETSc even could manage to overwrite > unrelated > vectors. Or do you suspect I overwrite the vector structure at somepoint > and > that confuses PETSc later on? > I guarantee you this is not a bug. This stuff is used by thousands of people every day. Its very simple and old code. There is a misunderstanding somewhere in your code about how this mechanism works. I suggest stripping down the code until there are just two vectors and you alter one. If you send me that, I will go through it to figure out what is happening. Matt > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From modhurita at gmail.com Sat Dec 17 18:05:31 2011 From: modhurita at gmail.com (Modhurita Mitra) Date: Sat, 17 Dec 2011 18:05:31 -0600 Subject: [petsc-users] Problems with configure Message-ID: Hi, I am trying to install PETSc while downloading MPICH, but I am facing problems running the configure script. Some errors from the configure.log file are: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running configure on MPICH: Could not execute "cd /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1 && ./configure --prefix=/export/home/modhurit/soft/petsc-3.2-p5/arch-linux2-c-debug CC="gcc" CFLAGS=" -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 " --disable-cxx --disable-fc F77="g77" FFLAGS=" -Wall -Wno-unused-variable -g " --with-device=ch3:sock --without-mpe --with-pm=hydra --enable-g=meminit --enable-fast": Configuring MPICH2 version 1.4.1p1 with '--prefix=/export/home/modhurit/soft/petsc-3.2-p5/arch-linux2-c-debug' 'CC=gcc' 'CFLAGS= -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 ' '--disable-cxx' '--disable-fc' 'F77=g77' 'FFLAGS= -Wall -Wno-unused-variable -g ' '--with-device=ch3:sock' '--without-mpe' '--with-pm=hydra' '--enable-g=meminit' '--enable-fast' Running on system: Linux faraday 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:40 EST 2010 i686 i686 i386 GNU/Linux ----------------- /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7850: /scr/faraday/modhurit/matlab/RM: is a directory mkdir: cannot create directory `conftest': File exists /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7877: cd: conftest: Not a directory mkdir: cannot create directory `out': File exists ------------------ /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7913: /scr/faraday/modhurit/matlab/RM: is a directory cat: confdefs.h: No such file or directory configure: error: Version information not found. Configuration aborted. configure: error: MPL configure failed ******************************************************************************* -------------------- File "/export/home/modhurit/soft/petsc-3.2-p5/config/BuildSystem/config/packages/MPI.py", line 528, in InstallMPICH raise RuntimeError('Error running configure on MPICH: '+str(e)) -------------------- What might be going wrong? Thanks, Modhurita -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 17 18:11:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 17 Dec 2011 18:11:27 -0600 Subject: [petsc-users] Problems with configure In-Reply-To: References: Message-ID: <8B0F230F-EA25-4574-85E4-616A60D5BBCB@mcs.anl.gov> Please send configure.log from the PETSc directory to petsc-maint at mcs.anl.gov Barry On Dec 17, 2011, at 6:05 PM, Modhurita Mitra wrote: > Hi, > > I am trying to install PETSc while downloading MPICH, but I am facing problems running the configure script. Some errors from the configure.log file are: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error running configure on MPICH: Could not execute "cd /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1 && ./configure --prefix=/export/home/modhurit/soft/petsc-3.2-p5/arch-linux2-c-debug CC="gcc" CFLAGS=" -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 " --disable-cxx --disable-fc F77="g77" FFLAGS=" -Wall -Wno-unused-variable -g " --with-device=ch3:sock --without-mpe --with-pm=hydra --enable-g=meminit --enable-fast": > Configuring MPICH2 version 1.4.1p1 with '--prefix=/export/home/modhurit/soft/petsc-3.2-p5/arch-linux2-c-debug' 'CC=gcc' 'CFLAGS= -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 ' '--disable-cxx' '--disable-fc' 'F77=g77' 'FFLAGS= -Wall -Wno-unused-variable -g ' '--with-device=ch3:sock' '--without-mpe' '--with-pm=hydra' '--enable-g=meminit' '--enable-fast' > Running on system: Linux faraday 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:40 EST 2010 i686 i686 i386 GNU/Linux > > ----------------- > > > /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7850: /scr/faraday/modhurit/matlab/RM: is a directory > mkdir: cannot create directory `conftest': File exists > /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7877: cd: conftest: Not a directory > mkdir: cannot create directory `out': File exists > > ------------------ > > /export/home/modhurit/soft/petsc-3.2-p5/externalpackages/mpich2-1.4.1p1/src/mpl/configure: line 7913: /scr/faraday/modhurit/matlab/RM: is a directory > cat: confdefs.h: No such file or directory > configure: error: Version information not found. Configuration aborted. > configure: error: MPL configure failed > ******************************************************************************* > > -------------------- > > File "/export/home/modhurit/soft/petsc-3.2-p5/config/BuildSystem/config/packages/MPI.py", line 528, in InstallMPICH > raise RuntimeError('Error running configure on MPICH: '+str(e)) > > -------------------- > > What might be going wrong? > > Thanks, > Modhurita From Sanjay.Kharche at liverpool.ac.uk Sun Dec 18 01:54:31 2011 From: Sanjay.Kharche at liverpool.ac.uk (Kharche, Sanjay) Date: Sun, 18 Dec 2011 07:54:31 +0000 Subject: [petsc-users] training Message-ID: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> Hi I am a new user of PetSc and would like to find training events for myself and members of the project I am involved in. Can someone suggest any please. thanks Sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Dec 18 02:40:23 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 18 Dec 2011 12:10:23 +0330 Subject: [petsc-users] Error in Petsc configuration In-Reply-To: References: Message-ID: On Sat, Dec 17, 2011 at 5:43 PM, Satish Balay wrote: > Are you sure these packages are installed at these locations? Did you > install them manually? > Note: If this is a location from previous --dowload-mpich > --download-hypre --download-parms builds - then these dirs just have > sources - and not installs. One should continue to use the --download > option [even though the sources are already downloaded] > > Yes. I exactly configured first with --download-XXX. Then How I should configure if the next configurations will not need to download the packages again? Thank you very much. > If you still have issues - correspond with petsc-maint with the > appropriate configure.log > > Satish > > On Sat, 17 Dec 2011, behzad baghapour wrote: > > > Dear Developers, > > > > I want to configure Petsc with given path for the external packages: > > > > ./configure > > > --with-f2cblaslapack-dir=/home/behzad/softs/petsc/externalpackages/f2cblaslapack-3.1.1.q > > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 > > --with-hypre-dir=/home/behzad/softs/petsc/externalpackages/hypre-2.7.0b > > --with-parms-dir=/home/behzad/softs/petsc/externalpackages/pARMS_3.2 > > --with-debugging=no --with-cxx=g++ -with-clanguage=cxx > > > > but I received the error: > > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > > ------------------------------------------------------------------------------- > > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 > did > > not work > > > ******************************************************************************* > > > > and the procedure was terminated. I know I may in wrong way, please show > me > > how should be the configuring in this matter?? > > > > Thanks a lot, > > BehZad > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Dec 18 08:34:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 18 Dec 2011 08:34:50 -0600 Subject: [petsc-users] Error in Petsc configuration In-Reply-To: References: Message-ID: On Sun, Dec 18, 2011 at 2:40 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > On Sat, Dec 17, 2011 at 5:43 PM, Satish Balay wrote: > >> Are you sure these packages are installed at these locations? Did you >> install them manually? >> > > Note: If this is a location from previous --dowload-mpich >> --download-hypre --download-parms builds - then these dirs just have >> sources - and not installs. One should continue to use the --download >> option [even though the sources are already downloaded] >> >> Yes. I exactly configured first with --download-XXX. Then How I should > configure if the next configurations will not need to download the packages > again? > Just use --download- and it will find that existing download and not retrieve it again. Matt > Thank you very much. > > >> If you still have issues - correspond with petsc-maint with the >> appropriate configure.log >> >> Satish >> >> On Sat, 17 Dec 2011, behzad baghapour wrote: >> >> > Dear Developers, >> > >> > I want to configure Petsc with given path for the external packages: >> > >> > ./configure >> > >> --with-f2cblaslapack-dir=/home/behzad/softs/petsc/externalpackages/f2cblaslapack-3.1.1.q >> > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 >> > --with-hypre-dir=/home/behzad/softs/petsc/externalpackages/hypre-2.7.0b >> > --with-parms-dir=/home/behzad/softs/petsc/externalpackages/pARMS_3.2 >> > --with-debugging=no --with-cxx=g++ -with-clanguage=cxx >> >> > >> > but I received the error: >> > >> > >> ******************************************************************************* >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >> for >> > details): >> > >> ------------------------------------------------------------------------------- >> > --with-mpi-dir=/home/behzad/softs/petsc/externalpackages/mpich2-1.4.1p1 >> did >> > not work >> > >> ******************************************************************************* >> > >> > and the procedure was terminated. I know I may in wrong way, please >> show me >> > how should be the configuring in this matter?? >> > >> > Thanks a lot, >> > BehZad >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Dec 18 08:36:55 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 18 Dec 2011 08:36:55 -0600 Subject: [petsc-users] training In-Reply-To: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> References: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> Message-ID: On Sun, Dec 18, 2011 at 1:54 AM, Kharche, Sanjay < Sanjay.Kharche at liverpool.ac.uk> wrote: > > Hi > > I am a new user of PetSc and would like to find training events for myself > and members of the project I am involved in. Can someone suggest any please. > You can look at the tutorials we have given: http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html Matt > thanks > Sanjay > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Dec 18 09:12:02 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 18 Dec 2011 18:42:02 +0330 Subject: [petsc-users] Error in Petsc configuration In-Reply-To: References: Message-ID: OK. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sanjay.Kharche at liverpool.ac.uk Sun Dec 18 09:40:11 2011 From: Sanjay.Kharche at liverpool.ac.uk (Kharche, Sanjay) Date: Sun, 18 Dec 2011 15:40:11 +0000 Subject: [petsc-users] training In-Reply-To: References: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk>, Message-ID: <04649ABFF695C94F8E6CF3BBBA9B1665B4F180@BHEXMBX1.livad.liv.ac.uk> Thanks Matt, I am already working thru' the tutorials. I am also looking for training events to attend as it is part of the project. thanks Sanjay ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Matthew Knepley [knepley at gmail.com] Sent: 18 December 2011 14:36 To: PETSc users list Subject: Re: [petsc-users] training On Sun, Dec 18, 2011 at 1:54 AM, Kharche, Sanjay > wrote: Hi I am a new user of PetSc and would like to find training events for myself and members of the project I am involved in. Can someone suggest any please. You can look at the tutorials we have given: http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html Matt thanks Sanjay -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Dec 18 09:42:42 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 18 Dec 2011 09:42:42 -0600 Subject: [petsc-users] training In-Reply-To: <04649ABFF695C94F8E6CF3BBBA9B1665B4F180@BHEXMBX1.livad.liv.ac.uk> References: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> <04649ABFF695C94F8E6CF3BBBA9B1665B4F180@BHEXMBX1.livad.liv.ac.uk> Message-ID: On Sun, Dec 18, 2011 at 9:40 AM, Kharche, Sanjay < Sanjay.Kharche at liverpool.ac.uk> wrote: > > Thanks Matt, I am already working thru' the tutorials. I am also looking > for training events to attend as it is part of the project. > We do not currently have any scheduled. When we do, we will post them. We do sometimes respond to requests for tutorials. Matt > thanks > Sanjay > > ------------------------------ > *From:* petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] > on behalf of Matthew Knepley [knepley at gmail.com] > *Sent:* 18 December 2011 14:36 > *To:* PETSc users list > *Subject:* Re: [petsc-users] training > > On Sun, Dec 18, 2011 at 1:54 AM, Kharche, Sanjay < > Sanjay.Kharche at liverpool.ac.uk> wrote: > >> >> Hi >> >> I am a new user of PetSc and would like to find training events for >> myself and members of the project I am involved in. Can someone suggest any >> please. >> > > You can look at the tutorials we have given: > > http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html > > Matt > > >> thanks >> Sanjay >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From errodrigues at inf.ufrgs.br Sun Dec 18 11:20:29 2011 From: errodrigues at inf.ufrgs.br (Eduardo Rocha Rodrigues) Date: Sun, 18 Dec 2011 15:20:29 -0200 Subject: [petsc-users] training In-Reply-To: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> References: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> Message-ID: Hi Sanjay, There is a course scheduled in the Xsede portal. One can attend the webcast. As far as I could understand, you can take part on it even if you are not a Xsede user. Have a look at: https://www.xsede.org/web/xup/course-calendar/-/training/class/9;jsessionid=99676C05F4F8735E698F36D50D16C9AB Best regards from Brazil, Eduardo On Sun, Dec 18, 2011 at 5:54 AM, Kharche, Sanjay wrote: > > Hi > > I am a new user of PetSc and would like to find training events for myself > and members of the project I am involved in. Can someone suggest any please. > > thanks > Sanjay > From wumengda at gmail.com Sun Dec 18 13:08:29 2011 From: wumengda at gmail.com (Mengda Wu) Date: Sun, 18 Dec 2011 14:08:29 -0500 Subject: [petsc-users] Petsc compilation with -MDd using visual studio c++ 2008 Message-ID: Hi all, I am trying to compile with a debug version of Petsc with -MDd options using visual studio c++ 2008. The following options are used. ./config/configure.py --with-cc='cl' --with-fc=0 --with-cxx='cl' --with-mpi=0 --with-debugging=1 --download-f2cblaslapack=/path/to/f2cblaslapack-3.1.1.q.tar.gz -CFLAGS='-MDd -wd4996' -CXXFLAGS='-MDd -wd4996' But it seems that f2cblaslapack does not recognize "-MDd" completely. Some files are still compiled with '-MD' instead of "-MDd". This will lead to a problem that not all libraries are linked with the same MS C library. I think it may be caused by the variable "CNOOPT" is not set properly in config/BuildSystem/config/packages/f2cblaslapack.py and config/BuildSystem/config/packages/BlasLapack.py. Only '-MD' is present there. See def getWindowsNonOptFlags(self,cflags): for flag in ['-MT','-MTd','-MD','-threads']: if cflags.find(flag) >=0: return flag return '' If I substitute '-MD' with '-MDd', then it works. But this will lead to problem when '-MD' is preferred. I am wondering if anyone can fix this problem. Thanks, Mengda -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Dec 18 13:11:47 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 18 Dec 2011 22:41:47 +0330 Subject: [petsc-users] To slow just in first step of SNES Message-ID: Dear Developers, I tried to following optimized matrix data setup for a block-sparse mat solver but when I run the code, too much time is taken "just" for the first step of SNES procedure (about 20 to 30 times than the next steps ?! ). I did above as follows: 1- Allocate matrix( matrices ): MatSetType ( JAC, MATSEQBAIJ ); MatSetSizes ( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); MatSeqBAIJSetPreallocation ( JAC, tot, 0, nnz ); MatSetFromOptions ( JAC ); where nnz[] is account regarded to connectivity 2- Matrix Data calculation: MatSetValues( JAC, tot, IDX, tot, JDX, VAL, ADD_VALUES ); where IDX, JDX, VAL are blocks of data with size tot. 3- Did the same for preconditioning matrix. 4- used PCILU for preconditioning. I also print out log_summary but I actually can't find out if I set the memory optimized ( I don't no how to follow the logfile and check what parameters ). Please let me know how would be the failure in my procedure... Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Dec 18 13:14:49 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 18 Dec 2011 13:14:49 -0600 (CST) Subject: [petsc-users] Petsc compilation with -MDd using visual studio c++ 2008 In-Reply-To: References: Message-ID: On Sun, 18 Dec 2011, Mengda Wu wrote: > Hi all, > > I am trying to compile with a debug version of Petsc with -MDd options > using visual studio c++ 2008. The following options are used. > > ./config/configure.py --with-cc='cl' --with-fc=0 --with-cxx='cl' > --with-mpi=0 --with-debugging=1 > --download-f2cblaslapack=/path/to/f2cblaslapack-3.1.1.q.tar.gz > -CFLAGS='-MDd -wd4996' -CXXFLAGS='-MDd -wd4996' > > But it seems that f2cblaslapack does not recognize "-MDd" completely. Some > files are still compiled with '-MD' instead of "-MDd". This will lead to a > problem that not all libraries are linked with the same MS C library. I > think it may be caused by the variable "CNOOPT" is not set properly in > config/BuildSystem/config/packages/f2cblaslapack.py and > config/BuildSystem/config/packages/BlasLapack.py. Only '-MD' is present > there. See > > def getWindowsNonOptFlags(self,cflags): > for flag in ['-MT','-MTd','-MD','-threads']: > if cflags.find(flag) >=0: return flag > return '' > > If I substitute '-MD' with '-MDd', then it works. But this will lead to > problem when '-MD' is preferred. > You can just add 'MDd' to that list. i.e for flag in ['-MT','-MTd','-MD','-MDd','-threads']: Will add this for the next patch update to petsc-3.2 Satish > I am wondering if anyone can fix this problem. > > Thanks, > Mengda > From bsmith at mcs.anl.gov Sun Dec 18 13:19:08 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 18 Dec 2011 13:19:08 -0600 Subject: [petsc-users] To slow just in first step of SNES In-Reply-To: References: Message-ID: Run with -info and search for malloc this will show you if you have not preallocated enough space. You can put this in the code after you preallocate the matrix but before calling MatAssembly and it will stop as soon as it needs to malloc more memory during MatSetValues() showing you have not properly preallocated MatSetOption(Mat A,MAT NO NEW NONZERO LOCATIONS,PETSC TRUE); Barry On Dec 18, 2011, at 1:11 PM, behzad baghapour wrote: > Dear Developers, > > I tried to following optimized matrix data setup for a block-sparse mat solver but when I run the code, too much time is taken "just" for the first step of SNES procedure (about 20 to 30 times than the next steps ?! ). > > I did above as follows: > > 1- Allocate matrix( matrices ): > > MatSetType ( JAC, MATSEQBAIJ ); > MatSetSizes ( JAC, PETSC_DECIDE, PETSC_DECIDE, nt, nt ); > MatSeqBAIJSetPreallocation ( JAC, tot, 0, nnz ); > MatSetFromOptions ( JAC ); > > where nnz[] is account regarded to connectivity > > 2- Matrix Data calculation: > > MatSetValues( JAC, tot, IDX, tot, JDX, VAL, ADD_VALUES ); > > where IDX, JDX, VAL are blocks of data with size tot. > > 3- Did the same for preconditioning matrix. > > 4- used PCILU for preconditioning. > > I also print out log_summary but I actually can't find out if I set the memory optimized ( I don't no how to follow the logfile and check what parameters ). > > Please let me know how would be the failure in my procedure... > > Thanks a lot, > BehZad > From behzad.baghapour at gmail.com Sun Dec 18 13:32:35 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 18 Dec 2011 23:02:35 +0330 Subject: [petsc-users] To slow just in first step of SNES In-Reply-To: References: Message-ID: Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Dec 18 19:54:00 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 18 Dec 2011 17:54:00 -0800 Subject: [petsc-users] training In-Reply-To: References: <04649ABFF695C94F8E6CF3BBBA9B1665B4F14B@BHEXMBX1.livad.liv.ac.uk> <04649ABFF695C94F8E6CF3BBBA9B1665B4F180@BHEXMBX1.livad.liv.ac.uk> Message-ID: Victor is teaching one at TACC on Jan 17 and I'm doing an advanced course there on Feb 20. On Dec 18, 2011 7:42 AM, "Matthew Knepley" wrote: > On Sun, Dec 18, 2011 at 9:40 AM, Kharche, Sanjay < > Sanjay.Kharche at liverpool.ac.uk> wrote: > >> >> Thanks Matt, I am already working thru' the tutorials. I am also looking >> for training events to attend as it is part of the project. >> > > We do not currently have any scheduled. When we do, we will post them. We > do sometimes respond to requests > for tutorials. > > Matt > > >> thanks >> Sanjay >> >> ------------------------------ >> *From:* petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] >> on behalf of Matthew Knepley [knepley at gmail.com] >> *Sent:* 18 December 2011 14:36 >> *To:* PETSc users list >> *Subject:* Re: [petsc-users] training >> >> On Sun, Dec 18, 2011 at 1:54 AM, Kharche, Sanjay < >> Sanjay.Kharche at liverpool.ac.uk> wrote: >> >>> >>> Hi >>> >>> I am a new user of PetSc and would like to find training events for >>> myself and members of the project I am involved in. Can someone suggest any >>> please. >>> >> >> You can look at the tutorials we have given: >> >> http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html >> >> Matt >> >> >>> thanks >>> Sanjay >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Dec 19 02:48:33 2011 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 19 Dec 2011 08:48:33 +0000 Subject: [petsc-users] openmpi 1.4.4 Message-ID: Thanks Satish, I'll give it a try. Chris Date: Fri, 16 Dec 2011 07:21:12 -0600 (CST) From: Satish Balay Subject: Re: [petsc-users] openmpi 1.4.4 To: PETSc users list Message-ID: Content-Type: TEXT/PLAIN; charset=US-ASCII At the relase time of petsc-3.2 openmpi-1.4.3 was the latest. And we generally don't update the default versions of externalpackages for a release version of petsc. However if the installation of 1.4.4 hasn't changed - the following might work for you: --download-openmpi=http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.4.tar.gz Satish dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From sreejithpk at gmail.com Mon Dec 19 04:23:57 2011 From: sreejithpk at gmail.com (Sreejith Pulloor Kuttanikkad) Date: Mon, 19 Dec 2011 11:23:57 +0100 Subject: [petsc-users] training Message-ID: > On Sun, Dec 18, 2011 at 5:54 AM, Kharche, Sanjay > wrote: > > > > Hi > > > > I am a new user of PetSc and would like to find training events for > myself > > and members of the project I am involved in. Can someone suggest any > please. > > > > thanks > > Sanjay > > > > Hello, I am also new to Petsc and interested in any training courses or workshops. Thank you Sreejith -------------- next part -------------- An HTML attachment was scrubbed... URL: From modhurita at gmail.com Mon Dec 19 12:21:58 2011 From: modhurita at gmail.com (Modhurita Mitra) Date: Mon, 19 Dec 2011 12:21:58 -0600 Subject: [petsc-users] Pseudoinverse of a large matrix Message-ID: Hi, I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc compute the SVD of this matrix without parallelization? If parallelization is needed, do I need to use SLEPc? Thanks, Modhurita -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 19 12:31:20 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 19 Dec 2011 12:31:20 -0600 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: > Hi, > > I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc > compute the SVD of this matrix without parallelization? If parallelization > is needed, do I need to use SLEPc? > With enough memory, yes. However, I am not sure you want to wait. I am not sure how SLEPc would help here. >From the very very little detail you have given, you would need parallel linear algebra, like Elemental. However, I would start out from a more fundamental viewpoint. Such as replacing "compute the psuedoinverse" with "solve a least-squares problem" if that is indeed the case. Matt > > Thanks, > Modhurita > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From modhurita at gmail.com Mon Dec 19 13:00:38 2011 From: modhurita at gmail.com (Modhurita Mitra) Date: Mon, 19 Dec 2011 13:00:38 -0600 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: References: Message-ID: I am trying to express the radiation pattern of an antenna in terms of spherical harmonic basis functions. I have radiation pattern data at N=324360 points. Therefore, my matrix is spherical harmonic basis functions evaluated till order sqrt(N) (which makes up at total of N elements), evaluated at N data points. So this is a linear least squares problem, and I have been trying to solve it by finding its pseudoinverse which uses SVD. The elements of the matrix are complex, and the matrix is non-sparse. I have solved it in MATLAB using a subsample of the data, but MATLAB runs out of memory while calculating the SVD at input matrix size 2500 X 2500. I need to solve this problem using the entire data. I was thinking of using SLEPc because I found a ready-to-use code which computes the SVD of a complex-valued matrix ( http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I am trying to figure out where to start and what I should learn. Thanks, Modhurita On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley wrote: > On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: > >> Hi, >> >> I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can >> PETSc compute the SVD of this matrix without parallelization? If >> parallelization is needed, do I need to use SLEPc? >> > > With enough memory, yes. However, I am not sure you want to wait. I am not > sure how SLEPc would help here. > From the very very little detail you have given, you would need parallel > linear algebra, like Elemental. However, > I would start out from a more fundamental viewpoint. Such as replacing > "compute the psuedoinverse" with > "solve a least-squares problem" if that is indeed the case. > > Matt > > >> >> Thanks, >> Modhurita >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Mon Dec 19 13:18:03 2011 From: juhaj at iki.fi (Juha =?utf-8?q?J=C3=A4ykk=C3=A4?=) Date: Mon, 19 Dec 2011 19:18:03 +0000 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: References: <201112171455.57152.juhaj@iki.fi> <201112172255.50679.juhaj@iki.fi> Message-ID: <201112191918.13791.juhaj@iki.fi> > I guarantee you this is not a bug. This stuff is used by thousands of > people every day. That's what I thought, too. > Its very simple and old code. There is a misunderstanding somewhere in your > code about how this mechanism works. I suggest stripping down the code until > there are just two vectors and you alter one. Stripping it down will take a while, it is a relatively long piece, but I did notice something that strikes me as odd: What is the correct Fields.hdr.state at the point before calling DAGlobalToLocalBegin()? My two vectors have a different value there and I suspect that might be the reason. (The other vec has state 4, the other has If that is so, how can that happen? I have checked quite a few times that all DAGlobalToLocalBegin's have the corresponding End-call, that I pair every DAVecGetArray with DAVecRestoreArray, and every DAGetLocalVector with DARestoreLocalVector. Does the order in which I process the vectors matter here? The order of DAGetLocalVector, DAGlobalToLocalBegin/End, and DAVecGetArray for each vector obviously matters. Cheers, Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Mon Dec 19 13:21:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 19 Dec 2011 13:21:48 -0600 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: References: Message-ID: On Mon, Dec 19, 2011 at 1:00 PM, Modhurita Mitra wrote: > I am trying to express the radiation pattern of an antenna in terms of > spherical harmonic basis functions. I have radiation pattern data at > N=324360 points. Therefore, my matrix is spherical harmonic basis functions > evaluated till order sqrt(N) (which makes up at total of N elements), > evaluated at N data points. So this is a linear least squares problem, and > I have been trying to solve it by finding its pseudoinverse which uses SVD. > The elements of the matrix are complex, and the matrix is non-sparse. I > have solved it in MATLAB using a subsample of the data, but MATLAB runs out > of memory while calculating the SVD at input matrix size 2500 X 2500. I > need to solve this problem using the entire data. > > I was thinking of using SLEPc because I found a ready-to-use code which > computes the SVD of a complex-valued matrix ( > http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I > am trying to figure out where to start and what I should learn. > 1) You could try Elemental in parallel (would probably use QR) 2) You could try LSQR in PETSc (preconditioner isa mystery) 3) The "right" solution for large LS problems is probably this compressed sensing approach (there is a great review on arXiv by Martinsson). Matt > Thanks, > Modhurita > > > On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley wrote: > >> On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: >> >>> Hi, >>> >>> I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can >>> PETSc compute the SVD of this matrix without parallelization? If >>> parallelization is needed, do I need to use SLEPc? >>> >> >> With enough memory, yes. However, I am not sure you want to wait. I am >> not sure how SLEPc would help here. >> From the very very little detail you have given, you would need parallel >> linear algebra, like Elemental. However, >> I would start out from a more fundamental viewpoint. Such as replacing >> "compute the psuedoinverse" with >> "solve a least-squares problem" if that is indeed the case. >> >> Matt >> >> >>> >>> Thanks, >>> Modhurita >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 19 13:26:42 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 19 Dec 2011 13:26:42 -0600 Subject: [petsc-users] Vec entries altered incorrectly In-Reply-To: <201112191918.13791.juhaj@iki.fi> References: <201112171455.57152.juhaj@iki.fi> <201112172255.50679.juhaj@iki.fi> <201112191918.13791.juhaj@iki.fi> Message-ID: On Mon, Dec 19, 2011 at 1:18 PM, Juha J?ykk? wrote: > > I guarantee you this is not a bug. This stuff is used by thousands of > > people every day. > > That's what I thought, too. > > > Its very simple and old code. There is a misunderstanding somewhere in > your > > code about how this mechanism works. I suggest stripping down the code > until > > there are just two vectors and you alter one. > > Stripping it down will take a while, it is a relatively long piece, but I > did > notice something that strikes me as odd: > > What is the correct Fields.hdr.state at the point before calling > DAGlobalToLocalBegin()? My two vectors have a different value there and I > suspect that might be the reason. (The other vec has state 4, the other has > I assume you mean the 'state' field for the 'hdr' struct. This is just a counter that gets incremented when something changes, so cached values can be recomputed. Matt > If that is so, how can that happen? I have checked quite a few times that > all > DAGlobalToLocalBegin's have the corresponding End-call, that I pair every > DAVecGetArray with DAVecRestoreArray, and every DAGetLocalVector with > DARestoreLocalVector. Does the order in which I process the vectors matter > here? The order of DAGetLocalVector, DAGlobalToLocalBegin/End, and > DAVecGetArray for each vector obviously matters. > > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetan.jhurani at gmail.com Mon Dec 19 13:35:27 2011 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Mon, 19 Dec 2011 11:35:27 -0800 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: References: Message-ID: <4eef9203.2515640a.0d20.ffffcfd7@mx.google.com> I'm assuming below that the matrix is either badly conditioned or is rank-deficient with unknown null-space which made you choose SVD. If none of these assumptions is true, better methods exist that can use the null-space information. Anyway. It is faster to use Rank-Revealing QR instead of SVD for pseudoinverse. You can use the attached Matlab script, which computes pseudoinverse and left and right null-spaces using RRQR, and compare the time with Matlab's SVD based pinv. Note that it computes the full pseudoinverse matrix instead of solving for a specific right hand side (which would be faster and not frowned upon). Couple of optimizations not there currently. 1. It can be changed to use sparse RRQR factorization for sparse input matrix. 2. It can be further optimized using the Woodbury identity for two cases - rank <= size or rank >= size to reduce the size of auxiliary internal Cholesky solve. Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Modhurita Mitra Sent: Monday, December 19, 2011 11:01 AM To: PETSc users list Subject: Re: [petsc-users] Pseudoinverse of a large matrix I am trying to express the radiation pattern of an antenna in terms of spherical harmonic basis functions. I have radiation pattern data at N=324360 points. Therefore, my matrix is spherical harmonic basis functions evaluated till order sqrt(N) (which makes up at total of N elements), evaluated at N data points. So this is a linear least squares problem, and I have been trying to solve it by finding its pseudoinverse which uses SVD. The elements of the matrix are complex, and the matrix is non-sparse. I have solved it in MATLAB using a subsample of the data, but MATLAB runs out of memory while calculating the SVD at input matrix size 2500 X 2500. I need to solve this problem using the entire data. I was thinking of using SLEPc because I found a ready-to-use code which computes the SVD of a complex-valued matrix ( http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html ). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I am trying to figure out where to start and what I should learn. Thanks, Modhurita On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley wrote: On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: Hi, I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc compute the SVD of this matrix without parallelization? If parallelization is needed, do I need to use SLEPc? With enough memory, yes. However, I am not sure you want to wait. I am not sure how SLEPc would help here. >From the very very little detail you have given, you would need parallel linear algebra, like Elemental. However, I would start out from a more fundamental viewpoint. Such as replacing "compute the psuedoinverse" with "solve a least-squares problem" if that is indeed the case. Matt Thanks, Modhurita -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pinv_rrqr.m Type: application/mathematica Size: 1724 bytes Desc: not available URL: From chetan.jhurani at gmail.com Mon Dec 19 13:38:34 2011 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Mon, 19 Dec 2011 11:38:34 -0800 Subject: [petsc-users] Pseudoinverse of a large matrix References: Message-ID: <4eef92ba.67b4ec0a.40e6.3c8a@mx.google.com> > It can be further optimized using the Woodbury identity for two cases - > rank <= size or rank >= size to reduce the size of auxiliary internal Cholesky solve. Sorry, that's wrong. I meant rank <= size/2 or rank >= size/2. Size here refers to square matrix size, but this analysis can be done for rectangular case too. Chetan From: Chetan Jhurani [mailto:chetan.jhurani at gmail.com] Sent: Monday, December 19, 2011 11:35 AM To: 'PETSc users list' Subject: RE: [petsc-users] Pseudoinverse of a large matrix Couple of optimizations not there currently. 1. It can be changed to use sparse RRQR factorization for sparse input matrix. 2. It can be further optimized using the Woodbury identity for two cases - rank <= size or rank >= size to reduce the size of auxiliary internal Cholesky solve. Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Modhurita Mitra Sent: Monday, December 19, 2011 11:01 AM To: PETSc users list Subject: Re: [petsc-users] Pseudoinverse of a large matrix I am trying to express the radiation pattern of an antenna in terms of spherical harmonic basis functions. I have radiation pattern data at N=324360 points. Therefore, my matrix is spherical harmonic basis functions evaluated till order sqrt(N) (which makes up at total of N elements), evaluated at N data points. So this is a linear least squares problem, and I have been trying to solve it by finding its pseudoinverse which uses SVD. The elements of the matrix are complex, and the matrix is non-sparse. I have solved it in MATLAB using a subsample of the data, but MATLAB runs out of memory while calculating the SVD at input matrix size 2500 X 2500. I need to solve this problem using the entire data. I was thinking of using SLEPc because I found a ready-to-use code which computes the SVD of a complex-valued matrix ( http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html ). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I am trying to figure out where to start and what I should learn. Thanks, Modhurita On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley wrote: On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: Hi, I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc compute the SVD of this matrix without parallelization? If parallelization is needed, do I need to use SLEPc? With enough memory, yes. However, I am not sure you want to wait. I am not sure how SLEPc would help here. >From the very very little detail you have given, you would need parallel linear algebra, like Elemental. However, I would start out from a more fundamental viewpoint. Such as replacing "compute the psuedoinverse" with "solve a least-squares problem" if that is indeed the case. Matt Thanks, Modhurita -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Mon Dec 19 13:50:28 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Mon, 19 Dec 2011 13:50:28 -0600 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: <4eef92ba.67b4ec0a.40e6.3c8a@mx.google.com> References: <4eef92ba.67b4ec0a.40e6.3c8a@mx.google.com> Message-ID: Chetan, I completely agree that (rank-revealing) QR should be the first choice. As a side note, from what he has said, his matrix is actually dense. If his matrix was sparse, I am not sure how much sparsity would be lost by the column pivoting inherent in a rank-revealing QR. I know that the MUMPS group is either working on or has finished a sparse QR, but I don't know any details about their approach to pivoting (though I would be very interested!). Hopefully it could simply reuse the threshold approach used for sparse LU and LDL. Jack On Mon, Dec 19, 2011 at 1:38 PM, Chetan Jhurani wrote: > > It can be further optimized using the Woodbury identity for two cases ?* > *** > > > rank <= size or rank >= size to reduce the size of auxiliary internal > Cholesky solve.**** > > ** ** > > Sorry, that?s wrong. I meant rank <= size/2 or rank >= size/2. Size here > **** > > refers to square matrix size, but this analysis can be done for rectangular > **** > > case too.**** > > ** ** > > Chetan**** > > ** ** > > *From:* Chetan Jhurani [mailto:chetan.jhurani at gmail.com] > *Sent:* Monday, December 19, 2011 11:35 AM > *To:* 'PETSc users list' > *Subject:* RE: [petsc-users] Pseudoinverse of a large matrix**** > > ** ** > > ** ** > > Couple of optimizations not there currently.**** > > ** ** > > **1. **It can be changed to use sparse RRQR factorization for sparse > input matrix.**** > > **2. **It can be further optimized using the Woodbury identity for two > cases ?**** > > rank <= size or rank >= size to reduce the size of auxiliary internal > Cholesky solve.**** > > ** ** > > Chetan**** > > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [ > mailto:petsc-users-bounces at mcs.anl.gov ] > *On Behalf Of *Modhurita Mitra > *Sent:* Monday, December 19, 2011 11:01 AM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Pseudoinverse of a large matrix**** > > ** ** > > I am trying to express the radiation pattern of an antenna in terms of > spherical harmonic basis functions. I have radiation pattern data at > N=324360 points. Therefore, my matrix is spherical harmonic basis functions > evaluated till order sqrt(N) (which makes up at total of N elements), > evaluated at N data points. So this is a linear least squares problem, and > I have been trying to solve it by finding its pseudoinverse which uses SVD. > The elements of the matrix are complex, and the matrix is non-sparse. I > have solved it in MATLAB using a subsample of the data, but MATLAB runs out > of memory while calculating the SVD at input matrix size 2500 X 2500. I > need to solve this problem using the entire data. > > > I was thinking of using SLEPc because I found a ready-to-use code which > computes the SVD of a complex-valued matrix ( > http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I > am trying to figure out where to start and what I should learn. > > Thanks, > Modhurita**** > > On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley > wrote:**** > > On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra > wrote:**** > > Hi, > > I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc > compute the SVD of this matrix without parallelization? If parallelization > is needed, do I need to use SLEPc?**** > > ** ** > > With enough memory, yes. However, I am not sure you want to wait. I am not > sure how SLEPc would help here.**** > > From the very very little detail you have given, you would need parallel > linear algebra, like Elemental. However,**** > > I would start out from a more fundamental viewpoint. Such as replacing > "compute the psuedoinverse" with**** > > "solve a least-squares problem" if that is indeed the case.**** > > ** ** > > Matt**** > > **** > > > Thanks, > Modhurita**** > > > > **** > > ** ** > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetan.jhurani at gmail.com Mon Dec 19 14:29:20 2011 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Mon, 19 Dec 2011 12:29:20 -0800 Subject: [petsc-users] Pseudoinverse of a large matrix In-Reply-To: References: <4eef92ba.67b4ec0a.40e6.3c8a@mx.google.com> Message-ID: <4eef9ea1.0b6d650a.166e.ffffd8e9@mx.google.com> Jack, The RRQR for sparse matrices comment was for generalizing the script if anyone else wanted to use it for serious business. My personal experience (and very qualitative here) with this package http://www.cise.ufl.edu/research/sparse/SPQR/, which also has easy to use Matlab wrappers, is that RRQR does lead to larger fill-in for 2D and 3D 9/27 point stencils compared to LU, for example. But it was faster than dense QR once the matrix size increased beyond a few hundreds. To repeat the well-known - it heavily depends on the original pattern. SPQR is for shared memory, and it's likely that you knew about it already. Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jack Poulson Sent: Monday, December 19, 2011 11:50 AM To: PETSc users list Subject: Re: [petsc-users] Pseudoinverse of a large matrix Chetan, I completely agree that (rank-revealing) QR should be the first choice. As a side note, from what he has said, his matrix is actually dense. If his matrix was sparse, I am not sure how much sparsity would be lost by the column pivoting inherent in a rank-revealing QR. I know that the MUMPS group is either working on or has finished a sparse QR, but I don't know any details about their approach to pivoting (though I would be very interested!). Hopefully it could simply reuse the threshold approach used for sparse LU and LDL. Jack On Mon, Dec 19, 2011 at 1:38 PM, Chetan Jhurani wrote: > It can be further optimized using the Woodbury identity for two cases - > rank <= size or rank >= size to reduce the size of auxiliary internal Cholesky solve. Sorry, that's wrong. I meant rank <= size/2 or rank >= size/2. Size here refers to square matrix size, but this analysis can be done for rectangular case too. Chetan From: Chetan Jhurani [mailto:chetan.jhurani at gmail.com] Sent: Monday, December 19, 2011 11:35 AM To: 'PETSc users list' Subject: RE: [petsc-users] Pseudoinverse of a large matrix Couple of optimizations not there currently. 1. It can be changed to use sparse RRQR factorization for sparse input matrix. 2. It can be further optimized using the Woodbury identity for two cases - rank <= size or rank >= size to reduce the size of auxiliary internal Cholesky solve. Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Modhurita Mitra Sent: Monday, December 19, 2011 11:01 AM To: PETSc users list Subject: Re: [petsc-users] Pseudoinverse of a large matrix I am trying to express the radiation pattern of an antenna in terms of spherical harmonic basis functions. I have radiation pattern data at N=324360 points. Therefore, my matrix is spherical harmonic basis functions evaluated till order sqrt(N) (which makes up at total of N elements), evaluated at N data points. So this is a linear least squares problem, and I have been trying to solve it by finding its pseudoinverse which uses SVD. The elements of the matrix are complex, and the matrix is non-sparse. I have solved it in MATLAB using a subsample of the data, but MATLAB runs out of memory while calculating the SVD at input matrix size 2500 X 2500. I need to solve this problem using the entire data. I was thinking of using SLEPc because I found a ready-to-use code which computes the SVD of a complex-valued matrix ( http://www.grycap.upv.es/slepc/documentation/current/src/svd/examples/tutorials/ex14.c.html ). I don't know how to use either PETSc or SLEPc (or Elemental) yet, so I am trying to figure out where to start and what I should learn. Thanks, Modhurita On Mon, Dec 19, 2011 at 12:31 PM, Matthew Knepley wrote: On Mon, Dec 19, 2011 at 12:21 PM, Modhurita Mitra wrote: Hi, I have to compute the pseudoinverse of a 324360 X 324360 matrix. Can PETSc compute the SVD of this matrix without parallelization? If parallelization is needed, do I need to use SLEPc? With enough memory, yes. However, I am not sure you want to wait. I am not sure how SLEPc would help here. >From the very very little detail you have given, you would need parallel linear algebra, like Elemental. However, I would start out from a more fundamental viewpoint. Such as replacing "compute the psuedoinverse" with "solve a least-squares problem" if that is indeed the case. Matt Thanks, Modhurita -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbacks at mcmillan-mcgee.com Mon Dec 19 18:21:00 2011 From: jbacks at mcmillan-mcgee.com (Jonathan Backs) Date: Mon, 19 Dec 2011 17:21:00 -0700 Subject: [petsc-users] Mat-view-draw Colors, Views Message-ID: <3AFBF632-9AB2-452E-9A21-A98627784005@mcmillan-mcgee.com> Hi, I am trying to troubleshoot my Jacobian using the command-line option -mat_view_draw. In the X-window that pops up, the non-zero structure is shown using three colors: cyan, blue, and red. What do these three colors represent? When I add the option -draw_pause -1, I am given three chances to right-click on the X-window for each Jacobian evaluation. Each time, a different view of the Jacobian is shown, though the first two views look the same. What do these different views show? Thank you for your time, Jonathan From knepley at gmail.com Mon Dec 19 18:26:27 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 19 Dec 2011 18:26:27 -0600 Subject: [petsc-users] Mat-view-draw Colors, Views In-Reply-To: <3AFBF632-9AB2-452E-9A21-A98627784005@mcmillan-mcgee.com> References: <3AFBF632-9AB2-452E-9A21-A98627784005@mcmillan-mcgee.com> Message-ID: On Mon, Dec 19, 2011 at 6:21 PM, Jonathan Backs wrote: > Hi, > > I am trying to troubleshoot my Jacobian using the command-line option > -mat_view_draw. In the X-window that pops up, the non-zero structure is > shown using three colors: cyan, blue, and red. What do these three colors > represent? > +, -, 0. > When I add the option -draw_pause -1, I am given three chances to > right-click on the X-window for each Jacobian evaluation. Each time, a > different view of the Jacobian is shown, though the first two views look > the same. What do these different views show? > Every time a matrix is assembled, it is shown. Matt > Thank you for your time, > > Jonathan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Dec 19 18:30:27 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 19 Dec 2011 18:30:27 -0600 (CST) Subject: [petsc-users] Mat-view-draw Colors, Views In-Reply-To: <3AFBF632-9AB2-452E-9A21-A98627784005@mcmillan-mcgee.com> References: <3AFBF632-9AB2-452E-9A21-A98627784005@mcmillan-mcgee.com> Message-ID: On Mon, 19 Dec 2011, Jonathan Backs wrote: > Hi, > > I am trying to troubleshoot my Jacobian using the command-line option -mat_view_draw. In the X-window that pops up, the non-zero structure is shown using three colors: cyan, blue, and red. What do these three colors represent? red : positive value bule: negative value cyan: zero value [thats entered with matsetvalues] > > When I add the option -draw_pause -1, I am given three chances to right-click on the X-window for each Jacobian evaluation. Each time, a different view of the Jacobian is shown, though the first two views look the same. What do these different views show? left mouse: zoom-in middle mouse: zoom-out right-mouse: continue the program So if you click the right-mouse - then the program continues - and what you see next is the 'next' matrix thats assembled in your program. It could be a factor matrix or something thats created in the preconditioner. This depends on the path your code takes. Satish > > Thank you for your time, > > Jonathan From mmnasr at gmail.com Mon Dec 19 20:48:04 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Mon, 19 Dec 2011 18:48:04 -0800 Subject: [petsc-users] boomerAmg scalability In-Reply-To: <003701ccbb45$f2bb23f0$d8316bd0$@com> References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> Message-ID: Hi Ravi, I have been using BoomerAMG as a preconditioner joint with an iterative solver, e.g. GMRES of BiCGs for regular 3D CFD problems. On the top of my head, I can not remember if I had the strong scaling tests done (I will look into it and let you know if you found any), but for the weak-scaling case, I definitely saw some scaling issues. As the size of the system increases, the number of iterations does also increase (somewhat significantly for my test problem, i.e. incompressible N-S with complex geometry) which ultimately deteriorates the weak-scaling behaviors. This is also reported in the reports given by the hypre team, cf. https://computation.llnl.gov/casc/linear_solvers/pubs/pmis_report.pdf (cf. see Table 6.1 for the Stokes flow simulation results and scaling). Also, inherited with the multigrid nature, there are always fine-tuning factors which are, unfortunately, problem dependent. Good luck, Mohamad On Thu, Dec 15, 2011 at 8:23 AM, Ravi Kannan wrote: > Dear All,**** > > ** ** > > This is Ravi Kannan from CFD Research Corporation. Recently, we are > experimenting with the BoomerAMG preconditioner for some ?stiff? CFD > problems. In that regard, all the other standard solver-preconditioner > combinations failed for the current CFD problem. The boomer is the only one > which is able to provide with ?converged? solutions.**** > > ** ** > > We noticed that the scalability of this boomer preconditioner is really > poor. For instance, even with a cell size of 2 million, we cannot scale to > even 16 partitions (in contrast, the other solver-preconditioner > combinations like the BI-CGS/BJacobi gave good enough scalability).**** > > ** ** > > Are we missing something? Do we need to use a more latest version of > boomer?**** > > ** ** > > Thanks,**** > > Ravi. **** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* petsc-dev-bounces at mcs.anl.gov [mailto: > petsc-dev-bounces at mcs.anl.gov] *On Behalf Of *Bobby Philip > *Sent:* Thursday, December 15, 2011 9:22 AM > *To:* For users of the development version of PETSc > *Subject:* Re: [petsc-dev] controlling vector values while doing matrix > free operations**** > > ** ** > > Hmmmm..sorry guys - the entire thread got put away in a folder by my mail > reader and I just discovered all your emails :-)**** > > ** ** > > SNESVI looks interesting but as someone pointed out I am using matrix > free. And as I understand from Barry a matrix free version of SNESVI is not > implemented. The idea of flipping the sign seems to be a poor mans approach > that might work for me though I would still need some mechanisms in SNES to > put the hooks in.**** > > ** ** > > Bobby**** > > ** ** > > On Dec 14, 2011, at 10:45 PM, Jed Brown wrote:**** > > > > **** > > On Wed, Dec 14, 2011 at 19:40, Dmitry Karpeev wrote: > **** > > The trouble is that the constraints can get "eliminated" only when they > become active. **** > > ** ** > > That's not what I meant. I meant to transform the algebraic system so that > those extra variables were eliminated. The point is that we tend to put a > lot of effort into designing effective preconditioners for a standard > formulation (e.g. conservative variables), and that is partly lost of we > have this other system in which the evaluations of constitutive relations > are added explicitly.**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Dec 19 21:17:24 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 19 Dec 2011 19:17:24 -0800 Subject: [petsc-users] boomerAmg scalability In-Reply-To: References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> Message-ID: On Mon, Dec 19, 2011 at 18:48, Mohamad M. Nasr-Azadani wrote: > I have been using BoomerAMG as a preconditioner joint with an iterative > solver, e.g. GMRES of BiCGs for regular 3D CFD problems. > On the top of my head, I can not remember if I had the strong scaling > tests done (I will look into it and let you know if you found any), but for > the weak-scaling case, I definitely saw some scaling issues. > As the size of the system increases, the number of iterations does also > increase (somewhat significantly for my test problem, i.e. incompressible > N-S with complex geometry) which ultimately deteriorates the weak-scaling > behaviors. > It is very dangerous to use AMG directly on the indefinite problem produced by most discretizations of incompressible flow. For example, for mixed finite element methods, I have on multiple occasions observed BoomerAMG produce a singular preconditioner (with huge null space), leading to the appearance of convergence in the preconditioned norm, but no actual convergence. > > This is also reported in the reports given by the hypre team, cf. > https://computation.llnl.gov/casc/linear_solvers/pubs/pmis_report.pdf > > (cf. see Table 6.1 for the Stokes flow simulation results and scaling). > This table is for a very special discretization, First Order System Least Squares (FOSLS). Advocates of FOSLS like to point out that the method does not require inf-sup compatibility between velocity and pressure spaces, so equal-order spaces can, in principle, be used, without harming the "optimal" convergence rates. The problem is that equal order spaces, which are almost always used in practice, cause systematic conservation errors. Even for very simple geometries, "incompressible" flow solutions can exhibit more than 90% mass loss. If you look at the literature, you will notice that many of the most prominent advocates of FOSLS were not forthcoming about this "detail" for the first decade of their publications. In the last five years, they have published on "enhanced mass conservation" techniques, with which, still for simple, well-resolved flow problems, they manage to get less than 20% mass loss with linear elements and less than 1% by using higher order elements (up to quartic). I have yet to meet an engineer who would consider such systematic mass loss acceptable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 19 22:57:39 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 19 Dec 2011 22:57:39 -0600 Subject: [petsc-users] boomerAmg scalability In-Reply-To: References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> Message-ID: <56F930DC-6856-4F2A-89A9-D8FBD53E9511@mcs.anl.gov> On Dec 19, 2011, at 9:17 PM, Jed Brown wrote: > On Mon, Dec 19, 2011 at 18:48, Mohamad M. Nasr-Azadani wrote: > I have been using BoomerAMG as a preconditioner joint with an iterative solver, e.g. GMRES of BiCGs for regular 3D CFD problems. > On the top of my head, I can not remember if I had the strong scaling tests done (I will look into it and let you know if you found any), but for the weak-scaling case, I definitely saw some scaling issues. > As the size of the system increases, the number of iterations does also increase (somewhat significantly for my test problem, i.e. incompressible N-S with complex geometry) which ultimately deteriorates the weak-scaling behaviors. > > It is very dangerous to use AMG directly on the indefinite problem produced by most discretizations of incompressible flow. For example, for mixed finite element methods, I have on multiple occasions observed BoomerAMG produce a singular preconditioner (with huge null space), leading to the appearance of convergence in the preconditioned norm, but no actual convergence. > So please tell use how we SHOULD use AMG with those "indefinite problem produced by most discretizations of incompressible flow" dear teacher :-) Barry From jedbrown at mcs.anl.gov Mon Dec 19 23:09:02 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 19 Dec 2011 21:09:02 -0800 Subject: [petsc-users] boomerAmg scalability In-Reply-To: <56F930DC-6856-4F2A-89A9-D8FBD53E9511@mcs.anl.gov> References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> <56F930DC-6856-4F2A-89A9-D8FBD53E9511@mcs.anl.gov> Message-ID: On Mon, Dec 19, 2011 at 20:57, Barry Smith wrote: > So please tell use how we SHOULD use AMG with those "indefinite problem > produced by most discretizations of incompressible flow" dear teacher :-) If only there was a nice complete answer... We can do block preconditioners advocated by Elman and others. These are the most flexible and the simplest for code reuse. For low Reynolds number, they can also have optimal complexity, although the constants are usually not the best. Most variants are well-supported by PCFieldSplit (e.g. with PCLSC), but some need the user to provide auxiliary operators (e.g. the "pressure convection-diffusion" variant). We could improve support for these cases, but it's a delicate balance and I don't know any way to avoid asking the user to understand a reasonable amount about the method and usually to provide auxiliary information. We can do coupled multigrid with fieldsplit or "distributed relaxation" as a smoother. These can often be made more robust, but they tend to be more intrusive to implement. These are not usually purely algebraic due to inf-sup issues when coarsening the dual variables (pressure), though Mark Adams' work on this for contact mechanics could be used to coarsen pressure algebraically. I would like to experiment with this in PCGAMG. We can do coupled multigrid with compatible Vanka-type smoothers. Whether these are algorithmically effective and/or efficient is quite dependent on the discretization. These methods are also usually geometric, though it's possible to algebraically define a Vanka-smoother (though not necessarily efficient). This is straightforward for MAC finite differences on structured grids. For continuous finite elements, the "rotated Q1" Rannacher-Turek elements are most attractive for these smoothers, but Rannacher-Turek elements do not satisfy a discrete Korn's inequality, so they are unusable for many problems. Some variants of DG for incompressible flow seem to be the most interesting for this approach in general domains. -------------- next part -------------- An HTML attachment was scrubbed... URL: From E.T.A.vanderWeide at utwente.nl Tue Dec 20 07:24:50 2011 From: E.T.A.vanderWeide at utwente.nl (E.T.A.vanderWeide at utwente.nl) Date: Tue, 20 Dec 2011 14:24:50 +0100 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES Message-ID: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> Hi, I was wondering if it is possible to provide PETSc the matrix vector product yourself for a matrix free computation inside the SNES environment. The reason for asking is that the truncation error in the standard finite difference approach may lead to some noisy behavior for our application. We have found a nice way to get rid off the truncation error completely by using dual numbers for the creation of the Jacobian matrices. The same technique can be used for the matrix vector product inside SNES, but this means that the user has to provide this product to PETSc. To be more specific. I would like to solve the nonlinear problem F(U) = 0 via the SNES environment using the matrix free option. So similarly to the function SNESSetFunction, which sets the function evaluation routine, I'm seeking for a function that sets the function to compute the matrix vector product dF/dU times V, where U and V are inputs. Is such a function available or is there a different way to provide the matrix vector product yourself? Thanks, Edwin --------------------------------------------------- Edwin van der Weide Department of Mechanical Engineering University of Twente Enschede, the Netherlands -------------- next part -------------- An HTML attachment was scrubbed... URL: From cchoudary at rnet-tech.com Tue Dec 20 08:29:36 2011 From: cchoudary at rnet-tech.com (Chekuri S. Choudary) Date: Tue, 20 Dec 2011 09:29:36 -0500 Subject: [petsc-users] Installing PETSc with make install Message-ID: <000001ccbf23$cdc96430$695c2c90$@rnet-tech.com> What is the correct way to do make install? Is it still working? I tried the following and failed: 1) (i) cd petsc-dev (ii) ./configure --prefix=$HOME (iii) make all (iv) make install PETSC_DIR=$PWD 2) (i) cd petsc-dev (ii) ./configure (iii) Make all (iv) make install DESTDIR=$HOME -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Dec 20 08:37:52 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 20 Dec 2011 08:37:52 -0600 (CST) Subject: [petsc-users] Installing PETSc with make install In-Reply-To: <000001ccbf23$cdc96430$695c2c90$@rnet-tech.com> References: <000001ccbf23$cdc96430$695c2c90$@rnet-tech.com> Message-ID: What problems are you encountering? Send relavent logs to petsc-maint. Satish On Tue, 20 Dec 2011, Chekuri S. Choudary wrote: > What is the correct way to do make install? Is it still working? > > > > I tried the following and failed: > > > > 1) > > (i) cd petsc-dev > > (ii) ./configure --prefix=$HOME > > (iii) make all > > (iv) make install PETSC_DIR=$PWD > > > > > > 2) > > (i) cd petsc-dev > > (ii) ./configure > > (iii) Make all > > (iv) make install DESTDIR=$HOME > > > > From knepley at gmail.com Tue Dec 20 08:46:26 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 20 Dec 2011 08:46:26 -0600 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES In-Reply-To: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> Message-ID: On Tue, Dec 20, 2011 at 7:24 AM, wrote: > ** > > Hi, > > I was wondering if it is possible to provide PETSc the matrix vector > product yourself for a matrix free computation inside the SNES environment. > The reason for asking is that the truncation error in the standard finite > difference approach may lead to some noisy behavior for our application. We > have found a nice way to get rid off the truncation error completely by > using dual numbers for the creation of the Jacobian matrices. The same > technique can be used for the matrix vector product inside SNES, but this > means that the user has to provide this product to PETSc. > > To be more specific. I would like to solve the nonlinear problem F(U) = 0 > via the SNES environment using the matrix free option. So similarly to the > function SNESSetFunction, which sets the function evaluation routine, I'm > seeking for a function that sets the function to compute the matrix vector > product dF/dU times V, where U and V are inputs. Is such a function > available or is there a different way to provide the matrix vector product > yourself? > No need to use the matrix-free option. Just provide a MATSHELL ( http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateShell.html ) as the Jacobian using SNESSetJacobian(). Then your FormJacobian() function will get a vector 'x' as input. Update the context for the MATSHELL with this vector in that function, which is all it would really do. Then when the MatMult() is called on your shell matrix, compute the action using that 'x' which is stored in the context. Matt > Thanks, > > Edwin > --------------------------------------------------- > Edwin van der Weide > Department of Mechanical Engineering > University of Twente > Enschede, the Netherlands > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Thomas.Witkowski at tu-dresden.de Tue Dec 20 13:58:18 2011 From: Thomas.Witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 20 Dec 2011 20:58:18 +0100 Subject: [petsc-users] Solving indefinite systems with iterative solvers Message-ID: <20111220205818.cimn0q4i04wgkgs0@mail.zih.tu-dresden.de> Can any of you give me some advise, e.g. some link to literature, how to solve indefinite systems with iterative solvers in an efficient way? I have to solve systems coming from FEM discretization of a 6th-order time dependent PDE on structured grids. The time is discretized with a standard backward euler scheme that introduces the current timestep in one of the submatrices. This leads to some problems. When the timestep is "small", the overall system has only positive (but complex) eigenvalues. Using an apropriate krylov method this can be solved within a reasonable number of iterations. When the timestep is increased, the smallest eigenvalues become negative and much larger in their magnitude. The number of iterations required for solving this systems rise dramatically. It may be possible to increase the timestep by just 1%, but the number of iterations of solving the system rise up to three order of magnitude. Does any of you know some methods/literature which can be used to deal with such systems? If it is of interest, the overall system looks like: A 0 I 0 A tI fI+2A I A A is the discretization of the Laplace operator, I the identity operator, t is the timestep and f: R^n -> R is some function defined on the domain. Thanks for any hint, Thomas From rudolph at berkeley.edu Tue Dec 20 19:35:36 2011 From: rudolph at berkeley.edu (Max Rudolph) Date: Tue, 20 Dec 2011 17:35:36 -0800 Subject: [petsc-users] -log_summary problem Message-ID: When I run my code with the -log_summary option, it hangs indefinitely after displaying: ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 0.00164938 Is this a common problem, and if so, how do I fix it? This does not happen when I run the example programs - only my own code, so I must be at fault but without an error message I am not sure where to start. I am using petsc-3.1-p7. Thanks for your help. Max From jedbrown at mcs.anl.gov Tue Dec 20 19:38:27 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 20 Dec 2011 19:38:27 -0600 Subject: [petsc-users] -log_summary problem In-Reply-To: References: Message-ID: On Tue, Dec 20, 2011 at 19:35, Max Rudolph wrote: > When I run my code with the -log_summary option, it hangs indefinitely > after displaying: > > > ======================================================================================================================== > Average time to get PetscTime(): 9.53674e-08 > Average time for MPI_Barrier(): 0.00164938 > > Is this a common problem, and if so, how do I fix it? This does not happen > when I run the example programs - only my own code, so I must be at fault > but without an error message I am not sure where to start. I am using > petsc-3.1-p7. Thanks for your help. > Are all processes calling PetscFinalize()? How did you set -log_summary? It should be provided at the time you invoke PetscInitialize() on all processes. Try running in a debugger, then break when it hangs and print the stack trace. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Wed Dec 21 01:04:42 2011 From: wolfshow at gmail.com (Fatcharm) Date: Wed, 21 Dec 2011 15:04:42 +0800 Subject: [petsc-users] How to access the value uarray[j+2][i] Message-ID: Dear All PETSc users, I am dealing with the fuction 'FormIFunction' which is simalar to the one used in the /petsc-3.2-p5/src/ts/examples/tutorials/ex15.c For my problem, I modified the code to be: ierr = DMDAGetCorners(da,&xs,&ys,PETSC_NULL,&xm,&ym,PETSC_NULL);CHKERRQ(ierr); for (j=ys; j I am successfully solving my indefinite systems with MUMPS but only for very small problems. To give a feeling, a mere couple thousands elements. If I only double the problem size, it silently freezes, even with max verbosity via the control parameters. Did anyone succeed here with big problems? Any recommendations for a drop-in replacement for MUMPS? Thanks for any hints, Dominik Options used: -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 Output: ****** FACTORIZATION STEP ******** GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... NUMBER OF WORKING PROCESSES = 16 OUT-OF-CORE OPTION (ICNTL(22)) = 0 REAL SPACE FOR FACTORS = 1438970073 INTEGER SPACE FOR FACTORS = 11376442 MAXIMUM FRONTAL SIZE (ESTIMATED) = 16868 NUMBER OF NODES IN THE TREE = 43676 Convergence error after scaling for ONE-NORM (option 7/8) = 0.21D+01 Maximum effective relaxed size of S = 231932340 Average effective relaxed size of S = 182366303 REDISTRIB: TOTAL DATA LOCAL/SENT = 1509215 22859750 GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8270 ** Memory relaxation parameter ( ICNTL(14) ) : 35 ** Rank of processor needing largest memory in facto : 0 ** Space in MBYTES used by this processor for facto : 2017 ** Avg. Space in MBYTES per working proc during facto : 1618 From jedbrown at mcs.anl.gov Wed Dec 21 06:39:27 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Dec 2011 06:39:27 -0600 Subject: [petsc-users] How to access the value uarray[j+2][i] In-Reply-To: References: Message-ID: On Wed, Dec 21, 2011 at 01:04, Fatcharm wrote: > I am wondering, why the above sentence "uxxx=..." can access > "uarray[j][i+2] " succesfully, while the program reports error when > accessing "uarray[j+2][i]" ? > Did you set the stencil width to 2 or larger? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Dec 21 07:01:31 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Dec 2011 07:01:31 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: -pc_type lu -pc_factor_mat_solver_package superlu_dist On Dec 21, 2011 6:19 AM, "Dominik Szczerba" wrote: > I am successfully solving my indefinite systems with MUMPS but only > for very small problems. To give a feeling, a mere couple thousands > elements. If I only double the problem size, it silently freezes, even > with max verbosity via the control parameters. Did anyone succeed here > with big problems? Any recommendations for a drop-in replacement for > MUMPS? > > Thanks for any hints, > Dominik > > > > Options used: > -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 > > Output: > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 16 > OUT-OF-CORE OPTION (ICNTL(22)) = 0 > REAL SPACE FOR FACTORS = 1438970073 > INTEGER SPACE FOR FACTORS = 11376442 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 16868 > NUMBER OF NODES IN THE TREE = 43676 > Convergence error after scaling for ONE-NORM (option 7/8) = 0.21D+01 > Maximum effective relaxed size of S = 231932340 > Average effective relaxed size of S = 182366303 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 1509215 22859750 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8270 > ** Memory relaxation parameter ( ICNTL(14) ) : 35 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 2017 > ** Avg. Space in MBYTES per working proc during facto : 1618 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Dec 21 07:51:01 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Dec 2011 07:51:01 -0600 Subject: [petsc-users] Get solution at a stage for ts_type arkimex In-Reply-To: <4EEB9645.9020307@lycos.com> References: <4EEB9645.9020307@lycos.com> Message-ID: On Fri, Dec 16, 2011 at 13:04, Konstantinos Kontzialis < ckontzialis at lycos.com> wrote: > I use ts_type arkimex, and I want at the end of every RK stage to apply a > limiter function. How can I set TS so that to apply the function at the end > of a RK stage? If you are doing something at every stage, it's best to write it into your residual/right-hand-side. This is how limiters work in TVD finite volume and discontinuous Galerkin methods. Accessing at each stage is bad because: 1. The meaning of what is solved in each stage varies between methods. 2. It may spoil the order of accuracy and/or other stability properties of the method. 3. The stages may not all be increasing or located inside the time interval. If you must, you can hook into SNES directly (e.g. by defining your own "line search" that applies the filter), but I don't recommend it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Wed Dec 21 08:19:19 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Wed, 21 Dec 2011 22:19:19 +0800 Subject: [petsc-users] How to access the value uarray[j+2][i] Message-ID: <201112212219173433190@gmail.com> Sorry, I didn't think about the stencil width. Now, the program can run on multi-processors. Thank you very much, Jed. Feng-Chao Wang ------------------------------ Message: 6 Date: Wed, 21 Dec 2011 06:39:27 -0600 From: Jed Brown Subject: Re: [petsc-users] How to access the value uarray[j+2][i] To: PETSc users list Message-ID: Content-Type: text/plain; charset="utf-8" On Wed, Dec 21, 2011 at 01:04, Fatcharm wrote: > I am wondering, why the above sentence "uxxx=..." can access > "uarray[j][i+2] " succesfully, while the program reports error when > accessing "uarray[j+2][i]" ? > Did you set the stencil width to 2 or larger? -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ From E.T.A.vanderWeide at utwente.nl Wed Dec 21 08:54:44 2011 From: E.T.A.vanderWeide at utwente.nl (E.T.A.vanderWeide at utwente.nl) Date: Wed, 21 Dec 2011 15:54:44 +0100 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> Message-ID: <850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl> Hi Matt, Thanks, this is working. The only problem I still have is that I have to store much more than just a vector in context of MATSHELL in order to carry out the matrix vector multiplication. I basically need all the contents of an object of a certain class. I tried to store the this pointer of the object in the context of MATSHELL, but then I get an error. So it looks like it is not allowed to pass the this pointer to MatShellSetContext, is that correct? My current workaround is to use a global variable. That works, but I don't like to use global variables. Is there a better way to make available a pointer to an object in MATSHELL? Thanks again, Edwin -----Original Message----- From: petsc-users-bounces at mcs.anl.gov on behalf of Matthew Knepley Sent: Tue 12/20/2011 3:46 PM To: PETSc users list Subject: Re: [petsc-users] Providing matrix vector product yourself for matrix free in SNES On Tue, Dec 20, 2011 at 7:24 AM, wrote: > ** > > Hi, > > I was wondering if it is possible to provide PETSc the matrix vector > product yourself for a matrix free computation inside the SNES environment. > The reason for asking is that the truncation error in the standard finite > difference approach may lead to some noisy behavior for our application. We > have found a nice way to get rid off the truncation error completely by > using dual numbers for the creation of the Jacobian matrices. The same > technique can be used for the matrix vector product inside SNES, but this > means that the user has to provide this product to PETSc. > > To be more specific. I would like to solve the nonlinear problem F(U) = 0 > via the SNES environment using the matrix free option. So similarly to the > function SNESSetFunction, which sets the function evaluation routine, I'm > seeking for a function that sets the function to compute the matrix vector > product dF/dU times V, where U and V are inputs. Is such a function > available or is there a different way to provide the matrix vector product > yourself? > No need to use the matrix-free option. Just provide a MATSHELL ( http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateShell.html ) as the Jacobian using SNESSetJacobian(). Then your FormJacobian() function will get a vector 'x' as input. Update the context for the MATSHELL with this vector in that function, which is all it would really do. Then when the MatMult() is called on your shell matrix, compute the action using that 'x' which is stored in the context. Matt > Thanks, > > Edwin > --------------------------------------------------- > Edwin van der Weide > Department of Mechanical Engineering > University of Twente > Enschede, the Netherlands > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Dec 21 09:08:51 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Dec 2011 09:08:51 -0600 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES In-Reply-To: <850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl> References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> <850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl> Message-ID: On Wed, Dec 21, 2011 at 08:54, wrote: > Thanks, this is working. The only problem I still have is that I have to > store much more than just a vector in context of MATSHELL in order to carry > out the matrix vector multiplication. I basically need all the contents of > an object of a certain class. I tried to store the this pointer of the > object in the context of MATSHELL, but then I get an error. So it looks > like it is not allowed to pass the this pointer to MatShellSetContext, is > that correct? How are you calling MatShellSetContext() and what is the error? You can pass "this", but the callback should be either a standalone function or a static member function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Dec 21 09:15:29 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 21 Dec 2011 09:15:29 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Direct solvers often require large memory for storing matrix factors. As Jed suggests, you may try superlu_dist. With mumps, I notice you use parallel analysis, which is relative new in mumps. What happens if you use default sequential analysis with different matrix orderings? I usually use matrix ordering '-mat_mumps_icntl_7 2'. Also, you can increase fill ratio, -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace increase (None) i.e., default ration is 20, you may try 50? (I notice that you already use 30). It seems you use 16 CPUs for "a mere couple thousands elements" problems, and mumps "silently freezes". I do not have this type of experience with mumps. I usually can solve sparse matrix of size 10k with 1 cpu using mumps. When mumps runs out of memory or gets other problems, it terminates execution and dumps out error message, not freezes. Something is wrong here. Use a debugger and figuring out where it freezes. Hong On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: > -pc_type lu -pc_factor_mat_solver_package superlu_dist > > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" wrote: >> >> I am successfully solving my indefinite systems with MUMPS but only >> for very small problems. To give a feeling, a mere couple thousands >> elements. If I only double the problem size, it silently freezes, even >> with max verbosity via the control parameters. Did anyone succeed here >> with big problems? Any recommendations for a drop-in replacement for >> MUMPS? >> >> Thanks for any hints, >> Dominik >> >> >> >> Options used: >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >> >> Output: >> >> ****** FACTORIZATION STEP ******** >> >> >> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >> >> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 From xiaohl1986 at gmail.com Wed Dec 21 11:19:01 2011 From: xiaohl1986 at gmail.com (Hailong Xiao) Date: Wed, 21 Dec 2011 09:19:01 -0800 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Hi Hong I got the error from MUMPS. When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, nonzeros (nz) = 785860, I could run it. And could get result. But when I run it with nz=3112820 n =61240 I am getting the following error 17 - : Could not convert index 1140850688 into a pointer The index may be an incorrect argument. Possible sources of this problem are a missing "include 'mpif.h'", a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) or a misspelled user variable for an MPI object (e.g., com instead of comm). [17] [] Aborting Program! Do you know what happened? Is that possible it is running out of memory? On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: > Direct solvers often require large memory for storing matrix factors. > As Jed suggests, you may try superlu_dist. > > With mumps, I notice you use parallel analysis, which is relative new in > mumps. > What happens if you use default sequential analysis with > different matrix orderings? > I usually use matrix ordering '-mat_mumps_icntl_7 2'. > > Also, you can increase fill ratio, > -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace > increase (None) > i.e., default ration is 20, you may try 50? (I notice that you already use > 30). > > It seems you use 16 CPUs for "a mere couple thousands > elements" problems, and mumps "silently freezes". I do not have this type > of experience with mumps. I usually can solve sparse matrix of size > 10k with 1 cpu using mumps. > When mumps runs out of memory or gets other problems, it terminates > execution and dumps out error message, > not freezes. > Something is wrong here. Use a debugger and figuring out where it freezes. > > Hong > > On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: > > -pc_type lu -pc_factor_mat_solver_package superlu_dist > > > > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" > wrote: > >> > >> I am successfully solving my indefinite systems with MUMPS but only > >> for very small problems. To give a feeling, a mere couple thousands > >> elements. If I only double the problem size, it silently freezes, even > >> with max verbosity via the control parameters. Did anyone succeed here > >> with big problems? Any recommendations for a drop-in replacement for > >> MUMPS? > >> > >> Thanks for any hints, > >> Dominik > >> > >> > >> > >> Options used: > >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 > >> > >> Output: > >> > >> ****** FACTORIZATION STEP ******** > >> > >> > >> GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > >> NUMBER OF WORKING PROCESSES = 16 > >> OUT-OF-CORE OPTION (ICNTL(22)) = 0 > >> REAL SPACE FOR FACTORS = 1438970073 > >> INTEGER SPACE FOR FACTORS = 11376442 > >> MAXIMUM FRONTAL SIZE (ESTIMATED) = 16868 > >> NUMBER OF NODES IN THE TREE = 43676 > >> Convergence error after scaling for ONE-NORM (option 7/8) = 0.21D+01 > >> Maximum effective relaxed size of S = 231932340 > >> Average effective relaxed size of S = 182366303 > >> > >> REDISTRIB: TOTAL DATA LOCAL/SENT = 1509215 22859750 > >> GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8270 > >> ** Memory relaxation parameter ( ICNTL(14) ) : 35 > >> ** Rank of processor needing largest memory in facto : 0 > >> ** Space in MBYTES used by this processor for facto : 2017 > >> ** Avg. Space in MBYTES per working proc during facto : 1618 > -- Hailong -------------- next part -------------- An HTML attachment was scrubbed... URL: From E.T.A.vanderWeide at utwente.nl Wed Dec 21 11:30:16 2011 From: E.T.A.vanderWeide at utwente.nl (E.T.A.vanderWeide at utwente.nl) Date: Wed, 21 Dec 2011 18:30:16 +0100 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl><850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl> Message-ID: <850A61E88A87D54E832F7C171A0BB889016B0917@ctwex4.ctw.utwente.nl> The call the MatShellSetContext is as follows ierr = MatShellSetContext(mat, this); CHKERRQ(ierr); Later on in the call to MatShellGetContext I get the following error [0]PETSC ERROR: MatShellGetContext() line 201 in src/mat/impls/shell/shell.c Null Pointer: Parameter # 2 The callback function is a standalone function, in which I call MatShellGetContext to get the context. In there it fails. Edwin -----Original Message----- From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown Sent: Wed 12/21/2011 4:08 PM To: PETSc users list Subject: Re: [petsc-users] Providing matrix vector product yourself for matrix free in SNES On Wed, Dec 21, 2011 at 08:54, wrote: > Thanks, this is working. The only problem I still have is that I have to > store much more than just a vector in context of MATSHELL in order to carry > out the matrix vector multiplication. I basically need all the contents of > an object of a certain class. I tried to store the this pointer of the > object in the context of MATSHELL, but then I get an error. So it looks > like it is not allowed to pass the this pointer to MatShellSetContext, is > that correct? How are you calling MatShellSetContext() and what is the error? You can pass "this", but the callback should be either a standalone function or a static member function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Dec 21 13:31:27 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Dec 2011 13:31:27 -0600 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES In-Reply-To: <850A61E88A87D54E832F7C171A0BB889016B0917@ctwex4.ctw.utwente.nl> References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl> <850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl> <850A61E88A87D54E832F7C171A0BB889016B0917@ctwex4.ctw.utwente.nl> Message-ID: On Wed, Dec 21, 2011 at 11:30, wrote: > ierr = MatShellSetContext(mat, this); > This is fine. > CHKERRQ(ierr); > > Later on in the call to MatShellGetContext I get the following error > [0]PETSC ERROR: MatShellGetContext() line 201 in > src/mat/impls/shell/shell.c Null Pointer: Parameter # 2 > You need to pass a valid address. Show us what this code looks like. It should be something like MyContext *ptr; MatShellGetContext(mat,&ptr); -------------- next part -------------- An HTML attachment was scrubbed... URL: From E.T.A.vanderWeide at utwente.nl Wed Dec 21 14:05:09 2011 From: E.T.A.vanderWeide at utwente.nl (E.T.A.vanderWeide at utwente.nl) Date: Wed, 21 Dec 2011 21:05:09 +0100 Subject: [petsc-users] Providing matrix vector product yourself for matrix free in SNES References: <850A61E88A87D54E832F7C171A0BB889016B090F@ctwex4.ctw.utwente.nl><850A61E88A87D54E832F7C171A0BB889016B0916@ctwex4.ctw.utwente.nl><850A61E88A87D54E832F7C171A0BB889016B0917@ctwex4.ctw.utwente.nl> Message-ID: <850A61E88A87D54E832F7C171A0BB889016B0918@ctwex4.ctw.utwente.nl> > You need to pass a valid address. Show us what this code looks like. It > should be something like > MyContext *ptr; > MatShellGetContext(mat,&ptr); That's it. I just sent ptr not &ptr. It is working now. Thanks, Edwin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rm93 at buffalo.edu Wed Dec 21 14:12:21 2011 From: rm93 at buffalo.edu (Reza Madankan) Date: Wed, 21 Dec 2011 15:12:21 -0500 Subject: [petsc-users] parallel PCASM Message-ID: Hello, I am trying to parallelize the ASM preconditioner while solving a linear system. For more detail, I have printed relevant parts of code in the following: *static char help[]="Reading in a matrix\n";* * * * #include* * * * #include* * * * #include* * * * #include "petscvec.h" /* This enables us to use vectors. */* * * * #include "petscmat.h" /* This enables us to use Matrices. It includes the* * petscvec header file*/* * * * #include "petscksp.h" /* Now we can solve linear systems. Solvers used are * * KSP. */* * * * extern PetscErrorCode MyKSPMonitor(KSP,PetscInt,PetscReal,void*);* * * * int main(int argc, char **argv)* * * * {* * * * /* Declaration of Matrix A and some vectors x*/* * * * Vec w, ypcq, mean_tmp, x, PzyTvec;* * Mat Ym, Y, Mean, Ypcq, YpcqT, Pzz, InnProd, Para, Pzy, PzyT,PzzT,K, KT, PK, PzzTMP, Pmat, XYmat, KY;* * * * FILE *fp, *fp1, *fp2, *fp3;* * * * PetscInt index,i,j,k, ns=16, nw=100, tindex_f=120, col,tind, mz, nz, its;* */* * *printf("Enter number of sensors (ns):\n");* *scanf("%d",&ns);* * * *printf("Enter number of time steps (tindex_f):\n");* *scanf("%d",&tindex_f);* **/* * PetscMPIInt size;* * * * PC pc;* * KSP ksp;* * PetscReal norm, tol=1.e-14;* * PetscBool nonzeroguess = PETSC_FALSE;* * PetscViewer mviewer;* * * * * * PetscScalar scalar,rhsy[ns*tindex_f*nw],rhsym[ns*tindex_f], rhsw[nw], Ymat[tindex_f*ns][nw], tmp, rhsp[2*nw], xvec[ns*tindex_f], pmat[4], xymat[2], rnorm;* * * * PetscErrorCode ierr;* * * * /* This part is needed for the help flag supplied at run-time*/* * * * ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);* * * * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* * //if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor example");* *PetscPrintf(PETSC_COMM_WORLD,"size is: %D\n",size);* *int nstindex_f=ns*tindex_f;* *ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr);* * * *.* *.* *.* * * * * *// Solution for K:* *// --------------* * * * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* * ierr = PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr);* * ierr = PetscOptionsGetBool(PETSC_NULL,"-nonzero_guess",&nonzeroguess,PETSC_NULL); CHKERRQ(ierr);* * * * * *// KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL,0);* * * * * * ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);* * ierr = PetscObjectSetName((PetscObject) x, "Solution");CHKERRQ(ierr);* * ierr = VecSetSizes(x,PETSC_DECIDE,ns*tindex_f);CHKERRQ(ierr);* * ierr = VecSetFromOptions(x);CHKERRQ(ierr);* * ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);* * * * * *// ierr = KSPMonitorSet(ksp,PetscInt n,PetscReal rnorm, void *); * *// ierr = monitor(ksp,Int it,PetscReal rnorm,);* * * * * * ierr = MatTranspose(Pzz,MAT_INITIAL_MATRIX,&PzzT);* * MatAssemblyBegin(PzzT,MAT_FINAL_ASSEMBLY);* * MatAssemblyEnd(PzzT,MAT_FINAL_ASSEMBLY);* * ierr = KSPSetOperators(ksp,PzzT,PzzT,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);* * * * ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);* * ierr = PCSetType(pc,PCASM);CHKERRQ(ierr);* * ierr = KSPSetTolerances(ksp,1e-6,PETSC_DEFAULT,PETSC_DEFAULT,1e+9);CHKERRQ(ierr);* * ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);* * if (nonzeroguess)* * {* * PetscScalar p = 0.5;* * ierr = VecSet(x,p);CHKERRQ(ierr);* * ierr = KSPSetInitialGuessNonzero(ksp,PETSC_TRUE);CHKERRQ(ierr);* * }* * ierr = MatTranspose(Pzy,MAT_INITIAL_MATRIX,&PzyT);* * MatAssemblyBegin(PzyT,MAT_FINAL_ASSEMBLY);* * MatAssemblyEnd(PzyT,MAT_FINAL_ASSEMBLY);* * ierr = MatCreate(PETSC_COMM_WORLD,&KT);CHKERRQ(ierr);* * ierr = MatSetSizes(KT,PETSC_DECIDE,PETSC_DECIDE,ns*tindex_f,2);CHKERRQ(ierr);* * ierr = MatSetFromOptions(KT);CHKERRQ(ierr);* * int ctr=0;* * for (ctr=0; ctr<2; ctr++)* * {* * ierr = VecCreate(PETSC_COMM_WORLD,&PzyTvec);* * ierr = VecSetSizes(PzyTvec,PETSC_DECIDE,ns*tindex_f);* * ierr = VecSetFromOptions(PzyTvec);* * ierr = MatGetColumnVector(PzyT,PzyTvec,ctr);* * * * ierr = KSPSolve(ksp,PzyTvec,x);CHKERRQ(ierr);* * ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);* * ierr = KSPGetResidualNorm(ksp,& norm); * * printf("residual norm is=%e\n",norm);* * printf("iteration number is=%d\n",its);* * ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);* * * *// printf("%d\n", ctr);* * for (i=0; i From knepley at gmail.com Wed Dec 21 14:19:27 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Dec 2011 14:19:27 -0600 Subject: [petsc-users] parallel PCASM In-Reply-To: References: Message-ID: On Wed, Dec 21, 2011 at 2:12 PM, Reza Madankan wrote: > Hello, > I am trying to parallelize the ASM preconditioner while solving a linear > system. For more detail, I have printed relevant parts of code in the > following: > I have no idea what "parallelize ASM" might mean. Below you likely have a bad value in b, the rhs you pass to the solver on line 487 of your code. Matt > *static char help[]="Reading in a matrix\n";* > * > * > * #include* > * > * > * #include* > * > * > * #include* > * > * > * #include "petscvec.h" /* This enables us to use vectors. */* > * > * > * #include "petscmat.h" /* This enables us to use Matrices. It includes > the* > * petscvec header file*/* > * > * > * #include "petscksp.h" /* Now we can solve linear systems. Solvers used > are* > * KSP. */* > * > * > * extern PetscErrorCode MyKSPMonitor(KSP,PetscInt,PetscReal,void*);* > * > * > * int main(int argc, char **argv)* > * > * > * {* > * > * > * /* Declaration of Matrix A and some vectors x*/* > * > * > * Vec w, ypcq, mean_tmp, x, PzyTvec;* > * Mat Ym, Y, Mean, Ypcq, YpcqT, Pzz, InnProd, Para, Pzy, PzyT,PzzT,K, KT, > PK, PzzTMP, Pmat, XYmat, KY;* > * > * > * FILE *fp, *fp1, *fp2, *fp3;* > * > * > * PetscInt index,i,j,k, ns=16, nw=100, tindex_f=120, col,tind, mz, nz, > its;* > */* * > *printf("Enter number of sensors (ns):\n");* > *scanf("%d",&ns);* > * > * > *printf("Enter number of time steps (tindex_f):\n");* > *scanf("%d",&tindex_f);* > **/* > * PetscMPIInt size;* > * > * > * PC pc;* > * KSP ksp;* > * PetscReal norm, tol=1.e-14;* > * PetscBool nonzeroguess = PETSC_FALSE;* > * PetscViewer mviewer;* > * > * > * > * > * PetscScalar scalar,rhsy[ns*tindex_f*nw],rhsym[ns*tindex_f], rhsw[nw], > Ymat[tindex_f*ns][nw], tmp, rhsp[2*nw], xvec[ns*tindex_f], pmat[4], > xymat[2], rnorm;* > * > * > * PetscErrorCode ierr;* > * > * > * /* This part is needed for the help flag supplied at run-time*/* > * > * > * ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);* > * * > * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* > * //if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor > example");* > *PetscPrintf(PETSC_COMM_WORLD,"size is: %D\n",size);* > *int nstindex_f=ns*tindex_f;* > *ierr = > PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr);* > * > * > *.* > *.* > *.* > * > * > * > * > *// Solution for K:* > *// --------------* > * > * > * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* > * ierr = > PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr);* > * ierr = > PetscOptionsGetBool(PETSC_NULL,"-nonzero_guess",&nonzeroguess,PETSC_NULL); > CHKERRQ(ierr);* > * > * > * > * > *// KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL,0);* > * > * > * > * > * ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);* > * ierr = PetscObjectSetName((PetscObject) x, "Solution");CHKERRQ(ierr); > * > * ierr = VecSetSizes(x,PETSC_DECIDE,ns*tindex_f);CHKERRQ(ierr);* > * ierr = VecSetFromOptions(x);CHKERRQ(ierr);* > * ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);* > * * > * > * > *// ierr = KSPMonitorSet(ksp,PetscInt n,PetscReal rnorm, void *); * > *// ierr = monitor(ksp,Int it,PetscReal rnorm,);* > * * > * > * > * ierr = MatTranspose(Pzz,MAT_INITIAL_MATRIX,&PzzT);* > * MatAssemblyBegin(PzzT,MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd(PzzT,MAT_FINAL_ASSEMBLY);* > * ierr = > KSPSetOperators(ksp,PzzT,PzzT,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);* > * > * > * ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);* > * ierr = PCSetType(pc,PCASM);CHKERRQ(ierr);* > * ierr = > KSPSetTolerances(ksp,1e-6,PETSC_DEFAULT,PETSC_DEFAULT,1e+9);CHKERRQ(ierr); > * > * ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);* > * if (nonzeroguess)* > * {* > * PetscScalar p = 0.5;* > * ierr = VecSet(x,p);CHKERRQ(ierr);* > * ierr = KSPSetInitialGuessNonzero(ksp,PETSC_TRUE);CHKERRQ(ierr);* > * }* > * ierr = MatTranspose(Pzy,MAT_INITIAL_MATRIX,&PzyT);* > * MatAssemblyBegin(PzyT,MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd(PzyT,MAT_FINAL_ASSEMBLY);* > * ierr = MatCreate(PETSC_COMM_WORLD,&KT);CHKERRQ(ierr);* > * ierr = > MatSetSizes(KT,PETSC_DECIDE,PETSC_DECIDE,ns*tindex_f,2);CHKERRQ(ierr);* > * ierr = MatSetFromOptions(KT);CHKERRQ(ierr);* > * int ctr=0;* > * for (ctr=0; ctr<2; ctr++)* > * {* > * ierr = VecCreate(PETSC_COMM_WORLD,&PzyTvec);* > * ierr = VecSetSizes(PzyTvec,PETSC_DECIDE,ns*tindex_f);* > * ierr = VecSetFromOptions(PzyTvec);* > * ierr = MatGetColumnVector(PzyT,PzyTvec,ctr);* > * > * > * ierr = KSPSolve(ksp,PzyTvec,x);CHKERRQ(ierr);* > * ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);* > * ierr = KSPGetResidualNorm(ksp,& norm); * > * printf("residual norm is=%e\n",norm);* > * printf("iteration number is=%d\n",its);* > * ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);* > * * > *// printf("%d\n", ctr);* > * for (i=0; i * {* > * ierr = VecGetValues(x,1,&i,&xvec[i]);* > * ierr = MatSetValues(KT,1,&i,1,&ctr,&xvec[i],INSERT_VALUES);* > * }* > * }* > * MatAssemblyBegin(KT,MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd(KT,MAT_FINAL_ASSEMBLY);* > * MatTranspose(KT,MAT_INITIAL_MATRIX,&K);* > * MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);* > * > * > * MatPtAP(Pzz,KT,MAT_INITIAL_MATRIX,1.0,&PK);* > *//MatCopy(Pzz,PzzTMP,SAME_NONZERO_PATTERN);* > * ierr = MatAXPY(Pmat,-1,PK,DIFFERENT_NONZERO_PATTERN);* > * MatView(Pmat,PETSC_VIEWER_STDOUT_WORLD);* > * > * > * ierr = MatAXPY(Ym,-1,Mean,DIFFERENT_NONZERO_PATTERN);* > * MatAssemblyBegin(Ym,MAT_FINAL_ASSEMBLY);* > * MatAssemblyEnd(Ym,MAT_FINAL_ASSEMBLY);* > * > * > * ierr = MatMatMult(K,Ym,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&KY);* > * ierr = MatAssemblyBegin(KY,MAT_FINAL_ASSEMBLY);* > * ierr = MatAssemblyEnd(KY,MAT_FINAL_ASSEMBLY);* > * ierr = MatAXPY(XYmat,1,KY,DIFFERENT_NONZERO_PATTERN);* > * MatView(XYmat,PETSC_VIEWER_STDOUT_WORLD);* > * > * > *ierr = MatDestroy(&Mean);* > * > * > *ierr=PetscFinalize();CHKERRQ(ierr);* > * > * > * return 0;* > * > * > * }* > * > * > > > I am using the following commands to compile and run the code: > > *mpicc pcasm.c -o pcasm -l petsc* > *mpiexec -np 2 ./pcasm* > > But when I run the code I get the following error message which I don't > know what it means. > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./pcasm on a linux-imp named k07n14.ccr.buffalo.edu by > rm93 Wed Dec 21 15:05:33 2011 > [0]PETSC ERROR: Libraries linked from > /util/petsc/petsc-3.2-p3/linux-impi-mkl/lib > [0]PETSC ERROR: Configure run at Fri Oct 21 08:36:23 2011 > [0]PETSC ERROR: Configure options --CC=/util/intel/impi/ > 4.0.3.008/intel64/bin/mpiicc --FC=/util/intel/impi/ > 4.0.3.008/intel64/bin/mpiifort --CXX=/util/intel/impi/ > 4.0.3.008/intel64/bin/mpiicpc--with-blas-lapack-dir=/util/intel/composer_xe_2011_sp1/mkl/lib/intel64 > --download-hypre=1 --with-debugging=0 -PETSC_ARCH=linux-impi-mkl > --with-shared-libraries=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: VecNormalize() line 261 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: GMREScycle() line 128 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in > src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 487 in "unknowndirectory/"pcasm.c > application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 > rank 0 in job 2 k07n14.ccr.buffalo.edu_51735 caused collective abort of > all ranks > exit status of rank 0: return code 72 > > > > > > > Could anyone help me on this problem? Thanks in advance for any help. > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rm93 at buffalo.edu Wed Dec 21 15:30:17 2011 From: rm93 at buffalo.edu (Reza Madankan) Date: Wed, 21 Dec 2011 16:30:17 -0500 Subject: [petsc-users] parallel PCASM In-Reply-To: References: Message-ID: Hi Matthew; Thank you for you quick reply. Sorry for the wrong terminology, it's just few days I started working with Petsc. The whole story is that I have a linear system Ax = b which I want to solve it using different preconditioners just to see the effect of each of those on the solution and residual error. I have already used PCJACOBI and PCILU to find the solution. I also would like to see the effect of using PCASM. PCASM on a single processor performs like PCILU, but its result will be different from PCILU if I implement it on multiple processors. So, I am trying to get it run for multiple processors case using the following command: mpiexec -np 2 ./pcasm but I get the error message that I showed before. Now my question is that how I can implement PCASM on multiple processors? Thanks again Reza On Wed, Dec 21, 2011 at 3:19 PM, Matthew Knepley wrote: > On Wed, Dec 21, 2011 at 2:12 PM, Reza Madankan wrote: > >> Hello, >> I am trying to parallelize the ASM preconditioner while solving a linear >> system. For more detail, I have printed relevant parts of code in the >> following: >> > > I have no idea what "parallelize ASM" might mean. > > Below you likely have a bad value in b, the rhs you pass to the solver on > line 487 of your code. > > Matt > > >> *static char help[]="Reading in a matrix\n";* >> * >> * >> * #include* >> * >> * >> * #include* >> * >> * >> * #include* >> * >> * >> * #include "petscvec.h" /* This enables us to use vectors. */* >> * >> * >> * #include "petscmat.h" /* This enables us to use Matrices. It includes >> the* >> * petscvec header file*/* >> * >> * >> * #include "petscksp.h" /* Now we can solve linear systems. Solvers used >> are* >> * KSP. */* >> * >> * >> * extern PetscErrorCode MyKSPMonitor(KSP,PetscInt,PetscReal,void*);* >> * >> * >> * int main(int argc, char **argv)* >> * >> * >> * {* >> * >> * >> * /* Declaration of Matrix A and some vectors x*/* >> * >> * >> * Vec w, ypcq, mean_tmp, x, PzyTvec;* >> * Mat Ym, Y, Mean, Ypcq, YpcqT, Pzz, InnProd, Para, Pzy, PzyT,PzzT,K, >> KT, PK, PzzTMP, Pmat, XYmat, KY;* >> * >> * >> * FILE *fp, *fp1, *fp2, *fp3;* >> * >> * >> * PetscInt index,i,j,k, ns=16, nw=100, tindex_f=120, col,tind, mz, nz, >> its;* >> */* * >> *printf("Enter number of sensors (ns):\n");* >> *scanf("%d",&ns);* >> * >> * >> *printf("Enter number of time steps (tindex_f):\n");* >> *scanf("%d",&tindex_f);* >> **/* >> * PetscMPIInt size;* >> * >> * >> * PC pc;* >> * KSP ksp;* >> * PetscReal norm, tol=1.e-14;* >> * PetscBool nonzeroguess = PETSC_FALSE;* >> * PetscViewer mviewer;* >> * >> * >> * >> * >> * PetscScalar scalar,rhsy[ns*tindex_f*nw],rhsym[ns*tindex_f], rhsw[nw], >> Ymat[tindex_f*ns][nw], tmp, rhsp[2*nw], xvec[ns*tindex_f], pmat[4], >> xymat[2], rnorm;* >> * >> * >> * PetscErrorCode ierr;* >> * >> * >> * /* This part is needed for the help flag supplied at run-time*/* >> * >> * >> * ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);* >> * * >> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >> * //if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor >> example");* >> *PetscPrintf(PETSC_COMM_WORLD,"size is: %D\n",size);* >> *int nstindex_f=ns*tindex_f;* >> *ierr = >> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >> * >> * >> * >> *.* >> *.* >> *.* >> * >> * >> * >> * >> *// Solution for K:* >> *// --------------* >> * >> * >> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >> * ierr = >> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >> * >> * ierr = >> PetscOptionsGetBool(PETSC_NULL,"-nonzero_guess",&nonzeroguess,PETSC_NULL); >> CHKERRQ(ierr);* >> * >> * >> * >> * >> *// KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL,0);* >> * >> * >> * >> * >> * ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);* >> * ierr = PetscObjectSetName((PetscObject) x, >> "Solution");CHKERRQ(ierr);* >> * ierr = VecSetSizes(x,PETSC_DECIDE,ns*tindex_f);CHKERRQ(ierr);* >> * ierr = VecSetFromOptions(x);CHKERRQ(ierr);* >> * ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);* >> * * >> * >> * >> *// ierr = KSPMonitorSet(ksp,PetscInt n,PetscReal rnorm, void *); * >> *// ierr = monitor(ksp,Int it,PetscReal rnorm,);* >> * * >> * >> * >> * ierr = MatTranspose(Pzz,MAT_INITIAL_MATRIX,&PzzT);* >> * MatAssemblyBegin(PzzT,MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd(PzzT,MAT_FINAL_ASSEMBLY);* >> * ierr = >> KSPSetOperators(ksp,PzzT,PzzT,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);* >> * >> * >> * ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);* >> * ierr = PCSetType(pc,PCASM);CHKERRQ(ierr);* >> * ierr = >> KSPSetTolerances(ksp,1e-6,PETSC_DEFAULT,PETSC_DEFAULT,1e+9);CHKERRQ(ierr); >> * >> * ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);* >> * if (nonzeroguess)* >> * {* >> * PetscScalar p = 0.5;* >> * ierr = VecSet(x,p);CHKERRQ(ierr);* >> * ierr = KSPSetInitialGuessNonzero(ksp,PETSC_TRUE);CHKERRQ(ierr); >> * >> * }* >> * ierr = MatTranspose(Pzy,MAT_INITIAL_MATRIX,&PzyT);* >> * MatAssemblyBegin(PzyT,MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd(PzyT,MAT_FINAL_ASSEMBLY);* >> * ierr = MatCreate(PETSC_COMM_WORLD,&KT);CHKERRQ(ierr);* >> * ierr = >> MatSetSizes(KT,PETSC_DECIDE,PETSC_DECIDE,ns*tindex_f,2);CHKERRQ(ierr);* >> * ierr = MatSetFromOptions(KT);CHKERRQ(ierr);* >> * int ctr=0;* >> * for (ctr=0; ctr<2; ctr++)* >> * {* >> * ierr = VecCreate(PETSC_COMM_WORLD,&PzyTvec);* >> * ierr = VecSetSizes(PzyTvec,PETSC_DECIDE,ns*tindex_f);* >> * ierr = VecSetFromOptions(PzyTvec);* >> * ierr = MatGetColumnVector(PzyT,PzyTvec,ctr);* >> * >> * >> * ierr = KSPSolve(ksp,PzyTvec,x);CHKERRQ(ierr);* >> * ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);* >> * ierr = KSPGetResidualNorm(ksp,& norm); * >> * printf("residual norm is=%e\n",norm);* >> * printf("iteration number is=%d\n",its);* >> * ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);* >> * * >> *// printf("%d\n", ctr);* >> * for (i=0; i> * {* >> * ierr = VecGetValues(x,1,&i,&xvec[i]);* >> * ierr = MatSetValues(KT,1,&i,1,&ctr,&xvec[i],INSERT_VALUES);* >> * }* >> * }* >> * MatAssemblyBegin(KT,MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd(KT,MAT_FINAL_ASSEMBLY);* >> * MatTranspose(KT,MAT_INITIAL_MATRIX,&K);* >> * MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);* >> * >> * >> * MatPtAP(Pzz,KT,MAT_INITIAL_MATRIX,1.0,&PK);* >> *//MatCopy(Pzz,PzzTMP,SAME_NONZERO_PATTERN);* >> * ierr = MatAXPY(Pmat,-1,PK,DIFFERENT_NONZERO_PATTERN);* >> * MatView(Pmat,PETSC_VIEWER_STDOUT_WORLD);* >> * >> * >> * ierr = MatAXPY(Ym,-1,Mean,DIFFERENT_NONZERO_PATTERN);* >> * MatAssemblyBegin(Ym,MAT_FINAL_ASSEMBLY);* >> * MatAssemblyEnd(Ym,MAT_FINAL_ASSEMBLY);* >> * >> * >> * ierr = MatMatMult(K,Ym,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&KY);* >> * ierr = MatAssemblyBegin(KY,MAT_FINAL_ASSEMBLY);* >> * ierr = MatAssemblyEnd(KY,MAT_FINAL_ASSEMBLY);* >> * ierr = MatAXPY(XYmat,1,KY,DIFFERENT_NONZERO_PATTERN);* >> * MatView(XYmat,PETSC_VIEWER_STDOUT_WORLD);* >> * >> * >> *ierr = MatDestroy(&Mean);* >> * >> * >> *ierr=PetscFinalize();CHKERRQ(ierr);* >> * >> * >> * return 0;* >> * >> * >> * }* >> * >> * >> >> >> I am using the following commands to compile and run the code: >> >> *mpicc pcasm.c -o pcasm -l petsc* >> *mpiexec -np 2 ./pcasm* >> >> But when I run the code I get the following error message which I don't >> know what it means. >> >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Floating point exception! >> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 >> CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./pcasm on a linux-imp named k07n14.ccr.buffalo.edu by >> rm93 Wed Dec 21 15:05:33 2011 >> [0]PETSC ERROR: Libraries linked from >> /util/petsc/petsc-3.2-p3/linux-impi-mkl/lib >> [0]PETSC ERROR: Configure run at Fri Oct 21 08:36:23 2011 >> [0]PETSC ERROR: Configure options --CC=/util/intel/impi/ >> 4.0.3.008/intel64/bin/mpiicc --FC=/util/intel/impi/ >> 4.0.3.008/intel64/bin/mpiifort --CXX=/util/intel/impi/ >> 4.0.3.008/intel64/bin/mpiicpc--with-blas-lapack-dir=/util/intel/composer_xe_2011_sp1/mkl/lib/intel64 >> --download-hypre=1 --with-debugging=0 -PETSC_ARCH=linux-impi-mkl >> --with-shared-libraries=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: VecNormalize() line 261 in src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: GMREScycle() line 128 in src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in >> src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: main() line 487 in "unknowndirectory/"pcasm.c >> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 >> rank 0 in job 2 k07n14.ccr.buffalo.edu_51735 caused collective abort >> of all ranks >> exit status of rank 0: return code 72 >> >> >> >> >> >> >> Could anyone help me on this problem? Thanks in advance for any help. >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Wed Dec 21 15:30:46 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 21 Dec 2011 22:30:46 +0100 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Hong, Jed, Thanks a lot for your valuable suggestions. I will be evaluating them. Regards, Dominik On Wed, Dec 21, 2011 at 4:15 PM, Hong Zhang wrote: > Direct solvers often require large memory for storing matrix factors. > As Jed suggests, you may try superlu_dist. > > With mumps, I notice you use parallel analysis, which is relative new in mumps. > What happens if you use default sequential analysis with > different matrix orderings? > I usually use matrix ordering '-mat_mumps_icntl_7 2'. > > Also, you can increase fill ratio, > -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace > increase (None) > i.e., default ration is 20, you may try 50? (I notice that you already use 30). > > It seems you use 16 CPUs for "a mere couple thousands > elements" problems, and mumps "silently freezes". I do not have this type > of experience with mumps. I usually can solve sparse matrix of size > 10k with 1 cpu using mumps. > When mumps runs out of memory or gets other problems, it terminates > execution and dumps out error message, > not freezes. > Something is wrong here. Use a debugger and figuring out where it freezes. > > Hong > > On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >> -pc_type lu -pc_factor_mat_solver_package superlu_dist >> >> On Dec 21, 2011 6:19 AM, "Dominik Szczerba" wrote: >>> >>> I am successfully solving my indefinite systems with MUMPS but only >>> for very small problems. To give a feeling, a mere couple thousands >>> elements. If I only double the problem size, it silently freezes, even >>> with max verbosity via the control parameters. Did anyone succeed here >>> with big problems? Any recommendations for a drop-in replacement for >>> MUMPS? >>> >>> Thanks for any hints, >>> Dominik >>> >>> >>> >>> Options used: >>> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >>> >>> Output: >>> >>> ****** FACTORIZATION STEP ******** >>> >>> >>> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >>> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >>> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >>> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >>> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >>> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >>> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >>> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >>> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >>> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >>> >>> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >>> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >>> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >>> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >>> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >>> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 > From knepley at gmail.com Wed Dec 21 15:40:54 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Dec 2011 15:40:54 -0600 Subject: [petsc-users] parallel PCASM In-Reply-To: References: Message-ID: On Wed, Dec 21, 2011 at 3:30 PM, Reza Madankan wrote: > Hi Matthew; > Thank you for you quick reply. > Sorry for the wrong terminology, it's just few days I started working with > Petsc. > The whole story is that I have a linear system Ax = b which I want to > solve it using different preconditioners just to see the effect of each of > those on the solution and residual error. I have already used PCJACOBI and > PCILU to find the solution. I also would like to see the effect of using > PCASM. PCASM on a single processor performs like PCILU, but its result > will be different from PCILU if I implement it on multiple processors. So, > I am trying to get it run for multiple processors case using the following > command: > > mpiexec -np 2 ./pcasm > > but I get the error message that I showed before. Now my question is that > how I can implement PCASM on multiple processors? > It is not ASM that is the problem. You have a bug in your code. Use valgrind to find it. The larger problem here is that you have completely misunderstood the point of PETSc. Whatever you do, never change your code to use a different solver. You should play with the PETSc examples for a while cd src/snes/examples/tutorials make ex5 ./ex5 -snes_monitor -ksp_monitor -snes_view $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view -pc_type jacobi $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view -pc_type asm Matt Thanks again > Reza > > > > On Wed, Dec 21, 2011 at 3:19 PM, Matthew Knepley wrote: > >> On Wed, Dec 21, 2011 at 2:12 PM, Reza Madankan wrote: >> >>> Hello, >>> I am trying to parallelize the ASM preconditioner while solving a linear >>> system. For more detail, I have printed relevant parts of code in the >>> following: >>> >> >> I have no idea what "parallelize ASM" might mean. >> >> Below you likely have a bad value in b, the rhs you pass to the solver on >> line 487 of your code. >> >> Matt >> >> >>> *static char help[]="Reading in a matrix\n";* >>> * >>> * >>> * #include* >>> * >>> * >>> * #include* >>> * >>> * >>> * #include* >>> * >>> * >>> * #include "petscvec.h" /* This enables us to use vectors. */* >>> * >>> * >>> * #include "petscmat.h" /* This enables us to use Matrices. It includes >>> the* >>> * petscvec header file*/* >>> * >>> * >>> * #include "petscksp.h" /* Now we can solve linear systems. Solvers >>> used are* >>> * KSP. */* >>> * >>> * >>> * extern PetscErrorCode MyKSPMonitor(KSP,PetscInt,PetscReal,void*);* >>> * >>> * >>> * int main(int argc, char **argv)* >>> * >>> * >>> * {* >>> * >>> * >>> * /* Declaration of Matrix A and some vectors x*/* >>> * >>> * >>> * Vec w, ypcq, mean_tmp, x, PzyTvec;* >>> * Mat Ym, Y, Mean, Ypcq, YpcqT, Pzz, InnProd, Para, Pzy, PzyT,PzzT,K, >>> KT, PK, PzzTMP, Pmat, XYmat, KY;* >>> * >>> * >>> * FILE *fp, *fp1, *fp2, *fp3;* >>> * >>> * >>> * PetscInt index,i,j,k, ns=16, nw=100, tindex_f=120, col,tind, mz, nz, >>> its;* >>> */* * >>> *printf("Enter number of sensors (ns):\n");* >>> *scanf("%d",&ns);* >>> * >>> * >>> *printf("Enter number of time steps (tindex_f):\n");* >>> *scanf("%d",&tindex_f);* >>> **/* >>> * PetscMPIInt size;* >>> * >>> * >>> * PC pc;* >>> * KSP ksp;* >>> * PetscReal norm, tol=1.e-14;* >>> * PetscBool nonzeroguess = PETSC_FALSE;* >>> * PetscViewer mviewer;* >>> * >>> * >>> * >>> * >>> * PetscScalar scalar,rhsy[ns*tindex_f*nw],rhsym[ns*tindex_f], rhsw[nw], >>> Ymat[tindex_f*ns][nw], tmp, rhsp[2*nw], xvec[ns*tindex_f], pmat[4], >>> xymat[2], rnorm;* >>> * >>> * >>> * PetscErrorCode ierr;* >>> * >>> * >>> * /* This part is needed for the help flag supplied at run-time*/* >>> * >>> * >>> * ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);* >>> * * >>> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >>> * //if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor >>> example");* >>> *PetscPrintf(PETSC_COMM_WORLD,"size is: %D\n",size);* >>> *int nstindex_f=ns*tindex_f;* >>> *ierr = >>> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >>> * >>> * >>> * >>> *.* >>> *.* >>> *.* >>> * >>> * >>> * >>> * >>> *// Solution for K:* >>> *// --------------* >>> * >>> * >>> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >>> * ierr = >>> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >>> * >>> * ierr = >>> PetscOptionsGetBool(PETSC_NULL,"-nonzero_guess",&nonzeroguess,PETSC_NULL); >>> CHKERRQ(ierr);* >>> * >>> * >>> * >>> * >>> *// KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL,0);* >>> * >>> * >>> * >>> * >>> * ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);* >>> * ierr = PetscObjectSetName((PetscObject) x, >>> "Solution");CHKERRQ(ierr);* >>> * ierr = VecSetSizes(x,PETSC_DECIDE,ns*tindex_f);CHKERRQ(ierr);* >>> * ierr = VecSetFromOptions(x);CHKERRQ(ierr);* >>> * ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);* >>> * * >>> * >>> * >>> *// ierr = KSPMonitorSet(ksp,PetscInt n,PetscReal rnorm, void *); * >>> *// ierr = monitor(ksp,Int it,PetscReal rnorm,);* >>> * * >>> * >>> * >>> * ierr = MatTranspose(Pzz,MAT_INITIAL_MATRIX,&PzzT);* >>> * MatAssemblyBegin(PzzT,MAT_FINAL_ASSEMBLY);* >>> * MatAssemblyEnd(PzzT,MAT_FINAL_ASSEMBLY);* >>> * ierr = >>> KSPSetOperators(ksp,PzzT,PzzT,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);* >>> * >>> * >>> * ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);* >>> * ierr = PCSetType(pc,PCASM);CHKERRQ(ierr);* >>> * ierr = >>> KSPSetTolerances(ksp,1e-6,PETSC_DEFAULT,PETSC_DEFAULT,1e+9);CHKERRQ(ierr); >>> * >>> * ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);* >>> * if (nonzeroguess)* >>> * {* >>> * PetscScalar p = 0.5;* >>> * ierr = VecSet(x,p);CHKERRQ(ierr);* >>> * ierr = >>> KSPSetInitialGuessNonzero(ksp,PETSC_TRUE);CHKERRQ(ierr);* >>> * }* >>> * ierr = MatTranspose(Pzy,MAT_INITIAL_MATRIX,&PzyT);* >>> * MatAssemblyBegin(PzyT,MAT_FINAL_ASSEMBLY);* >>> * MatAssemblyEnd(PzyT,MAT_FINAL_ASSEMBLY);* >>> * ierr = MatCreate(PETSC_COMM_WORLD,&KT);CHKERRQ(ierr);* >>> * ierr = >>> MatSetSizes(KT,PETSC_DECIDE,PETSC_DECIDE,ns*tindex_f,2);CHKERRQ(ierr);* >>> * ierr = MatSetFromOptions(KT);CHKERRQ(ierr);* >>> * int ctr=0;* >>> * for (ctr=0; ctr<2; ctr++)* >>> * {* >>> * ierr = VecCreate(PETSC_COMM_WORLD,&PzyTvec);* >>> * ierr = VecSetSizes(PzyTvec,PETSC_DECIDE,ns*tindex_f);* >>> * ierr = VecSetFromOptions(PzyTvec);* >>> * ierr = MatGetColumnVector(PzyT,PzyTvec,ctr);* >>> * >>> * >>> * ierr = KSPSolve(ksp,PzyTvec,x);CHKERRQ(ierr);* >>> * ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);* >>> * ierr = KSPGetResidualNorm(ksp,& norm); * >>> * printf("residual norm is=%e\n",norm);* >>> * printf("iteration number is=%d\n",its);* >>> * ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);* >>> * * >>> *// printf("%d\n", ctr);* >>> * for (i=0; i>> * {* >>> * ierr = VecGetValues(x,1,&i,&xvec[i]);* >>> * ierr = MatSetValues(KT,1,&i,1,&ctr,&xvec[i],INSERT_VALUES);* >>> * }* >>> * }* >>> * MatAssemblyBegin(KT,MAT_FINAL_ASSEMBLY);* >>> * MatAssemblyEnd(KT,MAT_FINAL_ASSEMBLY);* >>> * MatTranspose(KT,MAT_INITIAL_MATRIX,&K);* >>> * MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);* >>> * MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);* >>> * >>> * >>> * MatPtAP(Pzz,KT,MAT_INITIAL_MATRIX,1.0,&PK);* >>> *//MatCopy(Pzz,PzzTMP,SAME_NONZERO_PATTERN);* >>> * ierr = MatAXPY(Pmat,-1,PK,DIFFERENT_NONZERO_PATTERN);* >>> * MatView(Pmat,PETSC_VIEWER_STDOUT_WORLD);* >>> * >>> * >>> * ierr = MatAXPY(Ym,-1,Mean,DIFFERENT_NONZERO_PATTERN);* >>> * MatAssemblyBegin(Ym,MAT_FINAL_ASSEMBLY);* >>> * MatAssemblyEnd(Ym,MAT_FINAL_ASSEMBLY);* >>> * >>> * >>> * ierr = MatMatMult(K,Ym,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&KY);* >>> * ierr = MatAssemblyBegin(KY,MAT_FINAL_ASSEMBLY);* >>> * ierr = MatAssemblyEnd(KY,MAT_FINAL_ASSEMBLY);* >>> * ierr = MatAXPY(XYmat,1,KY,DIFFERENT_NONZERO_PATTERN);* >>> * MatView(XYmat,PETSC_VIEWER_STDOUT_WORLD);* >>> * >>> * >>> *ierr = MatDestroy(&Mean);* >>> * >>> * >>> *ierr=PetscFinalize();CHKERRQ(ierr);* >>> * >>> * >>> * return 0;* >>> * >>> * >>> * }* >>> * >>> * >>> >>> >>> I am using the following commands to compile and run the code: >>> >>> *mpicc pcasm.c -o pcasm -l petsc* >>> *mpiexec -np 2 ./pcasm* >>> >>> But when I run the code I get the following error message which I don't >>> know what it means. >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Floating point exception! >>> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 >>> 10:28:33 CDT 2011 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./pcasm on a linux-imp named k07n14.ccr.buffalo.edu by >>> rm93 Wed Dec 21 15:05:33 2011 >>> [0]PETSC ERROR: Libraries linked from >>> /util/petsc/petsc-3.2-p3/linux-impi-mkl/lib >>> [0]PETSC ERROR: Configure run at Fri Oct 21 08:36:23 2011 >>> [0]PETSC ERROR: Configure options --CC=/util/intel/impi/ >>> 4.0.3.008/intel64/bin/mpiicc --FC=/util/intel/impi/ >>> 4.0.3.008/intel64/bin/mpiifort --CXX=/util/intel/impi/ >>> 4.0.3.008/intel64/bin/mpiicpc--with-blas-lapack-dir=/util/intel/composer_xe_2011_sp1/mkl/lib/intel64 >>> --download-hypre=1 --with-debugging=0 -PETSC_ARCH=linux-impi-mkl >>> --with-shared-libraries=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: VecNormalize() line 261 in >>> src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: GMREScycle() line 128 in src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in >>> src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: main() line 487 in "unknowndirectory/"pcasm.c >>> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 >>> rank 0 in job 2 k07n14.ccr.buffalo.edu_51735 caused collective abort >>> of all ranks >>> exit status of rank 0: return code 72 >>> >>> >>> >>> >>> >>> >>> Could anyone help me on this problem? Thanks in advance for any help. >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Dec 21 16:55:44 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 21 Dec 2011 16:55:44 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Hailong: I've never seen this type of error from MUMPS. It seems programming bug. Are you sure smaller problem runs correctly? Use valgrind check it. Hong > I got the error from MUMPS. > > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, > nonzeros (nz) = 785860, > I could run it. And could get result. > But when I run it with > nz=3112820 > n =61240 > > > I am getting the following error > > > 17 - : Could not convert index 1140850688 into a pointer > The index may be an incorrect argument. > Possible sources of this problem are a missing "include 'mpif.h'", > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) > or a misspelled user variable for an MPI object (e.g., > com instead of comm). > [17] [] Aborting Program! > > > > Do you know what happened? > Is that possible it is running out of memory? > > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: >> >> Direct solvers often require large memory for storing matrix factors. >> As Jed suggests, you may try superlu_dist. >> >> With mumps, I notice you use parallel analysis, which is relative new in >> mumps. >> What happens if you use default sequential analysis with >> different matrix orderings? >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. >> >> Also, you can increase fill ratio, >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace >> increase (None) >> i.e., default ration is 20, you may try 50? (I notice that you already use >> 30). >> >> It seems you use 16 CPUs for "a mere couple thousands >> elements" problems, and mumps "silently freezes". I do not have this type >> of experience with mumps. I usually can solve sparse matrix of size >> 10k with 1 cpu using mumps. >> When mumps runs out of memory or gets other problems, it terminates >> execution and dumps out error message, >> not freezes. >> Something is wrong here. Use a debugger and figuring out where it freezes. >> >> Hong >> >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist >> > >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" >> > wrote: >> >> >> >> I am successfully solving my indefinite systems with MUMPS but only >> >> for very small problems. To give a feeling, a mere couple thousands >> >> elements. If I only double the problem size, it silently freezes, even >> >> with max verbosity via the control parameters. Did anyone succeed here >> >> with big problems? Any recommendations for a drop-in replacement for >> >> MUMPS? >> >> >> >> Thanks for any hints, >> >> Dominik >> >> >> >> >> >> >> >> Options used: >> >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >> >> >> >> Output: >> >> >> >> ****** FACTORIZATION STEP ******** >> >> >> >> >> >> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >> >> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >> >> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >> >> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >> >> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >> >> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >> >> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >> >> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >> >> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >> >> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >> >> >> >> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >> >> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >> >> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >> >> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >> >> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >> >> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 > > > > > -- > Hailong From mmnasr at gmail.com Wed Dec 21 18:40:44 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Wed, 21 Dec 2011 16:40:44 -0800 Subject: [petsc-users] boomerAmg scalability In-Reply-To: References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> <56F930DC-6856-4F2A-89A9-D8FBD53E9511@mcs.anl.gov> Message-ID: Thank you Jed for the detailed explanation. To tell you the truth, that was a bit overwhelming/scary for an engineer like me-that I have to be supercareful when using those solvers and preconditioners- :D For my case though, I decided to fix the pressure at one node in the domain so I won't end up with a null-space case of the Pressure equation. I assume that way, I would not have to worry about the nullspace problems and the convergence issues you pointed out, right? Best, Mohamad On Mon, Dec 19, 2011 at 9:09 PM, Jed Brown wrote: > On Mon, Dec 19, 2011 at 20:57, Barry Smith wrote: > >> So please tell use how we SHOULD use AMG with those "indefinite problem >> produced by most discretizations of incompressible flow" dear teacher :-) > > > If only there was a nice complete answer... > > We can do block preconditioners advocated by Elman and others. These are > the most flexible and the simplest for code reuse. For low Reynolds number, > they can also have optimal complexity, although the constants are usually > not the best. Most variants are well-supported by PCFieldSplit (e.g. with > PCLSC), but some need the user to provide auxiliary operators (e.g. the > "pressure convection-diffusion" variant). We could improve support for > these cases, but it's a delicate balance and I don't know any way to avoid > asking the user to understand a reasonable amount about the method and > usually to provide auxiliary information. > > We can do coupled multigrid with fieldsplit or "distributed relaxation" as > a smoother. These can often be made more robust, but they tend to be more > intrusive to implement. These are not usually purely algebraic due to > inf-sup issues when coarsening the dual variables (pressure), though Mark > Adams' work on this for contact mechanics could be used to coarsen pressure > algebraically. I would like to experiment with this in PCGAMG. > > We can do coupled multigrid with compatible Vanka-type smoothers. Whether > these are algorithmically effective and/or efficient is quite dependent on > the discretization. These methods are also usually geometric, though it's > possible to algebraically define a Vanka-smoother (though not necessarily > efficient). This is straightforward for MAC finite differences on > structured grids. For continuous finite elements, the "rotated Q1" > Rannacher-Turek elements are most attractive for these smoothers, but > Rannacher-Turek elements do not satisfy a discrete Korn's inequality, so > they are unusable for many problems. Some variants of DG for incompressible > flow seem to be the most interesting for this approach in general domains. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Dec 21 18:55:30 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Dec 2011 18:55:30 -0600 Subject: [petsc-users] boomerAmg scalability In-Reply-To: References: <9E6B0CE58F7CC24294FAB8EF9902362A75B89942E7@EXCHMBB.ornl.gov> <9E6B0CE58F7CC24294FAB8EF9902362A75B89B4875@EXCHMBB.ornl.gov> <9110062A-05FF-4096-8DC2-CDEC14E5F8CE@mcs.anl.gov> <0B0FC28A-D635-4803-9B38-4954DCE3974B@ornl.gov> <003701ccbb45$f2bb23f0$d8316bd0$@com> <56F930DC-6856-4F2A-89A9-D8FBD53E9511@mcs.anl.gov> Message-ID: On Wed, Dec 21, 2011 at 6:40 PM, Mohamad M. Nasr-Azadani wrote: > Thank you Jed for the detailed explanation. To tell you the truth, that > was a bit overwhelming/scary for an engineer like me-that I have to be > supercareful when using those solvers and preconditioners- :D > For my case though, I decided to fix the pressure at one node in the > domain so I won't end up with a null-space case of the Pressure equation. > I assume that way, I would not have to worry about the nullspace problems > and the convergence issues you pointed out, right? > Nope. You have removed the null space from the operator on the full domain, but not the subdomains that Jed was talking about. I think this method only complicates things since now one subdomain looks weird, and the behavior of the coarse operator in unpredictable since i don't know where this point is. Matt > Best, > Mohamad > > > > On Mon, Dec 19, 2011 at 9:09 PM, Jed Brown wrote: > >> On Mon, Dec 19, 2011 at 20:57, Barry Smith wrote: >> >>> So please tell use how we SHOULD use AMG with those "indefinite problem >>> produced by most discretizations of incompressible flow" dear teacher :-) >> >> >> If only there was a nice complete answer... >> >> We can do block preconditioners advocated by Elman and others. These are >> the most flexible and the simplest for code reuse. For low Reynolds number, >> they can also have optimal complexity, although the constants are usually >> not the best. Most variants are well-supported by PCFieldSplit (e.g. with >> PCLSC), but some need the user to provide auxiliary operators (e.g. the >> "pressure convection-diffusion" variant). We could improve support for >> these cases, but it's a delicate balance and I don't know any way to avoid >> asking the user to understand a reasonable amount about the method and >> usually to provide auxiliary information. >> >> We can do coupled multigrid with fieldsplit or "distributed relaxation" >> as a smoother. These can often be made more robust, but they tend to be >> more intrusive to implement. These are not usually purely algebraic due to >> inf-sup issues when coarsening the dual variables (pressure), though Mark >> Adams' work on this for contact mechanics could be used to coarsen pressure >> algebraically. I would like to experiment with this in PCGAMG. >> >> We can do coupled multigrid with compatible Vanka-type smoothers. Whether >> these are algorithmically effective and/or efficient is quite dependent on >> the discretization. These methods are also usually geometric, though it's >> possible to algebraically define a Vanka-smoother (though not necessarily >> efficient). This is straightforward for MAC finite differences on >> structured grids. For continuous finite elements, the "rotated Q1" >> Rannacher-Turek elements are most attractive for these smoothers, but >> Rannacher-Turek elements do not satisfy a discrete Korn's inequality, so >> they are unusable for many problems. Some variants of DG for incompressible >> flow seem to be the most interesting for this approach in general domains. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From wumengda at gmail.com Wed Dec 21 21:01:17 2011 From: wumengda at gmail.com (Mengda Wu) Date: Wed, 21 Dec 2011 22:01:17 -0500 Subject: [petsc-users] Least Squares options in Petsc Message-ID: Hi all, I am trying to solve an overdetermined system using Petsc. I am using un-preconditioned LSQR but it seems that it converges slowly. I am woer ndering what preconditioner may improve the convergence. Or are there any other ksp I can use? Thanks, Mengda -------------- next part -------------- An HTML attachment was scrubbed... URL: From rm93 at buffalo.edu Wed Dec 21 22:10:28 2011 From: rm93 at buffalo.edu (Reza Madankan) Date: Wed, 21 Dec 2011 23:10:28 -0500 Subject: [petsc-users] parallel PCASM In-Reply-To: References: Message-ID: Thanks a lot Matthew. That was a great help. It seems I was completely out of the track. I am looking at what you said, and made good progress til here... Thanks again Reza On Wed, Dec 21, 2011 at 4:40 PM, Matthew Knepley wrote: > On Wed, Dec 21, 2011 at 3:30 PM, Reza Madankan wrote: > >> Hi Matthew; >> Thank you for you quick reply. >> Sorry for the wrong terminology, it's just few days I started working >> with Petsc. >> The whole story is that I have a linear system Ax = b which I want to >> solve it using different preconditioners just to see the effect of each of >> those on the solution and residual error. I have already used PCJACOBI and >> PCILU to find the solution. I also would like to see the effect of using >> PCASM. PCASM on a single processor performs like PCILU, but its result >> will be different from PCILU if I implement it on multiple processors. So, >> I am trying to get it run for multiple processors case using the following >> command: >> >> mpiexec -np 2 ./pcasm >> >> but I get the error message that I showed before. Now my question is that >> how I can implement PCASM on multiple processors? >> > > It is not ASM that is the problem. You have a bug in your code. Use > valgrind to find it. > > The larger problem here is that you have completely misunderstood the > point of PETSc. > Whatever you do, never change your code to use a different solver. You > should play > with the PETSc examples for a while > > cd src/snes/examples/tutorials > make ex5 > ./ex5 -snes_monitor -ksp_monitor -snes_view > $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view > $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view -pc_type jacobi > $MPIEXEC -n 2 ./ex5 -snes_monitor -ksp_monitor -snes_view -pc_type asm > > Matt > > Thanks again >> Reza >> >> >> >> On Wed, Dec 21, 2011 at 3:19 PM, Matthew Knepley wrote: >> >>> On Wed, Dec 21, 2011 at 2:12 PM, Reza Madankan wrote: >>> >>>> Hello, >>>> I am trying to parallelize the ASM preconditioner while solving a >>>> linear system. For more detail, I have printed relevant parts of code in >>>> the following: >>>> >>> >>> I have no idea what "parallelize ASM" might mean. >>> >>> Below you likely have a bad value in b, the rhs you pass to the solver >>> on line 487 of your code. >>> >>> Matt >>> >>> >>>> *static char help[]="Reading in a matrix\n";* >>>> * >>>> * >>>> * #include* >>>> * >>>> * >>>> * #include* >>>> * >>>> * >>>> * #include* >>>> * >>>> * >>>> * #include "petscvec.h" /* This enables us to use vectors. */* >>>> * >>>> * >>>> * #include "petscmat.h" /* This enables us to use Matrices. It >>>> includes the* >>>> * petscvec header file*/* >>>> * >>>> * >>>> * #include "petscksp.h" /* Now we can solve linear systems. Solvers >>>> used are* >>>> * KSP. */* >>>> * >>>> * >>>> * extern PetscErrorCode MyKSPMonitor(KSP,PetscInt,PetscReal,void*);* >>>> * >>>> * >>>> * int main(int argc, char **argv)* >>>> * >>>> * >>>> * {* >>>> * >>>> * >>>> * /* Declaration of Matrix A and some vectors x*/* >>>> * >>>> * >>>> * Vec w, ypcq, mean_tmp, x, PzyTvec;* >>>> * Mat Ym, Y, Mean, Ypcq, YpcqT, Pzz, InnProd, Para, Pzy, PzyT,PzzT,K, >>>> KT, PK, PzzTMP, Pmat, XYmat, KY;* >>>> * >>>> * >>>> * FILE *fp, *fp1, *fp2, *fp3;* >>>> * >>>> * >>>> * PetscInt index,i,j,k, ns=16, nw=100, tindex_f=120, col,tind, mz, nz, >>>> its;* >>>> */* * >>>> *printf("Enter number of sensors (ns):\n");* >>>> *scanf("%d",&ns);* >>>> * >>>> * >>>> *printf("Enter number of time steps (tindex_f):\n");* >>>> *scanf("%d",&tindex_f);* >>>> **/* >>>> * PetscMPIInt size;* >>>> * >>>> * >>>> * PC pc;* >>>> * KSP ksp;* >>>> * PetscReal norm, tol=1.e-14;* >>>> * PetscBool nonzeroguess = PETSC_FALSE;* >>>> * PetscViewer mviewer;* >>>> * >>>> * >>>> * >>>> * >>>> * PetscScalar scalar,rhsy[ns*tindex_f*nw],rhsym[ns*tindex_f], >>>> rhsw[nw], Ymat[tindex_f*ns][nw], tmp, rhsp[2*nw], xvec[ns*tindex_f], >>>> pmat[4], xymat[2], rnorm;* >>>> * >>>> * >>>> * PetscErrorCode ierr;* >>>> * >>>> * >>>> * /* This part is needed for the help flag supplied at run-time*/* >>>> * >>>> * >>>> * ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);* >>>> * * >>>> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >>>> * //if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor >>>> example");* >>>> *PetscPrintf(PETSC_COMM_WORLD,"size is: %D\n",size);* >>>> *int nstindex_f=ns*tindex_f;* >>>> *ierr = >>>> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >>>> * >>>> * >>>> * >>>> *.* >>>> *.* >>>> *.* >>>> * >>>> * >>>> * >>>> * >>>> *// Solution for K:* >>>> *// --------------* >>>> * >>>> * >>>> * ierr = MPI_Comm_size(PETSC_COMM_WORLD, &size);CHKERRQ(ierr);* >>>> * ierr = >>>> PetscOptionsGetInt(PETSC_NULL,"-n",&nstindex_f,PETSC_NULL);CHKERRQ(ierr); >>>> * >>>> * ierr = >>>> PetscOptionsGetBool(PETSC_NULL,"-nonzero_guess",&nonzeroguess,PETSC_NULL); >>>> CHKERRQ(ierr);* >>>> * >>>> * >>>> * >>>> * >>>> *// KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL,0);* >>>> * >>>> * >>>> * >>>> * >>>> * ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);* >>>> * ierr = PetscObjectSetName((PetscObject) x, >>>> "Solution");CHKERRQ(ierr);* >>>> * ierr = VecSetSizes(x,PETSC_DECIDE,ns*tindex_f);CHKERRQ(ierr);* >>>> * ierr = VecSetFromOptions(x);CHKERRQ(ierr);* >>>> * ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);* >>>> * * >>>> * >>>> * >>>> *// ierr = KSPMonitorSet(ksp,PetscInt n,PetscReal rnorm, void *); * >>>> *// ierr = monitor(ksp,Int it,PetscReal rnorm,);* >>>> * * >>>> * >>>> * >>>> * ierr = MatTranspose(Pzz,MAT_INITIAL_MATRIX,&PzzT);* >>>> * MatAssemblyBegin(PzzT,MAT_FINAL_ASSEMBLY);* >>>> * MatAssemblyEnd(PzzT,MAT_FINAL_ASSEMBLY);* >>>> * ierr = >>>> KSPSetOperators(ksp,PzzT,PzzT,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); >>>> * >>>> * >>>> * >>>> * ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);* >>>> * ierr = PCSetType(pc,PCASM);CHKERRQ(ierr);* >>>> * ierr = >>>> KSPSetTolerances(ksp,1e-6,PETSC_DEFAULT,PETSC_DEFAULT,1e+9);CHKERRQ(ierr); >>>> * >>>> * ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);* >>>> * if (nonzeroguess)* >>>> * {* >>>> * PetscScalar p = 0.5;* >>>> * ierr = VecSet(x,p);CHKERRQ(ierr);* >>>> * ierr = >>>> KSPSetInitialGuessNonzero(ksp,PETSC_TRUE);CHKERRQ(ierr);* >>>> * }* >>>> * ierr = MatTranspose(Pzy,MAT_INITIAL_MATRIX,&PzyT);* >>>> * MatAssemblyBegin(PzyT,MAT_FINAL_ASSEMBLY);* >>>> * MatAssemblyEnd(PzyT,MAT_FINAL_ASSEMBLY);* >>>> * ierr = MatCreate(PETSC_COMM_WORLD,&KT);CHKERRQ(ierr);* >>>> * ierr = >>>> MatSetSizes(KT,PETSC_DECIDE,PETSC_DECIDE,ns*tindex_f,2);CHKERRQ(ierr);* >>>> * ierr = MatSetFromOptions(KT);CHKERRQ(ierr);* >>>> * int ctr=0;* >>>> * for (ctr=0; ctr<2; ctr++)* >>>> * {* >>>> * ierr = VecCreate(PETSC_COMM_WORLD,&PzyTvec);* >>>> * ierr = VecSetSizes(PzyTvec,PETSC_DECIDE,ns*tindex_f);* >>>> * ierr = VecSetFromOptions(PzyTvec);* >>>> * ierr = MatGetColumnVector(PzyT,PzyTvec,ctr);* >>>> * >>>> * >>>> * ierr = KSPSolve(ksp,PzyTvec,x);CHKERRQ(ierr);* >>>> * ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);* >>>> * ierr = KSPGetResidualNorm(ksp,& norm); * >>>> * printf("residual norm is=%e\n",norm);* >>>> * printf("iteration number is=%d\n",its);* >>>> * ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);* >>>> * * >>>> *// printf("%d\n", ctr);* >>>> * for (i=0; i>>> * {* >>>> * ierr = VecGetValues(x,1,&i,&xvec[i]);* >>>> * ierr = MatSetValues(KT,1,&i,1,&ctr,&xvec[i],INSERT_VALUES);* >>>> * }* >>>> * }* >>>> * MatAssemblyBegin(KT,MAT_FINAL_ASSEMBLY);* >>>> * MatAssemblyEnd(KT,MAT_FINAL_ASSEMBLY);* >>>> * MatTranspose(KT,MAT_INITIAL_MATRIX,&K);* >>>> * MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);* >>>> * MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);* >>>> * >>>> * >>>> * MatPtAP(Pzz,KT,MAT_INITIAL_MATRIX,1.0,&PK);* >>>> *//MatCopy(Pzz,PzzTMP,SAME_NONZERO_PATTERN);* >>>> * ierr = MatAXPY(Pmat,-1,PK,DIFFERENT_NONZERO_PATTERN);* >>>> * MatView(Pmat,PETSC_VIEWER_STDOUT_WORLD);* >>>> * >>>> * >>>> * ierr = MatAXPY(Ym,-1,Mean,DIFFERENT_NONZERO_PATTERN);* >>>> * MatAssemblyBegin(Ym,MAT_FINAL_ASSEMBLY);* >>>> * MatAssemblyEnd(Ym,MAT_FINAL_ASSEMBLY);* >>>> * >>>> * >>>> * ierr = MatMatMult(K,Ym,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&KY);* >>>> * ierr = MatAssemblyBegin(KY,MAT_FINAL_ASSEMBLY);* >>>> * ierr = MatAssemblyEnd(KY,MAT_FINAL_ASSEMBLY);* >>>> * ierr = MatAXPY(XYmat,1,KY,DIFFERENT_NONZERO_PATTERN);* >>>> * MatView(XYmat,PETSC_VIEWER_STDOUT_WORLD);* >>>> * >>>> * >>>> *ierr = MatDestroy(&Mean);* >>>> * >>>> * >>>> *ierr=PetscFinalize();CHKERRQ(ierr);* >>>> * >>>> * >>>> * return 0;* >>>> * >>>> * >>>> * }* >>>> * >>>> * >>>> >>>> >>>> I am using the following commands to compile and run the code: >>>> >>>> *mpicc pcasm.c -o pcasm -l petsc* >>>> *mpiexec -np 2 ./pcasm* >>>> >>>> But when I run the code I get the following error message which I don't >>>> know what it means. >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Floating point exception! >>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 >>>> 10:28:33 CDT 2011 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./pcasm on a linux-imp named k07n14.ccr.buffalo.edu by >>>> rm93 Wed Dec 21 15:05:33 2011 >>>> [0]PETSC ERROR: Libraries linked from >>>> /util/petsc/petsc-3.2-p3/linux-impi-mkl/lib >>>> [0]PETSC ERROR: Configure run at Fri Oct 21 08:36:23 2011 >>>> [0]PETSC ERROR: Configure options --CC=/util/intel/impi/ >>>> 4.0.3.008/intel64/bin/mpiicc --FC=/util/intel/impi/ >>>> 4.0.3.008/intel64/bin/mpiifort --CXX=/util/intel/impi/ >>>> 4.0.3.008/intel64/bin/mpiicpc--with-blas-lapack-dir=/util/intel/composer_xe_2011_sp1/mkl/lib/intel64 >>>> --download-hypre=1 --with-debugging=0 -PETSC_ARCH=linux-impi-mkl >>>> --with-shared-libraries=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: VecNormalize() line 261 in >>>> src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: GMREScycle() line 128 in src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in >>>> src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: main() line 487 in "unknowndirectory/"pcasm.c >>>> application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 >>>> rank 0 in job 2 k07n14.ccr.buffalo.edu_51735 caused collective abort >>>> of all ranks >>>> exit status of rank 0: return code 72 >>>> >>>> >>>> >>>> >>>> >>>> >>>> Could anyone help me on this problem? Thanks in advance for any help. >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 22 08:46:00 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 22 Dec 2011 08:46:00 -0600 Subject: [petsc-users] Petsc course In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 08:16, Sreejith Pulloor Kuttanikkad < sreejithpk at gmail.com> wrote: > Could you please provide more details on the TACC courses on 17 and 20? > Where is it? and do we get to do exercises ? Both courses are at TACC (UT Austin), but I think they will be broadcast through XSEDE. The course on Jan 17, taught by Victor Eijkhout, will be similar to last year's (search for "PETSc" at http://www.tacc.utexas.edu/user-services/training/course-materials). My Advanced PETSc course on Feb 20 will look at * multigrid and fieldsplit preconditioning for saddle point and stiff wave problems * composition of nonlinear solvers * solving variational inequalities * time integration * software design for coupled problems * extending PETSc Have you looked at slides and video from previous tutorials? http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 22 10:15:23 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 22 Dec 2011 10:15:23 -0600 Subject: [petsc-users] Least Squares options in Petsc In-Reply-To: References: Message-ID: On Dec 21, 2011, at 9:01 PM, Mengda Wu wrote: > Hi all, > > I am trying to solve an overdetermined system using Petsc. I am using un-preconditioned LSQR but it seems that it converges slowly. I am woer ndering what preconditioner may improve the convergence. Or > are there any other ksp I can use? What are the typical sizes of the matrix? Note that when running with preconditioning you need to call KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN) where B is A'*A which can be obtained with MatMatMultTranspose(). (in petsc-3.2). Then you can try -pc_type lu -pc_type ilu or any other preconditioner. Barry > > Thanks, > Mengda From bsmith at mcs.anl.gov Thu Dec 22 13:57:03 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 22 Dec 2011 13:57:03 -0600 Subject: [petsc-users] Using Petsc with multiple RHS In-Reply-To: <4EE5C4FE.1040404@gfz-potsdam.de> References: <4EE4CC68.2010605@gfz-potsdam.de> <4EE5C4FE.1040404@gfz-potsdam.de> Message-ID: The PETSc interface for MatMatSolve() for SuperLU_Dist does directly use the multiple right hand side solver of SuperLU so perhaps you can use this as it will be faster than one solve at a time. As Hong mentioned, MUMPS still requires centralizing the entire right hand side to one process before starting the linear solver. This is why we don't support multiple right hand sides directly with MUMPs, that would require too much memory on the first process. When MUMPS supports distributed right hand side then we'll add support for directly using their multiple right hand side solver. Barry On Dec 12, 2011, at 3:10 AM, Alexander Grayver wrote: > Hi Barry, > > Thanks for answer. I should have asked about that from the very beginning actually. > I get 2 times decrease in performance with 20 RHS, I can imagine how slow it will be when I will use thousands of them, > moreover it will require a lot of memory to store them as a dense matrix. > > On 11.12.2011 18:50, Barry Smith wrote: >> On Dec 11, 2011, at 9:29 AM, Alexander Grayver wrote: >> One by one > > I'm wondering why? All main direct packages like MUMPS, SuperLU_DIST, PaSTiX support multiple RHS. > >> We do not handle a sparse right hand side. > > Since I already transferred my code to PETSc anyway, my question now if it's possible to implement sparse multiple RHS and solve them simultaneously? > Something like MatMatSolveSparse. I would implement it myself, the question is there a way to integrate it into PETSc as a patch or something like that? > > Regards, > Alexander From mmnasr at gmail.com Thu Dec 22 19:20:34 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Thu, 22 Dec 2011 17:20:34 -0800 Subject: [petsc-users] KSP diverging Message-ID: Hello, I am using GMRES + Hypre (BoomerAMG) as the preconditioner to solve the Poisson equation for pressure. Pressure linear system has a nullspace (Neumann boundary condition everywhere). After a long time of time integration of my CFD simulations, all of the sudden, my code starts diverging, i.e. delta_t = 0. I am still trying to find what exactly the problem is, but what I have faced is when I solve the Poisson equation with the given KSP and PC, the solution does not converge due to KSP_DIVERGED_DTOL Any hints? I have tried to fix the pressure at one node and also use the MatNullSpace to handle that, no luck and both cases diverge. The interesting thing is that when I use GMRES and other preconditioners, e.g. PCASM or PCJACOBI, the linear system does not converge at all. It reaches the maximum number of iterations (iter=10000) which technically should not since the linear system is not that big. There is only 1.6 million grid points in the domain. Any hints on what is happening here or how I could narrow it down? I know that we had an early discussion on the convergnce for the incompressible CFD problems and boomerAMG preconditioning, but it seems that for this case, that is the only one that does not blow up for most of the time integration I have. I am running it on 24 processors and 1.6 million grid points. Thanks in advance, Best, Mohamad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 22 19:31:04 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 22 Dec 2011 19:31:04 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 19:20, Mohamad M. Nasr-Azadani wrote: > Any hints on what is happening here or how I could narrow it down? I would start by following the general advice in the FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged > I know that we had an early discussion on the convergnce for the > incompressible CFD problems and boomerAMG preconditioning, That was about solving the coupled system. That shouldn't be an issue for the Poisson problem. > but it seems that for this case, that is the only one that does not blow > up for most of the time integration I have. Since this problem only occurs after long time integration, you might consider writing out the model state when you see divergence so that you can experiment with alternative methods quickly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmnasr at gmail.com Thu Dec 22 19:37:02 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Thu, 22 Dec 2011 17:37:02 -0800 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: Thanks so much Jed for your prompt response. I will start experimenting with all the suggestions. Best, Mohamad On Thu, Dec 22, 2011 at 5:31 PM, Jed Brown wrote: > On Thu, Dec 22, 2011 at 19:20, Mohamad M. Nasr-Azadani wrote: > >> Any hints on what is happening here or how I could narrow it down? > > > I would start by following the general advice in the FAQ > > http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged > > >> I know that we had an early discussion on the convergnce for the >> incompressible CFD problems and boomerAMG preconditioning, > > > That was about solving the coupled system. That shouldn't be an issue for > the Poisson problem. > > >> but it seems that for this case, that is the only one that does not blow >> up for most of the time integration I have. > > > Since this problem only occurs after long time integration, you might > consider writing out the model state when you see divergence so that you > can experiment with alternative methods quickly. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 22 19:46:45 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 22 Dec 2011 19:46:45 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 7:37 PM, Mohamad M. Nasr-Azadani wrote: > Thanks so much Jed for your prompt response. I will start experimenting > with all the suggestions. dtol failure is impossible for GMRES (and CG). Consider checking with that. Matt > Best, > Mohamad > > > On Thu, Dec 22, 2011 at 5:31 PM, Jed Brown wrote: > >> On Thu, Dec 22, 2011 at 19:20, Mohamad M. Nasr-Azadani wrote: >> >>> Any hints on what is happening here or how I could narrow it down? >> >> >> I would start by following the general advice in the FAQ >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged >> >> >>> I know that we had an early discussion on the convergnce for the >>> incompressible CFD problems and boomerAMG preconditioning, >> >> >> That was about solving the coupled system. That shouldn't be an issue for >> the Poisson problem. >> >> >>> but it seems that for this case, that is the only one that does not blow >>> up for most of the time integration I have. >> >> >> Since this problem only occurs after long time integration, you might >> consider writing out the model state when you see divergence so that you >> can experiment with alternative methods quickly. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 22 19:51:37 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 22 Dec 2011 19:51:37 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 19:46, Matthew Knepley wrote: > dtol failure is impossible for GMRES (and CG). Consider checking with that. Well, if you turn of restarts in GMRES and if orthogonalization is working correctly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmnasr at gmail.com Thu Dec 22 20:50:09 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Thu, 22 Dec 2011 18:50:09 -0800 Subject: [petsc-users] -ksp_diagonal_scale Message-ID: Hi guys, Three questions: 1) As mentioned on the web when we use *-ksp_diagonal_scale_fix: *scale the matrix back AFTER the solve with -ksp_diagonal_scale does it also scale back the RHS vector associated with the KSP solver? 2) Also, as mentioned on the web for -ksp_diagonal_scale "Tells KSP to symmetrically diagonally scale the system before solving. " When I used this option to solve for the Pressure (Poisson equation with Neumann B.C., descritized via structured fisytenite-difference method), the number of iterations for convergence was trippled, e.g. without this option, it converges in 18 iteration, but with this option it goes up to 50-60 iterations. (GMRES + BoomerAMG as the preconditioner). Is this something you would expect? 3) How could the linear system resulting from the Poisson equation for pressure (incompressible flow) can be more diagonally scaled? To elaborate more, for the uniform grid case and let's say a second order central scheme finite difference discretization, the sum of the off-diagonal coefficients are equal to the diagonal coefficient. Thanks, Best, Mohamad -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Dec 23 04:11:04 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 23 Dec 2011 11:11:04 +0100 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: References: Message-ID: <4EF453B8.9010700@gfz-potsdam.de> Hi, Sorry for small offtopic. I would also be interested more to know how it scales system? I mean what is idea behind this scaling. Regards, Alexander On 23.12.2011 03:50, Mohamad M. Nasr-Azadani wrote: > Hi guys, > > Three questions: > 1) As mentioned on the web when we use > *-ksp_diagonal_scale_fix: *scale the matrix back AFTER the solve > > with -ksp_diagonal_scale does it also scale back the RHS vector > associated with the KSP solver? > > 2) Also, as mentioned on the web for -ksp_diagonal_scale > "Tells KSP > to > symmetrically diagonally scale the system before solving. " > > When I used this option to solve for the Pressure (Poisson equation > with Neumann B.C., descritized via structured fisytenite-difference > method), the number of iterations for convergence was trippled, e.g. > without this option, it converges in 18 iteration, but with this > option it goes up to 50-60 iterations. > (GMRES + BoomerAMG as the preconditioner). > Is this something you would expect? > > 3) How could the linear system resulting from the Poisson equation for > pressure (incompressible flow) can be more diagonally scaled? To > elaborate more, for the uniform grid case and let's say a second order > central scheme finite difference discretization, the sum of the > off-diagonal coefficients are equal to the diagonal coefficient. > > Thanks, > Best, > Mohamad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 23 07:43:54 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 23 Dec 2011 07:43:54 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 7:51 PM, Jed Brown wrote: > On Thu, Dec 22, 2011 at 19:46, Matthew Knepley wrote: > >> dtol failure is impossible for GMRES (and CG). Consider checking with >> that. > > > Well, if you turn of restarts in GMRES and if orthogonalization is working > correctly. > Even if you lose orthogonality, it will just stagnate since you always have the solution from the first few vectors. Likewise, I do not see how you can get an diverging sequence from restarting since you always have the last residual, you just stagnate. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 23 08:38:26 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Dec 2011 15:38:26 +0100 Subject: [petsc-users] Petsc course In-Reply-To: References: Message-ID: > Both courses are at TACC (UT Austin), but I think they will be broadcast > through XSEDE. The course on Jan 17, taught by Victor Eijkhout, will be > similar to last year's (search for "PETSc" at > http://www.tacc.utexas.edu/user-services/training/course-materials). My > Advanced PETSc course on Feb 20 will look at > > * multigrid and fieldsplit preconditioning for saddle point and stiff wave > problems > * composition of nonlinear solvers > * solving variational inequalities > * time integration > * software design for coupled problems > * extending PETSc I would be very glad to see the video and slides... Do you plan to make them available? Dominik From jedbrown at mcs.anl.gov Fri Dec 23 09:23:04 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 09:23:04 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 07:43, Matthew Knepley wrote: > Even if you lose orthogonality, it will just stagnate since you always > have the solution > from the first few vectors. Likewise, I do not see how you can get an > diverging sequence > from restarting since you always have the last residual, you just stagnate. > The true residual can grow even though the estimate of the preconditioned residual decreases monotonically. When you restart, you re-evaluate the actual residual, which might be much larger. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 23 12:52:25 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Dec 2011 19:52:25 +0100 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: <4EF453B8.9010700@gfz-potsdam.de> References: <4EF453B8.9010700@gfz-potsdam.de> Message-ID: I am scaling my systems when dealing with small (microscopic) scales in SI dimensions. This is because otherwise I have very small (near epsilon) entries in the matrix and the solution fails or takes significantly longer to converge. I used to do it by hand so far, if there is a way to do it in Petsc - especially automatically detecting the optimal scale - I am all ears. Dominik On Fri, Dec 23, 2011 at 11:11 AM, Alexander Grayver wrote: > Hi, > > Sorry for small offtopic. > I would also be interested more to know how it scales system? I mean what is > idea behind this scaling. > > Regards, > Alexander > > > On 23.12.2011 03:50, Mohamad M. Nasr-Azadani wrote: > > Hi guys, > > Three questions: > 1) As mentioned on the web when we use > -ksp_diagonal_scale_fix:?scale the matrix back AFTER the solve > > with -ksp_diagonal_scale does it also scale back the RHS vector associated > with the KSP solver? > > 2) Also, as mentioned on the web for -ksp_diagonal_scale > "Tells?KSP?to symmetrically diagonally scale the system before solving.?" > > When I used this option to solve for the Pressure (Poisson equation with > Neumann B.C., descritized via structured fisytenite-difference method), the > number of iterations for convergence was trippled, e.g. without this option, > it converges in 18 iteration, but with this option it goes up to 50-60 > iterations. > (GMRES + BoomerAMG as the preconditioner). > Is this something you would expect? > > 3) How could the linear system resulting from the Poisson equation for > pressure (incompressible flow) can be more diagonally scaled? To elaborate > more, for the uniform grid case and let's say a second order central scheme > finite difference discretization, the sum of the off-diagonal coefficients > are equal to the diagonal coefficient. > > Thanks, > Best, > Mohamad > > From jedbrown at mcs.anl.gov Fri Dec 23 13:21:17 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 13:21:17 -0600 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 20:50, Mohamad M. Nasr-Azadani wrote: > 1) As mentioned on the web when we use > *-ksp_diagonal_scale_fix: *scale the matrix back AFTER the solve > > with -ksp_diagonal_scale does it also scale back the RHS vector > associated with the KSP solver? > yes > > 2) Also, as mentioned on the web for -ksp_diagonal_scale > "Tells KSP to > symmetrically diagonally scale the system before solving. " > > When I used this option to solve for the Pressure (Poisson equation with > Neumann B.C., descritized via structured fisytenite-difference method), the > number of iterations for convergence was trippled, e.g. without this > option, it converges in 18 iteration, but with this option it goes up to > 50-60 iterations. > (GMRES + BoomerAMG as the preconditioner). > Is this something you would expect? > It's perhaps slightly surprising that it's that much worse, but there is nothing saying that the diagonal scaled matrix will be better. Most preconditioners correct the diagonal scaling. > > 3) How could the linear system resulting from the Poisson equation for > pressure (incompressible flow) can be more diagonally scaled? To elaborate > more, for the uniform grid case and let's say a second order central scheme > finite difference discretization, the sum of the off-diagonal coefficients > are equal to the diagonal coefficient. > You probably don't have a reason to use diagonal scaling. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 23 13:24:50 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 13:24:50 -0600 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: <4EF453B8.9010700@gfz-potsdam.de> References: <4EF453B8.9010700@gfz-potsdam.de> Message-ID: On Fri, Dec 23, 2011 at 04:11, Alexander Grayver wrote: > I would also be interested more to know how it scales system? I mean what > is idea behind this scaling. http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/KSP/KSPSetDiagonalScale.html It takes the inverse square root of the diagonal and does symmetric scaling. If you used this with no additional preconditioner, it would be like preconditioning with Jacobi. The point is to correct some ill-scaling *before* sending the matrix to an algebraic preconditioner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 23 13:27:26 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 13:27:26 -0600 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: References: <4EF453B8.9010700@gfz-potsdam.de> Message-ID: On Fri, Dec 23, 2011 at 12:52, Dominik Szczerba wrote: > I am scaling my systems when dealing with small (microscopic) scales > in SI dimensions. > The best choice is to choose suitable units so that the system is intrinsically well-scaled. > This is because otherwise I have very small (near > epsilon) entries in the matrix and the solution fails or takes > significantly longer to converge. I used to do it by hand so far, if > there is a way to do it in Petsc - especially automatically detecting > the optimal scale - I am all ears. > You are best off doing it by hand, it is better to avoid -ksp_diagonal_scale when it's reasonable to do so. If you have trouble determining a reasonable scale at assembly time, it might make sense to use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 23 13:31:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 23 Dec 2011 13:31:06 -0600 Subject: [petsc-users] -ksp_diagonal_scale In-Reply-To: References: <4EF453B8.9010700@gfz-potsdam.de> Message-ID: On Fri, Dec 23, 2011 at 1:27 PM, Jed Brown wrote: > On Fri, Dec 23, 2011 at 12:52, Dominik Szczerba wrote: > >> I am scaling my systems when dealing with small (microscopic) scales >> in SI dimensions. >> > > The best choice is to choose suitable units so that the system is > intrinsically well-scaled. > I think the best approach is always to non-dimensionalize after taking input in your favorite units. We do this in PyLith. On output, just put the dimension back in. It also lets you see the scaling factors explicitly, which is useful for reasoning physically about the problem. Matt > > >> This is because otherwise I have very small (near >> epsilon) entries in the matrix and the solution fails or takes >> significantly longer to converge. I used to do it by hand so far, if >> there is a way to do it in Petsc - especially automatically detecting >> the optimal scale - I am all ears. >> > > You are best off doing it by hand, it is better to avoid > -ksp_diagonal_scale when it's reasonable to do so. If you have trouble > determining a reasonable scale at assembly time, it might make sense to use. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 23 15:34:04 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Dec 2011 22:34:04 +0100 Subject: [petsc-users] error attaching debugger (again) Message-ID: I again can not -start_in_debugger, here is the output: mpiexec -n 2 Solver run.xml -start_in_debugger -display :0.0 [0]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16633 on display :0.0 on machine tharsis [1]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16634 on display :0.0 on machine tharsis xterm Xt error: Can't open display: :0.0 xterm Xt error: Can't open display: :0.0 I could always go along with the first two messages so far, xterms would open. But now with no apparent reason or system change (other than usual systematic system updates) the additional errors appear and xterms do not open. This is Ubuntu 11.10 64 bit. Any ideas what to check please? I saw the FAQ about ptrace, it solver my problem once before, but now it does not seem to be the case. Regards, Dominik From jedbrown at mcs.anl.gov Fri Dec 23 15:37:06 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 15:37:06 -0600 Subject: [petsc-users] error attaching debugger (again) In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 15:34, Dominik Szczerba wrote: > mpiexec -n 2 Solver run.xml -start_in_debugger -display :0.0 > [0]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16633 on display > :0.0 on machine tharsis > [1]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16634 on display > :0.0 on machine tharsis > xterm Xt error: Can't open display: :0.0 > xterm Xt error: Can't open display: :0.0 > -start_in_debugger usually works correctly without also adding the -display option. If that fails, you need to figure out what does work. You can try xterm -display tharsis:0 perhaps fix xhost, perhaps use ssh -X. Once you have figured out how to forward the display with a simple program, try the PETSc program. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Dec 23 15:48:40 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 15:48:40 -0600 Subject: [petsc-users] Petsc course In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 08:38, Dominik Szczerba wrote: > I would be very glad to see the video and slides... Do you plan to > make them available? > The slides will be made available. If TACC records video, we will post it too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 23 15:55:54 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Dec 2011 22:55:54 +0100 Subject: [petsc-users] error attaching debugger (again) In-Reply-To: References: Message-ID: >> mpiexec -n 2 Solver run.xml -start_in_debugger -display :0.0 >> [0]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16633 on display >> :0.0 on machine tharsis >> [1]PETSC ERROR: PETSC: Attaching gdb to Solver of pid 16634 on display >> :0.0 on machine tharsis >> xterm Xt error: Can't open display: :0.0 >> xterm Xt error: Can't open display: :0.0 > > > -start_in_debugger usually works correctly without also adding the -display > option. If that fails, you need to figure out what does work. You can try > > xterm -display tharsis:0 > > perhaps fix xhost, perhaps use ssh -X. Once you have figured out how to > forward the display with a simple program, try the PETSc program. xtern and other x11 apps were fine as well as xtern and ssh forwarding. I once had to add -display for the very same error. But now somehow removing it brings xterms up again... Strainge... Thanks a lot Dominik From jedbrown at mcs.anl.gov Fri Dec 23 16:04:26 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 16:04:26 -0600 Subject: [petsc-users] error attaching debugger (again) In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 15:55, Dominik Szczerba wrote: > xtern and other x11 apps were fine as well as xtern and ssh > forwarding. I once had to add -display for the very same error. But > now somehow removing it brings xterms up again... Strainge... > I changed the default a year or two ago so that it work work on most "normally configured" systems. It might work if you specify "-display :0" (instead of "-display :0.0"), but if the default works without a -display argument, that's great. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 23 16:36:31 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Dec 2011 23:36:31 +0100 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Hi Hong, I tried all of your suggestions, unfortunately, still get MUMPS not moving an inch. Running in debugger and interrupting reveals: 0x000000000139d233 in dgemm (transa=..., transb=..., m=3621, n=3621, k=251, alpha=-1, a=..., lda=3872, b=..., ldb=3872, beta=1, c=..., ldc=3872, _transa=1, _transb=1) at dgemm.f:242 242 C(I,J) = C(I,J) + TEMP*A(I,L) So it's seems to be doing something, but on a small problem it takes a few minutes, so I do not expect a problem of 2-3 times the size take longer than a day... Regards, Dominik On Wed, Dec 21, 2011 at 4:15 PM, Hong Zhang wrote: > Direct solvers often require large memory for storing matrix factors. > As Jed suggests, you may try superlu_dist. > > With mumps, I notice you use parallel analysis, which is relative new in mumps. > What happens if you use default sequential analysis with > different matrix orderings? > I usually use matrix ordering '-mat_mumps_icntl_7 2'. > > Also, you can increase fill ratio, > -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace > increase (None) > i.e., default ration is 20, you may try 50? (I notice that you already use 30). > > It seems you use 16 CPUs for "a mere couple thousands > elements" problems, and mumps "silently freezes". I do not have this type > of experience with mumps. I usually can solve sparse matrix of size > 10k with 1 cpu using mumps. > When mumps runs out of memory or gets other problems, it terminates > execution and dumps out error message, > not freezes. > Something is wrong here. Use a debugger and figuring out where it freezes. > > Hong > > On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >> -pc_type lu -pc_factor_mat_solver_package superlu_dist >> >> On Dec 21, 2011 6:19 AM, "Dominik Szczerba" wrote: >>> >>> I am successfully solving my indefinite systems with MUMPS but only >>> for very small problems. To give a feeling, a mere couple thousands >>> elements. If I only double the problem size, it silently freezes, even >>> with max verbosity via the control parameters. Did anyone succeed here >>> with big problems? Any recommendations for a drop-in replacement for >>> MUMPS? >>> >>> Thanks for any hints, >>> Dominik >>> >>> >>> >>> Options used: >>> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >>> >>> Output: >>> >>> ****** FACTORIZATION STEP ******** >>> >>> >>> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >>> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >>> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >>> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >>> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >>> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >>> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >>> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >>> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >>> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >>> >>> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >>> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >>> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >>> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >>> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >>> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 > From jedbrown at mcs.anl.gov Fri Dec 23 16:40:26 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 23 Dec 2011 16:40:26 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 16:36, Dominik Szczerba wrote: > I tried all of your suggestions, unfortunately, still get MUMPS not > moving an inch. Running in debugger and interrupting reveals: > > 0x000000000139d233 in dgemm (transa=..., transb=..., m=3621, n=3621, k=251, > alpha=-1, a=..., lda=3872, b=..., ldb=3872, beta=1, c=..., ldc=3872, > _transa=1, _transb=1) at dgemm.f:242 > 242 C(I,J) = C(I,J) + TEMP*A(I,L) > > So it's seems to be doing something, but on a small problem it takes a > few minutes, so I do not expect a problem of 2-3 times the size take > longer than a day... > Is one mesh just a refined version of the other, or is the shape different? Factorization time should be roughly quadratic in problem size for 3D problems. If the increased fill caused the system to run out of physical memory and start swapping, then it would not be surprising for it to take days. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Fri Dec 23 16:56:38 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Fri, 23 Dec 2011 16:56:38 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: It looks like it's due to mixing different MPI implementations together (i.e., including the wrong 'mpif.h'): http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-July/007559.html If I recall correctly, MUMPS only uses ScaLAPACK to factor the root separator when it is sufficiently large, and that would explain why it works for him for smaller problems. I would double check that ScaLAPACK, PETSc, and MUMPS are all compiled with the same MPI implementation. Jack On Wed, Dec 21, 2011 at 4:55 PM, Hong Zhang wrote: > Hailong: > I've never seen this type of error from MUMPS. > It seems programming bug. Are you sure smaller problem runs correctly? > Use valgrind check it. > > Hong > > > I got the error from MUMPS. > > > > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, > > nonzeros (nz) = 785860, > > I could run it. And could get result. > > But when I run it with > > nz=3112820 > > n =61240 > > > > > > I am getting the following error > > > > > > 17 - : Could not convert index 1140850688 into a > pointer > > The index may be an incorrect argument. > > Possible sources of this problem are a missing "include 'mpif.h'", > > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) > > or a misspelled user variable for an MPI object (e.g., > > com instead of comm). > > [17] [] Aborting Program! > > > > > > > > Do you know what happened? > > Is that possible it is running out of memory? > > > > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: > >> > >> Direct solvers often require large memory for storing matrix factors. > >> As Jed suggests, you may try superlu_dist. > >> > >> With mumps, I notice you use parallel analysis, which is relative new in > >> mumps. > >> What happens if you use default sequential analysis with > >> different matrix orderings? > >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. > >> > >> Also, you can increase fill ratio, > >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace > >> increase (None) > >> i.e., default ration is 20, you may try 50? (I notice that you already > use > >> 30). > >> > >> It seems you use 16 CPUs for "a mere couple thousands > >> elements" problems, and mumps "silently freezes". I do not have this > type > >> of experience with mumps. I usually can solve sparse matrix of size > >> 10k with 1 cpu using mumps. > >> When mumps runs out of memory or gets other problems, it terminates > >> execution and dumps out error message, > >> not freezes. > >> Something is wrong here. Use a debugger and figuring out where it > freezes. > >> > >> Hong > >> > >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown > wrote: > >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist > >> > > >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" > >> > wrote: > >> >> > >> >> I am successfully solving my indefinite systems with MUMPS but only > >> >> for very small problems. To give a feeling, a mere couple thousands > >> >> elements. If I only double the problem size, it silently freezes, > even > >> >> with max verbosity via the control parameters. Did anyone succeed > here > >> >> with big problems? Any recommendations for a drop-in replacement for > >> >> MUMPS? > >> >> > >> >> Thanks for any hints, > >> >> Dominik > >> >> > >> >> > >> >> > >> >> Options used: > >> >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 > >> >> > >> >> Output: > >> >> > >> >> ****** FACTORIZATION STEP ******** > >> >> > >> >> > >> >> GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > >> >> NUMBER OF WORKING PROCESSES = 16 > >> >> OUT-OF-CORE OPTION (ICNTL(22)) = 0 > >> >> REAL SPACE FOR FACTORS = 1438970073 > >> >> INTEGER SPACE FOR FACTORS = 11376442 > >> >> MAXIMUM FRONTAL SIZE (ESTIMATED) = 16868 > >> >> NUMBER OF NODES IN THE TREE = 43676 > >> >> Convergence error after scaling for ONE-NORM (option 7/8) = > 0.21D+01 > >> >> Maximum effective relaxed size of S = 231932340 > >> >> Average effective relaxed size of S = 182366303 > >> >> > >> >> REDISTRIB: TOTAL DATA LOCAL/SENT = 1509215 22859750 > >> >> GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8270 > >> >> ** Memory relaxation parameter ( ICNTL(14) ) : 35 > >> >> ** Rank of processor needing largest memory in facto : 0 > >> >> ** Space in MBYTES used by this processor for facto : 2017 > >> >> ** Avg. Space in MBYTES per working proc during facto : 1618 > > > > > > > > > > -- > > Hailong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 23 18:35:49 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 23 Dec 2011 18:35:49 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: <2B498C81-3F24-41DE-BCA3-9B1904C4707A@mcs.anl.gov> If you have PETSc ./configure do all the installs this decreases the chance of problems like this. Use --download-blacs --download-scalapack --download-mumps --download-parmetis --download-ptscotch Barry On Dec 23, 2011, at 4:56 PM, Jack Poulson wrote: > It looks like it's due to mixing different MPI implementations together (i.e., including the wrong 'mpif.h'): > http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-July/007559.html > > If I recall correctly, MUMPS only uses ScaLAPACK to factor the root separator when it is sufficiently large, and that would explain why it works for him for smaller problems. I would double check that ScaLAPACK, PETSc, and MUMPS are all compiled with the same MPI implementation. > > Jack > > On Wed, Dec 21, 2011 at 4:55 PM, Hong Zhang wrote: > Hailong: > I've never seen this type of error from MUMPS. > It seems programming bug. Are you sure smaller problem runs correctly? > Use valgrind check it. > > Hong > > > I got the error from MUMPS. > > > > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, > > nonzeros (nz) = 785860, > > I could run it. And could get result. > > But when I run it with > > nz=3112820 > > n =61240 > > > > > > I am getting the following error > > > > > > 17 - : Could not convert index 1140850688 into a pointer > > The index may be an incorrect argument. > > Possible sources of this problem are a missing "include 'mpif.h'", > > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) > > or a misspelled user variable for an MPI object (e.g., > > com instead of comm). > > [17] [] Aborting Program! > > > > > > > > Do you know what happened? > > Is that possible it is running out of memory? > > > > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: > >> > >> Direct solvers often require large memory for storing matrix factors. > >> As Jed suggests, you may try superlu_dist. > >> > >> With mumps, I notice you use parallel analysis, which is relative new in > >> mumps. > >> What happens if you use default sequential analysis with > >> different matrix orderings? > >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. > >> > >> Also, you can increase fill ratio, > >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace > >> increase (None) > >> i.e., default ration is 20, you may try 50? (I notice that you already use > >> 30). > >> > >> It seems you use 16 CPUs for "a mere couple thousands > >> elements" problems, and mumps "silently freezes". I do not have this type > >> of experience with mumps. I usually can solve sparse matrix of size > >> 10k with 1 cpu using mumps. > >> When mumps runs out of memory or gets other problems, it terminates > >> execution and dumps out error message, > >> not freezes. > >> Something is wrong here. Use a debugger and figuring out where it freezes. > >> > >> Hong > >> > >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: > >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist > >> > > >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" > >> > wrote: > >> >> > >> >> I am successfully solving my indefinite systems with MUMPS but only > >> >> for very small problems. To give a feeling, a mere couple thousands > >> >> elements. If I only double the problem size, it silently freezes, even > >> >> with max verbosity via the control parameters. Did anyone succeed here > >> >> with big problems? Any recommendations for a drop-in replacement for > >> >> MUMPS? > >> >> > >> >> Thanks for any hints, > >> >> Dominik > >> >> > >> >> > >> >> > >> >> Options used: > >> >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 > >> >> > >> >> Output: > >> >> > >> >> ****** FACTORIZATION STEP ******** > >> >> > >> >> > >> >> GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > >> >> NUMBER OF WORKING PROCESSES = 16 > >> >> OUT-OF-CORE OPTION (ICNTL(22)) = 0 > >> >> REAL SPACE FOR FACTORS = 1438970073 > >> >> INTEGER SPACE FOR FACTORS = 11376442 > >> >> MAXIMUM FRONTAL SIZE (ESTIMATED) = 16868 > >> >> NUMBER OF NODES IN THE TREE = 43676 > >> >> Convergence error after scaling for ONE-NORM (option 7/8) = 0.21D+01 > >> >> Maximum effective relaxed size of S = 231932340 > >> >> Average effective relaxed size of S = 182366303 > >> >> > >> >> REDISTRIB: TOTAL DATA LOCAL/SENT = 1509215 22859750 > >> >> GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8270 > >> >> ** Memory relaxation parameter ( ICNTL(14) ) : 35 > >> >> ** Rank of processor needing largest memory in facto : 0 > >> >> ** Space in MBYTES used by this processor for facto : 2017 > >> >> ** Avg. Space in MBYTES per working proc during facto : 1618 > > > > > > > > > > -- > > Hailong > From mmnasr at gmail.com Fri Dec 23 19:14:35 2011 From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani) Date: Fri, 23 Dec 2011 17:14:35 -0800 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: Thank you guys, What happens right now when I use GMRES is that it reaches the maximum number of iterations I set, i.e. 1000 (which should not). I have set the restart to 20 (and then increased it to 1000) but no luck. It never converges. On Fri, Dec 23, 2011 at 7:23 AM, Jed Brown wrote: > On Fri, Dec 23, 2011 at 07:43, Matthew Knepley wrote: > >> Even if you lose orthogonality, it will just stagnate since you always >> have the solution >> from the first few vectors. Likewise, I do not see how you can get an >> diverging sequence >> from restarting since you always have the last residual, you just >> stagnate. >> > > The true residual can grow even though the estimate of the preconditioned > residual decreases monotonically. When you restart, you re-evaluate the > actual residual, which might be much larger. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 23 19:18:12 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 23 Dec 2011 19:18:12 -0600 Subject: [petsc-users] KSP diverging In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 7:14 PM, Mohamad M. Nasr-Azadani wrote: > Thank you guys, > What happens right now when I use GMRES is that it reaches the maximum > number of iterations I set, i.e. 1000 (which should not). I have set the > restart to 20 (and then increased it to 1000) but no luck. It never > converges. It sounds like your operator is singular. Matt > > On Fri, Dec 23, 2011 at 7:23 AM, Jed Brown wrote: > >> On Fri, Dec 23, 2011 at 07:43, Matthew Knepley wrote: >> >>> Even if you lose orthogonality, it will just stagnate since you always >>> have the solution >>> from the first few vectors. Likewise, I do not see how you can get an >>> diverging sequence >>> from restarting since you always have the last residual, you just >>> stagnate. >>> >> >> The true residual can grow even though the estimate of the preconditioned >> residual decreases monotonically. When you restart, you re-evaluate the >> actual residual, which might be much larger. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Dec 24 07:42:54 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 24 Dec 2011 14:42:54 +0100 Subject: [petsc-users] Using HYPRE AMG for only one of the linear eqns Message-ID: <4EF5D6DE.6050502@gmail.com> Hi, I am solving 4 linear equations in my cfd code. I have declared them as: KSP ksp,ksp_semi_x,ksp_semi_y,ksp_semi_z PC pc,pc_semi_x,pc_semi_y,pc_semi_z PCType ptype KSPType ksptype However, I only want to solve the 1st eqn via hypre (thru command prompt), and the others via KSPBCGS. I now add: ksp_type richardson -pc_type hypre -pc_type_hypre boomeramg Are all the eqns now solved using hypre? How should I specify in the command prompt? -- Yours sincerely, TAY wee-beng From zonexo at gmail.com Sat Dec 24 07:56:37 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 24 Dec 2011 14:56:37 +0100 Subject: [petsc-users] Using different ksp solvers depending on the operating system Message-ID: <4EF5DA15.5040608@gmail.com> Hi, I am now using KSPBCGS and hypre to solve my Poisson eqn on the win7 and linux operating system. This is because I can't install and use hypre successfully on win7. To my knowledge, it still does not work, is that so? So is there a way to specify which solver to use depending on the OS? Btw, my code is Fortran. thanks! -- Yours sincerely, TAY wee-beng From knepley at gmail.com Sat Dec 24 08:07:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 08:07:52 -0600 Subject: [petsc-users] Using HYPRE AMG for only one of the linear eqns In-Reply-To: <4EF5D6DE.6050502@gmail.com> References: <4EF5D6DE.6050502@gmail.com> Message-ID: On Sat, Dec 24, 2011 at 7:42 AM, TAY wee-beng wrote: > Hi, > > I am solving 4 linear equations in my cfd code. I have declared them as: > > KSP ksp,ksp_semi_x,ksp_semi_y,ksp_**semi_z > > PC pc,pc_semi_x,pc_semi_y,pc_**semi_z > > PCType ptype > > KSPType ksptype > > However, I only want to solve the 1st eqn via hypre (thru command prompt), > and the others via KSPBCGS. > > I now add: > > ksp_type richardson -pc_type hypre -pc_type_hypre boomeramg > > Are all the eqns now solved using hypre? > > How should I specify in the command prompt? Are you using DMDA? If so, its easy to do PCFIELDSPLIT from the command line. Matt > > > -- > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Dec 24 08:10:56 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 08:10:56 -0600 Subject: [petsc-users] Using different ksp solvers depending on the operating system In-Reply-To: <4EF5DA15.5040608@gmail.com> References: <4EF5DA15.5040608@gmail.com> Message-ID: On Sat, Dec 24, 2011 at 7:56 AM, TAY wee-beng wrote: > Hi, > > I am now using KSPBCGS and hypre to solve my Poisson eqn on the win7 and > linux operating system. > > This is because I can't install and use hypre successfully on win7. To my > knowledge, it still does not work, is that so? > > So is there a way to specify which solver to use depending on the OS? > The best way to specify solvers is through the command line. Matt > Btw, my code is Fortran. > > thanks! > > -- > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Dec 24 08:11:55 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 24 Dec 2011 15:11:55 +0100 Subject: [petsc-users] Using HYPRE AMG for only one of the linear eqns In-Reply-To: References: <4EF5D6DE.6050502@gmail.com> Message-ID: <4EF5DDAB.3050702@gmail.com> Hi, No I'm not. Will using ptype2 or ksptype2 work? Yours sincerely, TAY wee-beng On 24/12/2011 3:07 PM, Matthew Knepley wrote: > On Sat, Dec 24, 2011 at 7:42 AM, TAY wee-beng > wrote: > > Hi, > > I am solving 4 linear equations in my cfd code. I have declared > them as: > > KSP ksp,ksp_semi_x,ksp_semi_y,ksp_semi_z > > PC pc,pc_semi_x,pc_semi_y,pc_semi_z > > PCType ptype > > KSPType ksptype > > However, I only want to solve the 1st eqn via hypre (thru command > prompt), and the others via KSPBCGS. > > I now add: > > ksp_type richardson -pc_type hypre -pc_type_hypre boomeramg > > Are all the eqns now solved using hypre? > > How should I specify in the command prompt? > > > > Are you using DMDA? If so, its easy to do PCFIELDSPLIT from the > command line. > > Matt > > > > -- > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Dec 24 08:14:40 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 08:14:40 -0600 Subject: [petsc-users] Using HYPRE AMG for only one of the linear eqns In-Reply-To: <4EF5DDAB.3050702@gmail.com> References: <4EF5D6DE.6050502@gmail.com> <4EF5DDAB.3050702@gmail.com> Message-ID: On Sat, Dec 24, 2011 at 8:11 AM, TAY wee-beng wrote: > Hi, > > No I'm not. Will using ptype2 or ksptype2 work? > I think you should setup http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFIELDSPLIT.html probably using http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFieldSplitSetIS.html#PCFieldSplitSetIS Matt > Yours sincerely, > > TAY wee-beng > > > On 24/12/2011 3:07 PM, Matthew Knepley wrote: > > On Sat, Dec 24, 2011 at 7:42 AM, TAY wee-beng wrote: > >> Hi, >> >> I am solving 4 linear equations in my cfd code. I have declared them as: >> >> KSP ksp,ksp_semi_x,ksp_semi_y,ksp_semi_z >> >> PC pc,pc_semi_x,pc_semi_y,pc_semi_z >> >> PCType ptype >> >> KSPType ksptype >> >> However, I only want to solve the 1st eqn via hypre (thru command >> prompt), and the others via KSPBCGS. >> >> I now add: >> >> ksp_type richardson -pc_type hypre -pc_type_hypre boomeramg >> >> Are all the eqns now solved using hypre? >> >> How should I specify in the command prompt? > > > > Are you using DMDA? If so, its easy to do PCFIELDSPLIT from the command > line. > > Matt > >> >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Dec 24 09:03:20 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 24 Dec 2011 09:03:20 -0600 Subject: [petsc-users] Using HYPRE AMG for only one of the linear eqns In-Reply-To: <4EF5D6DE.6050502@gmail.com> References: <4EF5D6DE.6050502@gmail.com> Message-ID: On Sat, Dec 24, 2011 at 07:42, TAY wee-beng wrote: > I am solving 4 linear equations in my cfd code. I have declared them as: > > KSP ksp,ksp_semi_x,ksp_semi_y,ksp_**semi_z > > PC pc,pc_semi_x,pc_semi_y,pc_**semi_z > > PCType ptype > > KSPType ksptype > > However, I only want to solve the 1st eqn via hypre (thru command prompt), > and the others via KSPBCGS. > > I now add: > > ksp_type richardson -pc_type hypre -pc_type_hypre boomeramg > > Are all the eqns now solved using hypre? > > How should I specify in the command prompt? > As Matt says, you can use PCFIELDSPLIT to solve the coupled system. If you really want to set up your own KSP for each system, then use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetOptionsPrefix.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Sat Dec 24 09:32:32 2011 From: john.fettig at gmail.com (John Fettig) Date: Sat, 24 Dec 2011 09:32:32 -0600 Subject: [petsc-users] Using different ksp solvers depending on the operating system In-Reply-To: <4EF5DA15.5040608@gmail.com> References: <4EF5DA15.5040608@gmail.com> Message-ID: On Sat, Dec 24, 2011 at 7:56 AM, TAY wee-beng wrote: > Hi, > > I am now using KSPBCGS and hypre to solve my Poisson eqn on the win7 and > linux operating system. > > This is because I can't install and use hypre successfully on win7. To my > knowledge, it still does not work, is that so? > I believe the hypre tarball that was downloaded by 3.1-p8 was patched to build on windows. I haven't tried the 3.2 release. You can build hypre on windows, it's just takes some hacking. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Dec 24 09:36:42 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 16:36:42 +0100 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors Message-ID: I am doing a transient calculation, allocating and deallocating several matrices and vectors both between nonlinear iterations as well as between time steps. I measure (with dstat) a constant linear memory increase as the calculation progresses. I have run my solver on a small example through valgrind, however, no leaks / unfreed memory have been reported. Are there any other possibilities to find out where the memory goes to? Many thanks for any hints. Dominik From balay at mcs.anl.gov Sat Dec 24 10:36:33 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 24 Dec 2011 10:36:33 -0600 (CST) Subject: [petsc-users] Using different ksp solvers depending on the operating system In-Reply-To: References: <4EF5DA15.5040608@gmail.com> Message-ID: On Sat, 24 Dec 2011, John Fettig wrote: > On Sat, Dec 24, 2011 at 7:56 AM, TAY wee-beng wrote: > > > > This is because I can't install and use hypre successfully on win7. To my > > knowledge, it still does not work, is that so? > > I believe the hypre tarball that was downloaded by 3.1-p8 was patched to > build on windows. I haven't tried the 3.2 release. You can build hypre on > windows, it's just takes some hacking. Well --download-hypre should work with gnu compilers. However with MS/Intel compilers - hypre should be installed separately - and sepcified to PETSc configure [with --with-hypre-include, --with-hypre-lib optons]. Neither of 3.1/3.2 support --download-hypre. The hypre tarball should have the relavent instructions for a windows install. [in the users manual] Satish From dominik at itis.ethz.ch Sat Dec 24 10:58:35 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 17:58:35 +0100 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: > Is one mesh just a refined version of the other, or is the shape different? > Factorization time should be roughly quadratic in problem size for 3D > problems. Yes, that's simply a mesh refinement. > > If the increased fill caused the system to run out of physical memory and > start swapping, then it would not be surprising for it to take days. It does not swap, it consume memory that I can handle, it just does nothing, or gets stuck somewhere. This is a toy problem, solved by PARDISO in no time. Dominik From dominik at itis.ethz.ch Sat Dec 24 11:00:16 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 18:00:16 +0100 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: <2B498C81-3F24-41DE-BCA3-9B1904C4707A@mcs.anl.gov> References: <2B498C81-3F24-41DE-BCA3-9B1904C4707A@mcs.anl.gov> Message-ID: Jack: I do not even have these packages installed anywhere on my system. Barry: That's what I did, I downloaded everything via configure. Anywhere else to look? Dominik On Sat, Dec 24, 2011 at 1:35 AM, Barry Smith wrote: > > ? If you have PETSc ./configure do all the installs this decreases the chance of problems like this. ?Use --download-blacs --download-scalapack --download-mumps --download-parmetis --download-ptscotch > > > ? Barry > > On Dec 23, 2011, at 4:56 PM, Jack Poulson wrote: > >> It looks like it's due to mixing different MPI implementations together (i.e., including the wrong 'mpif.h'): >> http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-July/007559.html >> >> If I recall correctly, MUMPS only uses ScaLAPACK to factor the root separator when it is sufficiently large, and that would explain why it works for him for smaller problems. I would double check that ScaLAPACK, PETSc, and MUMPS are all compiled with the same MPI implementation. >> >> Jack >> >> On Wed, Dec 21, 2011 at 4:55 PM, Hong Zhang wrote: >> Hailong: >> I've never seen this type of error from MUMPS. >> It seems programming bug. Are you sure smaller problem runs correctly? >> Use valgrind check it. >> >> Hong >> >> > I got the error from MUMPS. >> > >> > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, >> > nonzeros (nz) = 785860, >> > I could run it. And could get result. >> > But when I run it with >> > nz=3112820 >> > n =61240 >> > >> > >> > I am getting the following error >> > >> > >> > 17 - : Could not convert index 1140850688 into a pointer >> > The index may be an incorrect argument. >> > Possible sources of this problem are a missing "include 'mpif.h'", >> > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) >> > or a misspelled user variable for an MPI object (e.g., >> > com instead of comm). >> > [17] [] Aborting Program! >> > >> > >> > >> > Do you know what happened? >> > Is that possible it is running out of memory? >> > >> > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: >> >> >> >> Direct solvers often require large memory for storing matrix factors. >> >> As Jed suggests, you may try superlu_dist. >> >> >> >> With mumps, I notice you use parallel analysis, which is relative new in >> >> mumps. >> >> What happens if you use default sequential analysis with >> >> different matrix orderings? >> >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. >> >> >> >> Also, you can increase fill ratio, >> >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace >> >> increase (None) >> >> i.e., default ration is 20, you may try 50? (I notice that you already use >> >> 30). >> >> >> >> It seems you use 16 CPUs for "a mere couple thousands >> >> elements" problems, and mumps "silently freezes". I do not have this type >> >> of experience with mumps. I usually can solve sparse matrix of size >> >> 10k with 1 cpu using mumps. >> >> When mumps runs out of memory or gets other problems, it terminates >> >> execution and dumps out error message, >> >> not freezes. >> >> Something is wrong here. Use a debugger and figuring out where it freezes. >> >> >> >> Hong >> >> >> >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >> >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist >> >> > >> >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" >> >> > wrote: >> >> >> >> >> >> I am successfully solving my indefinite systems with MUMPS but only >> >> >> for very small problems. To give a feeling, a mere couple thousands >> >> >> elements. If I only double the problem size, it silently freezes, even >> >> >> with max verbosity via the control parameters. Did anyone succeed here >> >> >> with big problems? Any recommendations for a drop-in replacement for >> >> >> MUMPS? >> >> >> >> >> >> Thanks for any hints, >> >> >> Dominik >> >> >> >> >> >> >> >> >> >> >> >> Options used: >> >> >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >> >> >> >> >> >> Output: >> >> >> >> >> >> ****** FACTORIZATION STEP ******** >> >> >> >> >> >> >> >> >> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >> >> >> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >> >> >> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >> >> >> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >> >> >> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >> >> >> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >> >> >> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >> >> >> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >> >> >> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >> >> >> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >> >> >> >> >> >> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >> >> >> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >> >> >> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >> >> >> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >> >> >> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >> >> >> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 >> > >> > >> > >> > >> > -- >> > Hailong >> > From dominik at itis.ethz.ch Sat Dec 24 11:04:01 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 18:04:01 +0100 Subject: [petsc-users] Using different ksp solvers depending on the operating system In-Reply-To: References: <4EF5DA15.5040608@gmail.com> Message-ID: For a native build, hypre must be treated manually and passed to petsc with proper configure switches, as Satish says. It's quite tricky but can be done. On Sat, Dec 24, 2011 at 5:36 PM, Satish Balay wrote: > On Sat, 24 Dec 2011, John Fettig wrote: > >> On Sat, Dec 24, 2011 at 7:56 AM, TAY wee-beng wrote: >> > >> > This is because I can't install and use hypre successfully on win7. To my >> > knowledge, it still does not work, is that so? > >> >> I believe the hypre tarball that was downloaded by 3.1-p8 was patched to >> build on windows. ?I haven't tried the 3.2 release. ?You can build hypre on >> windows, it's just takes some hacking. > > Well --download-hypre should work with gnu compilers. > > However with MS/Intel compilers - hypre should be installed separately > - and sepcified to PETSc configure [with --with-hypre-include, > --with-hypre-lib optons]. Neither of 3.1/3.2 support --download-hypre. > > The hypre tarball should have the relavent instructions for a windows > install. [in the users manual] > > Satish > From jedbrown at mcs.anl.gov Sat Dec 24 11:08:41 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 24 Dec 2011 11:08:41 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 10:58, Dominik Szczerba wrote: > It does not swap, it consume memory that I can handle, it just does > nothing, or gets stuck somewhere. This is a toy problem, solved by > PARDISO in no time. > Time to send the MUMPS developers a test case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Dec 24 11:12:31 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 11:12:31 -0600 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 9:36 AM, Dominik Szczerba wrote: > I am doing a transient calculation, allocating and deallocating > several matrices and vectors both between nonlinear iterations as well > as between time steps. > I measure (with dstat) a constant linear memory increase as the > calculation progresses. I have run my solver on a small example > through valgrind, however, no leaks / unfreed memory have been > reported. Are there any other possibilities to find out where the > memory goes to? > > Many thanks for any hints. Its always possible that dstat is showing memory fragmentation. If the problems are the same size, you might benefit from pooling. Matt > > Dominik -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Dec 24 13:01:23 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 20:01:23 +0100 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: > Its always possible that dstat is showing memory fragmentation. If the > problems are the same size, you might benefit from pooling. > > ? ?Matt What is pooling, can you provide a pointer please? Thanks Dominik From knepley at gmail.com Sat Dec 24 13:03:58 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 13:03:58 -0600 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 1:01 PM, Dominik Szczerba wrote: > > Its always possible that dstat is showing memory fragmentation. If the > > problems are the same size, you might benefit from pooling. > > > > Matt > > What is pooling, can you provide a pointer please? > http://en.wikipedia.org/wiki/Memory_pool Its what DMDAGetGlobalVector() does. Matt > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Dec 24 13:11:10 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 20:11:10 +0100 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: > http://en.wikipedia.org/wiki/Memory_pool > > Its what DMDAGetGlobalVector() does. > > ? ?Matt Is this not only for structured grids? Dominik From knepley at gmail.com Sat Dec 24 13:11:58 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 24 Dec 2011 13:11:58 -0600 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 1:11 PM, Dominik Szczerba wrote: > > http://en.wikipedia.org/wiki/Memory_pool > > > > Its what DMDAGetGlobalVector() does. > > > > Matt > > Is this not only for structured grids? This is only to ILLUSTRATE the idea of memory pooling. Matt > > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Dec 24 13:12:05 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 24 Dec 2011 13:12:05 -0600 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 13:11, Dominik Szczerba wrote: > > http://en.wikipedia.org/wiki/Memory_pool > > > > Its what DMDAGetGlobalVector() does. > > > > Matt > > Is this not only for structured grids? DMDAGetGlobalVector() is for structured grids, but managing reuse of data structures is a very general concept. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Dec 24 14:10:10 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 21:10:10 +0100 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 8:12 PM, Jed Brown wrote: > On Sat, Dec 24, 2011 at 13:11, Dominik Szczerba > wrote: >> >> > http://en.wikipedia.org/wiki/Memory_pool >> > >> > Its what DMDAGetGlobalVector() does. >> > >> > ? ?Matt >> >> Is this not only for structured grids? > > > DMDAGetGlobalVector() is for structured grids, but managing reuse of data > structures is a very general concept. I do not understand: is there something in Petsc of this sort that I could use for my unstructured meshes or you meant I have to implement it myself? Thanks Dominik From jedbrown at mcs.anl.gov Sat Dec 24 14:13:28 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 24 Dec 2011 14:13:28 -0600 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 14:10, Dominik Szczerba wrote: > I do not understand: is there something in Petsc of this sort that I > could use for my unstructured meshes or you meant I have to implement > it myself? > If you implement DM to the extent of providing DMCreateGlobalVector(), then DMGetGlobalVector()/DMRestoreGlobalVector() will work for you. Otherwise you implement it yourself. This kind of caching is a very common pattern for data structure reuse. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Dec 24 14:15:41 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 24 Dec 2011 21:15:41 +0100 Subject: [petsc-users] memory increase in a transient calculation but no valgrind errors In-Reply-To: References: Message-ID: On Sat, Dec 24, 2011 at 9:13 PM, Jed Brown wrote: > On Sat, Dec 24, 2011 at 14:10, Dominik Szczerba > wrote: >> >> I do not understand: is there something in Petsc of this sort that I >> could use for my unstructured meshes or you meant I have to implement >> it myself? > > > If you implement DM to the extent of providing DMCreateGlobalVector(), then > DMGetGlobalVector()/DMRestoreGlobalVector() will work for you. Otherwise you > implement it yourself. This kind of caching is a very common pattern for > data structure reuse. I see. I guess it's out of reach for the moment. Latest news: at some point the linearly growing memory usage line starts to saturate. Would it not confirm your guess about fragmentation. Thanks, Dominik From dominik at itis.ethz.ch Sun Dec 25 10:17:16 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sun, 25 Dec 2011 17:17:16 +0100 Subject: [petsc-users] question about Create/Duplicate functions Message-ID: If I call a Create (or Duplicate) function on a non-Destroyed object, will be it destroyed automatically before the new allocation, or is a memory leak expected? Thanks Dominik From jedbrown at mcs.anl.gov Sun Dec 25 10:19:00 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 25 Dec 2011 10:19:00 -0600 Subject: [petsc-users] question about Create/Duplicate functions In-Reply-To: References: Message-ID: On Sun, Dec 25, 2011 at 10:17, Dominik Szczerba wrote: > If I call a Create (or Duplicate) function on a non-Destroyed object, > will be it destroyed automatically before the new allocation, or is a > memory leak expected? > What do you mean? Show the function call you are talking about? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Sun Dec 25 11:06:36 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Sun, 25 Dec 2011 11:06:36 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: <2B498C81-3F24-41DE-BCA3-9B1904C4707A@mcs.anl.gov> Message-ID: Dominik, I apologize for the confusion, but, if you read the quoted text, you will see that I was replying to Hong about a branch from this thread concerning Hailong. Barry's response was slso related to said branch. Jack On Saturday, December 24, 2011, Dominik Szczerba wrote: > Jack: I do not even have these packages installed anywhere on my system. > Barry: That's what I did, I downloaded everything via configure. > > Anywhere else to look? > > Dominik > > On Sat, Dec 24, 2011 at 1:35 AM, Barry Smith wrote: >> >> If you have PETSc ./configure do all the installs this decreases the chance of problems like this. Use --download-blacs --download-scalapack --download-mumps --download-parmetis --download-ptscotch >> >> >> Barry >> >> On Dec 23, 2011, at 4:56 PM, Jack Poulson wrote: >> >>> It looks like it's due to mixing different MPI implementations together (i.e., including the wrong 'mpif.h'): >>> http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-July/007559.html >>> >>> If I recall correctly, MUMPS only uses ScaLAPACK to factor the root separator when it is sufficiently large, and that would explain why it works for him for smaller problems. I would double check that ScaLAPACK, PETSc, and MUMPS are all compiled with the same MPI implementation. >>> >>> Jack >>> >>> On Wed, Dec 21, 2011 at 4:55 PM, Hong Zhang wrote: >>> Hailong: >>> I've never seen this type of error from MUMPS. >>> It seems programming bug. Are you sure smaller problem runs correctly? >>> Use valgrind check it. >>> >>> Hong >>> >>> > I got the error from MUMPS. >>> > >>> > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620, >>> > nonzeros (nz) = 785860, >>> > I could run it. And could get result. >>> > But when I run it with >>> > nz=3112820 >>> > n =61240 >>> > >>> > >>> > I am getting the following error >>> > >>> > >>> > 17 - : Could not convert index 1140850688 into a pointer >>> > The index may be an incorrect argument. >>> > Possible sources of this problem are a missing "include 'mpif.h'", >>> > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD) >>> > or a misspelled user variable for an MPI object (e.g., >>> > com instead of comm). >>> > [17] [] Aborting Program! >>> > >>> > >>> > >>> > Do you know what happened? >>> > Is that possible it is running out of memory? >>> > >>> > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang wrote: >>> >> >>> >> Direct solvers often require large memory for storing matrix factors. >>> >> As Jed suggests, you may try superlu_dist. >>> >> >>> >> With mumps, I notice you use parallel analysis, which is relative new in >>> >> mumps. >>> >> What happens if you use default sequential analysis with >>> >> different matrix orderings? >>> >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. >>> >> >>> >> Also, you can increase fill ratio, >>> >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace >>> >> increase (None) >>> >> i.e., default ration is 20, you may try 50? (I notice that you already use >>> >> 30). >>> >> >>> >> It seems you use 16 CPUs for "a mere couple thousands >>> >> elements" problems, and mumps "silently freezes". I do not have this type >>> >> of experience with mumps. I usually can solve sparse matrix of size >>> >> 10k with 1 cpu using mumps. >>> >> When mumps runs out of memory or gets other problems, it terminates >>> >> execution and dumps out error message, >>> >> not freezes. >>> >> Something is wrong here. Use a debugger and figuring out where it freezes. >>> >> >>> >> Hong >>> >> >>> >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >>> >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist >>> >> > >>> >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" From hzhang at mcs.anl.gov Sun Dec 25 11:17:43 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sun, 25 Dec 2011 11:17:43 -0600 Subject: [petsc-users] mumps freezes for bigger problems In-Reply-To: References: Message-ID: Hmm, this is scalapack routine. Try skip it using '-mat_mumps_icntl_13 1' Hong On Fri, Dec 23, 2011 at 4:36 PM, Dominik Szczerba wrote: > Hi Hong, > > I tried all of your suggestions, unfortunately, still get MUMPS not > moving an inch. Running in debugger and interrupting reveals: > > 0x000000000139d233 in dgemm (transa=..., transb=..., m=3621, n=3621, k=251, > ? ?alpha=-1, a=..., lda=3872, b=..., ldb=3872, beta=1, c=..., ldc=3872, > ? ?_transa=1, _transb=1) at dgemm.f:242 > 242 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? C(I,J) = C(I,J) + TEMP*A(I,L) > > So it's seems to be doing something, but on a small problem it takes a > few minutes, so I do not expect a problem of 2-3 times the size take > longer than a day... > > Regards, > Dominik > > > On Wed, Dec 21, 2011 at 4:15 PM, Hong Zhang wrote: >> Direct solvers often require large memory for storing matrix factors. >> As Jed suggests, you may try superlu_dist. >> >> With mumps, I notice you use parallel analysis, which is relative new in mumps. >> What happens if you use default sequential analysis with >> different matrix orderings? >> I usually use matrix ordering '-mat_mumps_icntl_7 2'. >> >> Also, you can increase fill ratio, >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace >> increase (None) >> i.e., default ration is 20, you may try 50? (I notice that you already use 30). >> >> It seems you use 16 CPUs for "a mere couple thousands >> elements" problems, and mumps "silently freezes". I do not have this type >> of experience with mumps. I usually can solve sparse matrix of size >> 10k with 1 cpu using mumps. >> When mumps runs out of memory or gets other problems, it terminates >> execution and dumps out error message, >> not freezes. >> Something is wrong here. Use a debugger and figuring out where it freezes. >> >> Hong >> >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown wrote: >>> -pc_type lu -pc_factor_mat_solver_package superlu_dist >>> >>> On Dec 21, 2011 6:19 AM, "Dominik Szczerba" wrote: >>>> >>>> I am successfully solving my indefinite systems with MUMPS but only >>>> for very small problems. To give a feeling, a mere couple thousands >>>> elements. If I only double the problem size, it silently freezes, even >>>> with max verbosity via the control parameters. Did anyone succeed here >>>> with big problems? Any recommendations for a drop-in replacement for >>>> MUMPS? >>>> >>>> Thanks for any hints, >>>> Dominik >>>> >>>> >>>> >>>> Options used: >>>> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 >>>> >>>> Output: >>>> >>>> ****** FACTORIZATION STEP ******** >>>> >>>> >>>> ?GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... >>>> ?NUMBER OF WORKING PROCESSES ? ? ? ? ? ? ?= ? ? ? ? ?16 >>>> ?OUT-OF-CORE OPTION (ICNTL(22)) ? ? ? ? ? = ? ? ? ? ? 0 >>>> ?REAL SPACE FOR FACTORS ? ? ? ? ? ? ? ? ? = ?1438970073 >>>> ?INTEGER SPACE FOR FACTORS ? ? ? ? ? ? ? ?= ? ?11376442 >>>> ?MAXIMUM FRONTAL SIZE (ESTIMATED) ? ? ? ? = ? ? ? 16868 >>>> ?NUMBER OF NODES IN THE TREE ? ? ? ? ? ? ?= ? ? ? 43676 >>>> ?Convergence error after scaling for ONE-NORM (option 7/8) ? = 0.21D+01 >>>> ?Maximum effective relaxed size of S ? ? ? ? ? ? ?= ? 231932340 >>>> ?Average effective relaxed size of S ? ? ? ? ? ? ?= ? 182366303 >>>> >>>> ?REDISTRIB: TOTAL DATA LOCAL/SENT ? ? ? ? = ? ? 1509215 ? ?22859750 >>>> ?GLOBAL TIME FOR MATRIX DISTRIBUTION ? ? ? = ? ? ?0.8270 >>>> ?** Memory relaxation parameter ( ICNTL(14) ?) ? ? ? ? ? ?: ? ? ? ?35 >>>> ?** Rank of processor needing largest memory in facto ? ? : ? ? ? ? 0 >>>> ?** Space in MBYTES used by this processor for facto ? ? ?: ? ? ?2017 >>>> ?** Avg. Space in MBYTES per working proc during facto ? ?: ? ? ?1618 >> From knepley at gmail.com Sun Dec 25 11:28:31 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 25 Dec 2011 11:28:31 -0600 Subject: [petsc-users] question about Create/Duplicate functions In-Reply-To: References: Message-ID: On Sun, Dec 25, 2011 at 10:17 AM, Dominik Szczerba wrote: > If I call a Create (or Duplicate) function on a non-Destroyed object, > will be it destroyed automatically before the new allocation, or is a > memory leak expected? > It will leak. Matt > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Dec 25 11:55:53 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 25 Dec 2011 11:55:53 -0600 Subject: [petsc-users] Computational Science StackExchange: another venue for questions Message-ID: Many of you may be familiar with StackOverflow and similar sites that have appeared over the last three years. Now there is a beta site devoted to computational science questions. http://scicomp.stackexchange.com Notable advantages relative to mailing lists are * automatic suggestions for related questions in which your problem may have already been addressed displayed *while you are typing your question* * better indexing by search engines * voting on questions and answers tends to reduce the time to find an authoritative answer * ability to edit questions and answers to stay relevant to later releases and to improve quality/comprehensiveness * automatic indexing of related and "linked" questions There are already some PETSc questions there and several of the developers are active on the site. I encourage you to consider asking questions of general interest there. If you ask a question involving PETSc, please use the "petsc" tag. Let us know what you think. -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 26 00:37:11 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 26 Dec 2011 10:07:11 +0330 Subject: [petsc-users] Using Inexact Newton Iterations Message-ID: Dear all, I used "-snes_ksp_ew_conv" to inexactly solve the linear system with Petsc prescribed forcing term BUT I didn't catch any difference or improvement. I just screen out the "rtol" of KSP during SNES iteration and found out It is constant (1e-5) in both exact and inexact cases. Please let me know how I should implement "inexact" method in SNES and what is the relation between KSP tolerances and inexact setup? Merry Xmas, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 26 06:03:14 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 26 Dec 2011 15:33:14 +0330 Subject: [petsc-users] Unable to use SNESSetConvergenceTest() Message-ID: Dear Developers, I need to change "rtol" manually for each KSP linear solver in each SNES iteration (some sort of inexact Newton method). I decided to use SNESSetConvergenceTest() as bellow: in main code: ierr = SNESSetConvergenceTest( snes, SNESConvTest, PETSC_NULL, PETSC_NULL ); CHKERRQ( ierr ); in convergence function: PetscErrorCode SNESConvTest( SNES snes, PetscInt it, PetscReal xn, PetscReal pn, PetscReal fn, SNESConvergedReason* res, void* ctx) { KSP ksp2; PetscReal rtol, atol; SNESGetKSP ( snes, &ksp2 ); KSPGetTolerances ( ksp2, &rtol, PETSC_NULL, PETSC_NULL, PETSC_NULL ); SNESGetTolerances ( snes, &atol, PETSC_NULL, PETSC_NULL, PETSC_NULL, PETSC_NULL ); if( fn > 1e-3 ){ rtol = 0.01; }else{ rtol = 1e-5; } KSPSetTolerances( ksp2, rtol, PETSC_NULL, PETSC_NULL, PETSC_NULL ); if( fn < atol ){ *res = SNES_CONVERGED_FNORM_ABS; return 0; }else{ *res = SNES_CONVERGED_ITERATING; } } BUT I really don't know how to control the convergence criterion for SNES. So when I run my previous code with the new convergence test I received divergence error after "first iteration": Linear solve did not converge due to DIVERGED_DTOL iterations 0 Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE How should I possibly deal with this?? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 26 07:15:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 07:15:49 -0600 Subject: [petsc-users] Using Inexact Newton Iterations In-Reply-To: References: Message-ID: On Mon, Dec 26, 2011 at 12:37 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear all, > > I used "-snes_ksp_ew_conv" to inexactly solve the linear system with Petsc > prescribed forcing term BUT I didn't catch any difference or improvement. I > just screen out the "rtol" of KSP during > SNES iteration and found out It is constant (1e-5) in both exact and > inexact cases. Please let me know how I should implement "inexact" method > in SNES and what is the relation between KSP tolerances and inexact setup? > -snes_view will tell you if it is being used, and the option is wrong http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESKSPSetUseEW.html Matt > Merry Xmas, > BehZad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 26 07:22:53 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 07:22:53 -0600 Subject: [petsc-users] Unable to use SNESSetConvergenceTest() In-Reply-To: References: Message-ID: I notice you do not have a corect return value below. Matt On Mon, Dec 26, 2011 at 6:03 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear Developers, > > I need to change "rtol" manually for each KSP linear solver in each SNES > iteration (some sort of inexact Newton method). > > I decided to use SNESSetConvergenceTest() as bellow: > > in main code: > > ierr = SNESSetConvergenceTest( snes, SNESConvTest, PETSC_NULL, PETSC_NULL > ); CHKERRQ( ierr ); > > in convergence function: > > PetscErrorCode SNESConvTest( SNES snes, PetscInt it, PetscReal xn, > PetscReal pn, PetscReal fn, SNESConvergedReason* res, void* ctx) > { > KSP ksp2; > PetscReal rtol, atol; > SNESGetKSP ( snes, &ksp2 ); > KSPGetTolerances ( ksp2, &rtol, PETSC_NULL, PETSC_NULL, PETSC_NULL ); > SNESGetTolerances ( snes, &atol, PETSC_NULL, PETSC_NULL, PETSC_NULL, > PETSC_NULL ); > if( fn > 1e-3 ){ > rtol = 0.01; > }else{ > rtol = 1e-5; > } > KSPSetTolerances( ksp2, rtol, PETSC_NULL, PETSC_NULL, PETSC_NULL ); > if( fn < atol ){ > *res = SNES_CONVERGED_FNORM_ABS; > return 0; > }else{ > *res = SNES_CONVERGED_ITERATING; > } > } > > BUT I really don't know how to control the convergence criterion for SNES. > So when I run my previous code with the new convergence test I received > divergence error after "first iteration": > > Linear solve did not converge due to DIVERGED_DTOL iterations 0 > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE > > How should I possibly deal with this?? > > Thanks a lot, > BehZad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 26 07:35:31 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 26 Dec 2011 17:05:31 +0330 Subject: [petsc-users] Unable to use SNESSetConvergenceTest() In-Reply-To: References: Message-ID: I want to check the condition: fn < atol, else, continuous SNES iteration until reach the maximum iterations. So, How should I set the return value? Thanks a lot -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 26 07:44:19 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 07:44:19 -0600 Subject: [petsc-users] Unable to use SNESSetConvergenceTest() In-Reply-To: References: Message-ID: On Mon, Dec 26, 2011 at 7:35 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > I want to check the condition: fn < atol, else, continuous SNES iteration > until reach the maximum iterations. So, How should I set the return value? > PetscFunctionReturn(0); http://petsc.cs.iit.edu/petsc/releases/petsc-3.2/file/6e64ab10774d/src/snes/interface/snesut.c#l389 > Thanks a lot -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Dec 26 07:48:11 2011 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 26 Dec 2011 17:18:11 +0330 Subject: [petsc-users] Unable to use SNESSetConvergenceTest() In-Reply-To: References: Message-ID: Thanks, I will work on it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Dec 26 08:14:08 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 26 Dec 2011 15:14:08 +0100 Subject: [petsc-users] symmetric matrix Message-ID: <4EF88130.4050107@gfz-potsdam.de> Hello all, I have two questions: 1. Is that possible to set MatSetOption(A,MAT_SPD,PETSC_TRUE); through command line? 2. If I set option above for MPIAIJ matrix do solvers ignore lower triangular matrix? Regards, Alexander From knepley at gmail.com Mon Dec 26 08:25:35 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 08:25:35 -0600 Subject: [petsc-users] symmetric matrix In-Reply-To: <4EF88130.4050107@gfz-potsdam.de> References: <4EF88130.4050107@gfz-potsdam.de> Message-ID: On Mon, Dec 26, 2011 at 8:14 AM, Alexander Grayver wrote: > Hello all, > > I have two questions: > > 1. Is that possible to set MatSetOption(A,MAT_SPD,PETSC_**TRUE); through > command line? > No. > 2. If I set option above for MPIAIJ matrix do solvers ignore lower > triangular matrix? > No, they only do this for SBAIJ matrices. This is just a flag that can be checked. Matt > Regards, > Alexander > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 26 08:34:37 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 26 Dec 2011 15:34:37 +0100 Subject: [petsc-users] question about Create/Duplicate functions In-Reply-To: References: Message-ID: On Sun, Dec 25, 2011 at 5:19 PM, Jed Brown wrote: > On Sun, Dec 25, 2011 at 10:17, Dominik Szczerba > wrote: >> >> If I call a Create (or Duplicate) function on a non-Destroyed object, >> will be it destroyed automatically before the new allocation, or is a >> memory leak expected? > > > What do you mean? Show the function call you are talking about? As simple as: Vec x VecCreate... VecCreate... exit application I would expect a leak to occur, as Matt confirmed, but was surprised that valgrind was not reporting any issues with this code, and that's why I asked my question. With valgrind option --leak-check=full (option not listed in FAQ) lost memory is reported in the first call to VecCreate, but no issue is reported due to the lack of VecDestroy. Does Petsc finalize function do some type of garbage collection? Thanks Dominik From knepley at gmail.com Mon Dec 26 08:49:17 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 08:49:17 -0600 Subject: [petsc-users] question about Create/Duplicate functions In-Reply-To: References: Message-ID: On Mon, Dec 26, 2011 at 8:34 AM, Dominik Szczerba wrote: > On Sun, Dec 25, 2011 at 5:19 PM, Jed Brown wrote: > > On Sun, Dec 25, 2011 at 10:17, Dominik Szczerba > > wrote: > >> > >> If I call a Create (or Duplicate) function on a non-Destroyed object, > >> will be it destroyed automatically before the new allocation, or is a > >> memory leak expected? > > > > > > What do you mean? Show the function call you are talking about? > > As simple as: > > Vec x > VecCreate... > VecCreate... > exit application > > I would expect a leak to occur, as Matt confirmed, but was surprised > that valgrind was not reporting any issues with this code, and that's > why I asked my question. > With valgrind option --leak-check=full (option not listed in FAQ) lost > memory is reported in the first call to VecCreate, but no issue is > reported due to the lack of VecDestroy. Does Petsc finalize function > do some type of garbage collection? > No, you will see it using -malloc_dump. Valgrind should show you overwriting the pointer. Matt > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Dec 26 09:00:49 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 26 Dec 2011 16:00:49 +0100 Subject: [petsc-users] valgrind reports with --leak-check=full Message-ID: I am getting the following reports running with --leak-check=full (which could be mentioned somewhere in FAQ). Is this something to worry about? Thanks Dominik ==10052== 300 (60 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 14 of 15 ==10052== at 0x4C28F9F: malloc (vg_replace_malloc.c:236) ==10052== by 0x67AB855: nss_parse_service_list (nsswitch.c:626) ==10052== by 0x67ABE39: __nss_database_lookup (nsswitch.c:167) ==10052== by 0x74B6823: ??? ==10052== by 0x67648AC: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) ==10052== by 0x67641A2: getpwuid (getXXbyYY.c:117) ==10052== by 0x99BE9D: PetscGetUserName (fuser.c:66) ==10052== by 0x5D2A20: PetscErrorPrintfInitialize (errtrace.c:68) ==10052== by 0x5F1057: PetscInitialize (pinit.c:697) ==10051== 8 bytes in 1 blocks are definitely lost in loss record 2 of 15 ==10051== at 0x4C28F9F: malloc (vg_replace_malloc.c:236) ==10051== by 0x1296EB9: GKmalloc__ (util.c:151) ==10051== by 0x1296E00: fmalloc__ (util.c:112) ==10051== by 0x1295CBC: CheckInputs__ (weird.c:188) ==10051== by 0x126F8B8: ParMETIS_V3_PartKway (kmetis.c:56) ==10051== by 0xD26874: MatPartitioningApply_Parmetis (pmetis.c:96) ==10051== by 0x6DDCFC: MatPartitioningApply (partition.c:226) From dominik at itis.ethz.ch Mon Dec 26 09:09:16 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 26 Dec 2011 16:09:16 +0100 Subject: [petsc-users] PetscMallocSpace() and PetscGetResidentSetSize() are absent Message-ID: >From the FAQ: "Use the commands PetscMallocSpace() and PetscGetResidentSetSize() to monitor memory allocated and total memory used as the code progresses." These two functions, however, are missing entries in http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/singleindex.html Where is their documentation? Thanks Dominik From knepley at gmail.com Mon Dec 26 09:35:36 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 09:35:36 -0600 Subject: [petsc-users] valgrind reports with --leak-check=full In-Reply-To: References: Message-ID: On Mon, Dec 26, 2011 at 9:00 AM, Dominik Szczerba wrote: > I am getting the following reports running with --leak-check=full > (which could be mentioned somewhere in FAQ). Is this something to > worry about? > The first one is just OS stuff. The second could be a leak, but its only 8 bytes, and seems independent of the problem size. I will check it out. Matt > Thanks > Dominik > > > ==10052== 300 (60 direct, 240 indirect) bytes in 1 blocks are > definitely lost in loss record 14 of 15 > ==10052== at 0x4C28F9F: malloc (vg_replace_malloc.c:236) > ==10052== by 0x67AB855: nss_parse_service_list (nsswitch.c:626) > ==10052== by 0x67ABE39: __nss_database_lookup (nsswitch.c:167) > ==10052== by 0x74B6823: ??? > ==10052== by 0x67648AC: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) > ==10052== by 0x67641A2: getpwuid (getXXbyYY.c:117) > ==10052== by 0x99BE9D: PetscGetUserName (fuser.c:66) > ==10052== by 0x5D2A20: PetscErrorPrintfInitialize (errtrace.c:68) > ==10052== by 0x5F1057: PetscInitialize (pinit.c:697) > > ==10051== 8 bytes in 1 blocks are definitely lost in loss record 2 of 15 > ==10051== at 0x4C28F9F: malloc (vg_replace_malloc.c:236) > ==10051== by 0x1296EB9: GKmalloc__ (util.c:151) > ==10051== by 0x1296E00: fmalloc__ (util.c:112) > ==10051== by 0x1295CBC: CheckInputs__ (weird.c:188) > ==10051== by 0x126F8B8: ParMETIS_V3_PartKway (kmetis.c:56) > ==10051== by 0xD26874: MatPartitioningApply_Parmetis (pmetis.c:96) > ==10051== by 0x6DDCFC: MatPartitioningApply (partition.c:226) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 26 09:37:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Dec 2011 09:37:50 -0600 Subject: [petsc-users] PetscMallocSpace() and PetscGetResidentSetSize() are absent In-Reply-To: References: Message-ID: On Mon, Dec 26, 2011 at 9:09 AM, Dominik Szczerba wrote: > >From the FAQ: > > "Use the commands PetscMallocSpace() and PetscGetResidentSetSize() to > monitor memory allocated and total memory used as the code > progresses." > You want http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscMemoryGetCurrentUsage.html http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscMallocGetCurrentUsage.html I will fix the FAQ Matt > These two functions, however, are missing entries in > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/singleindex.html > Where is their documentation? > > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Tue Dec 27 00:32:02 2011 From: wolfshow at gmail.com (Fatcharm) Date: Tue, 27 Dec 2011 14:32:02 +0800 Subject: [petsc-users] Question about petsc-3.2-p5/src/ts/examples/tutorials/ex15.c Message-ID: Dear all, I am working with an example of "petsc-3.2-p5/src/ts/examples/tutorials/ex15.c" I noticed that in ex15.c, Jtype == 0 means that /* use user provided Jacobian evaluation routine */ Jtype == 1 is for /* slow finite difference J; */ and Jtype == 2 is /* Use coloring to compute finite difference J efficiently */ I found a problem, when I run ex15 with mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 0 -drawcontours or mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 2 -drawcontours , it works well. However, when I run ex15 with mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 1 -drawcontours the solution remains the same as the initial condition. If I reduced the da_grid_x and da_grid_y, the command mpiexec -n 8 ./ex15 -da_grid_x 51 -da_grid_y 51 -Jtype 1 -drawcontours works well. The manual tells me that "Although SNESDefaultComputeJacobian() is not recommended for general use in large-scale applications, It can be useful in checking the correctness of a user-provided Jacobian." I am confused that if SNESDefaultComputeJacobian() can not give the correct Jacobian when using "mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 1 -drawcontours", then how can it be used to check the user-provided Jacobian? Is there somebody encountering the same problem or give me some advice? Thank you very much! Feng-Chao Wang From hzhang at mcs.anl.gov Tue Dec 27 10:16:10 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 27 Dec 2011 10:16:10 -0600 Subject: [petsc-users] Question about petsc-3.2-p5/src/ts/examples/tutorials/ex15.c In-Reply-To: References: Message-ID: Feng-Chao, Jtype == 1 is for /* slow finite difference J; */ computes dense Jacobian one column at a time, taking prohibitively long time for large matrices. For 101*101 matrix, i.e., it evaluates 1.e+4 columns (evaluate function 1.e+4 times). Running it on my Mac, the execution terminates after a hang with error timestep 0: time 0, solution norm 391.65, max 1, min 0 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: ! [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE! With -da_grid_x 51 -da_grid_y 51, it runs, but is extremely slow. As stated in the petsc manual, SNESDefaultComputeJacobian() is not recommended for general use in large-scale applications. It only works on tiny matrices for checking the correctness of a user-provided Jacobian. For ex15.c, it obviously only works for up to 51x51 grids on the machine we tested. Hong On Tue, Dec 27, 2011 at 12:32 AM, Fatcharm wrote: > Dear all, > > I am working with an example of "petsc-3.2-p5/src/ts/examples/tutorials/ex15.c" > > I noticed that in ex15.c, > > Jtype == 0 means ?that /* use user provided Jacobian evaluation routine */ > Jtype == 1 is for /* slow finite difference J; */ > and Jtype == 2 is /* Use coloring to compute ?finite difference J efficiently */ > > I found a problem, when I run ex15 with > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 0 -drawcontours > > or > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 2 -drawcontours > > , it works well. > > However, when I run ex15 with > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 1 -drawcontours > > the solution remains the same as the initial condition. > > If I reduced the da_grid_x and da_grid_y, the command > > mpiexec -n 8 ./ex15 -da_grid_x 51 -da_grid_y 51 -Jtype 1 -drawcontours > > works well. > > The manual tells me that "Although SNESDefaultComputeJacobian() is not > recommended for general use in large-scale applications, It can be > useful in checking the correctness of a user-provided Jacobian." > > I am confused that if SNESDefaultComputeJacobian() can not give the > correct Jacobian when using "mpiexec -n 8 ./ex15 -da_grid_x 101 > -da_grid_y 101 -Jtype 1 -drawcontours", then how can it be used to > check the user-provided Jacobian? > > Is there somebody encountering the same problem or give me some advice? > > Thank you very much! > > Feng-Chao Wang From wolfshow at gmail.com Tue Dec 27 18:07:06 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Wed, 28 Dec 2011 08:07:06 +0800 Subject: [petsc-users] Question about petsc-3.2-p5/src/ts/examples/tutorials/ex15.c References: Message-ID: <201112280807035267878@gmail.com> Dear Hong, Thank you very much for your patient explanations. I got that. Thank you! Feng-Chao 2011-12-28 Feng-Chao Wang Message: 2 Date: Tue, 27 Dec 2011 10:16:10 -0600 From: Hong Zhang Subject: Re: [petsc-users] Question about petsc-3.2-p5/src/ts/examples/tutorials/ex15.c To: PETSc users list Message-ID: Content-Type: text/plain; charset=ISO-8859-1 Feng-Chao, Jtype == 1 is for /* slow finite difference J; */ computes dense Jacobian one column at a time, taking prohibitively long time for large matrices. For 101*101 matrix, i.e., it evaluates 1.e+4 columns (evaluate function 1.e+4 times). Running it on my Mac, the execution terminates after a hang with error timestep 0: time 0, solution norm 391.65, max 1, min 0 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: ! [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE! With -da_grid_x 51 -da_grid_y 51, it runs, but is extremely slow. As stated in the petsc manual, SNESDefaultComputeJacobian() is not recommended for general use in large-scale applications. It only works on tiny matrices for checking the correctness of a user-provided Jacobian. For ex15.c, it obviously only works for up to 51x51 grids on the machine we tested. Hong On Tue, Dec 27, 2011 at 12:32 AM, Fatcharm wrote: > Dear all, > > I am working with an example of "petsc-3.2-p5/src/ts/examples/tutorials/ex15.c" > > I noticed that in ex15.c, > > Jtype == 0 means ?that /* use user provided Jacobian evaluation routine */ > Jtype == 1 is for /* slow finite difference J; */ > and Jtype == 2 is /* Use coloring to compute ?finite difference J efficiently */ > > I found a problem, when I run ex15 with > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 0 -drawcontours > > or > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 2 -drawcontours > > , it works well. > > However, when I run ex15 with > > mpiexec -n 8 ./ex15 -da_grid_x 101 -da_grid_y 101 -Jtype 1 -drawcontours > > the solution remains the same as the initial condition. > > If I reduced the da_grid_x and da_grid_y, the command > > mpiexec -n 8 ./ex15 -da_grid_x 51 -da_grid_y 51 -Jtype 1 -drawcontours > > works well. > > The manual tells me that "Although SNESDefaultComputeJacobian() is not > recommended for general use in large-scale applications, It can be > useful in checking the correctness of a user-provided Jacobian." > > I am confused that if SNESDefaultComputeJacobian() can not give the > correct Jacobian when using "mpiexec -n 8 ./ex15 -da_grid_x 101 > -da_grid_y 101 -Jtype 1 -drawcontours", then how can it be used to > check the user-provided Jacobian? > > Is there somebody encountering the same problem or give me some advice? > > Thank you very much! > > Feng-Chao Wang ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 36, Issue 80 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Tue Dec 27 19:58:38 2011 From: wolfshow at gmail.com (Feng-Chao Wang) Date: Wed, 28 Dec 2011 09:58:38 +0800 Subject: [petsc-users] Strongly nonlinear equation solved within the framework of PETSc Message-ID: <201112280958344175801@gmail.com> Dear all, I want to numerically solve a strongly nonlinear fourth-order equation (2-D), which is used to describe the dynamics of a liquid film. please find the form of the equation below or in the attachment, The thickness of the film, H(X,Y,T) is the function to be solved, C, G, A0 are constant parameters. I wrote a PETSc programs for this problem, using CN method in time. However, it does not work well. In some cases, there are some negativevalues in the solution (Obviously, the thickness of the liquid film could not be negative.) While in some other cases, the solution remains as the initial condition, looking like the governing equation doesn't work at all. I think it is because that the solution of this equation may form infinite fradient when the equation is developed. Some literatures reported that the similar problem was solved successfully using the ADI (Alternating Direction Implicit) method. Unfortunately, I found that the current PETSc doesn't support ADI. I also read a paper in which the similar equation was solved by using the CN scheme in time. Besides, the author employed the modified second-order upwind difference method to handle the nonlinear terms due to the inability of the centered differences in space. I prefer to use PETSc because this powerful toolkit enables easy parallel computation of PDE since I am not familar with MPI. Could anyone please give me some suggestions on how I can solve the above equation within the framework of PETSc? Thanks very much in advance! Feng-Chao Wang 2011-12-28 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 52349 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: equation.jpg Type: image/jpeg Size: 24483 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Tue Dec 27 22:01:15 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 27 Dec 2011 22:01:15 -0600 Subject: [petsc-users] Strongly nonlinear equation solved within the framework of PETSc In-Reply-To: <201112280958344175801@gmail.com> References: <201112280958344175801@gmail.com> Message-ID: On Tue, Dec 27, 2011 at 19:58, Feng-Chao Wang wrote: > ** > Dear all, > > I want to numerically solve a strongly nonlinear fourth-order equation > (2-D), which is used to describe the dynamics of a liquid film. > please find the form of the equation below or in the attachment, The > thickness of the film, H(X,Y,T) is the function to be solved, C, G, A0 are > constant parameters. > > > I wrote a PETSc programs for this problem, using CN method in time. > However, it does not work well. In some cases, there are some > negativevalues in the solution (Obviously, the thickness of the liquid film > could not be negative.) While in some other cases, the solution remains as > the initial condition, looking like the governing equation doesn't work at > all. > 1. Make sure that the equations are actually being solved, for instance with -snes_monitor -snes_converged_reason -ksp_converged_reason. 2. Start with a time integration method that has a discrete maximum principle. Implicit Euler has this property for many classes of systems, but CN does not. 3. Consider writing the system as a variational inequality if positivity is still a problem with the time steps you want to take. Then use SNESVIRS (or SNESVIRSAUG or SNESVISS). > > I think it is because that the solution of this equation may form infinite > fradient when the equation is developed. > > Some literatures reported that the similar problem was solved successfully > using the ADI (Alternating Direction Implicit) method. Unfortunately, I > found that the current PETSc doesn't support ADI. > > I also read a paper in which the similar equation was solved by using the > CN scheme in time. Besides, the author employed the modified second-order > upwind difference method to handle the nonlinear terms due to the inability > of the centered differences in space. > > I prefer to use PETSc because this powerful toolkit enables easy parallel > computation of PDE since I am not familar with MPI. Could anyone please > give me some suggestions on how I can solve the above equation within the > framework of PETSc? > > Thanks very much in advance! > > Feng-Chao Wang > > 2011-12-28 > ------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 52349 bytes Desc: not available URL: From irving at naml.us Tue Dec 27 22:16:09 2011 From: irving at naml.us (Geoffrey Irving) Date: Tue, 27 Dec 2011 19:16:09 -0900 Subject: [petsc-users] invalid object classid 0 between PETSc and SLEPc Message-ID: Hello, I've installed PETSc 3.2 and SLEPc 3.2 on Mac OS 10.7 through MacPorts 2.0.3. PETSc works fine by itself, but SLEPc fails on the initial EPSCreate call. The error follows. This is presumably a MacPorts misconfiguration issue, but thought I'd check here first to see if anyone has seen this before in a similar situation. Only static libraries exist, so as far as I know the suggested explanation doesn't apply. The conf directory is at http://naml.us/~irving/random/petsc-conf.tar.gz if it helps. Thanks, Geoffrey [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Invalid object classid 0 This often happens if you compile with PETSC_USE_DYNAMIC_LIBRARIES, but link with static libraries.! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving Tue Dec 27 19:07:59 2011 [0]PETSC ERROR: Libraries linked from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib [0]PETSC ERROR: Configure run at Mon Nov 21 14:00:28 2011 [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python --with-debugging=0 --with-c-support=1 --with-c++-support=1 --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 --PETSC_ARCH=darwin --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc --with-cc=/opt/local/bin/openmpicc --with-cxx=/opt/local/bin/openmpicxx --with-mpiexec=/opt/local/bin/openmpiexec --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscClassRegLogGetClass() line 282 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.2-p5/src/sys/plog/utils/classLog.c [0]PETSC ERROR: PetscLogObjCreateDefault() line 310 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.2-p5/src/sys/plog/utils/classLog.c [0]PETSC ERROR: EPSCreate() line 436 in src/eps/interface/basic.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving Tue Dec 27 19:07:59 2011 [0]PETSC ERROR: Libraries linked from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib [0]PETSC ERROR: Configure run at Mon Nov 21 14:00:28 2011 [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python --with-debugging=0 --with-c-support=1 --with-c++-support=1 --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 --PETSC_ARCH=darwin --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc --with-cc=/opt/local/bin/openmpicc --with-cxx=/opt/local/bin/openmpicxx --with-mpiexec=/opt/local/bin/openmpiexec --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: sparse_symmetric_spectrum() line 55 in petsc/slepc.cpp From jroman at dsic.upv.es Wed Dec 28 04:02:05 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 28 Dec 2011 11:02:05 +0100 Subject: [petsc-users] invalid object classid 0 between PETSc and SLEPc In-Reply-To: References: Message-ID: I will forward this to the MacPorts maintainer. Jose El 28/12/2011, a las 05:16, Geoffrey Irving escribi?: > Hello, > > I've installed PETSc 3.2 and SLEPc 3.2 on Mac OS 10.7 through MacPorts > 2.0.3. PETSc works fine by itself, but SLEPc fails on the initial > EPSCreate call. The error follows. > > This is presumably a MacPorts misconfiguration issue, but thought I'd > check here first to see if anyone has seen this before in a similar > situation. Only static libraries exist, so as far as I know the > suggested explanation doesn't apply. The conf directory is at > http://naml.us/~irving/random/petsc-conf.tar.gz if it helps. > > Thanks, > Geoffrey > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Invalid object classid 0 > This often happens if you compile with PETSC_USE_DYNAMIC_LIBRARIES, > but link with static libraries.! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 > 13:45:54 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving > Tue Dec 27 19:07:59 2011 > [0]PETSC ERROR: Libraries linked from > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib > [0]PETSC ERROR: Configure run at Mon Nov 21 14:00:28 2011 > [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python > --with-debugging=0 --with-c-support=1 --with-c++-support=1 > --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 > --PETSC_ARCH=darwin > --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc > --with-cc=/opt/local/bin/openmpicc > --with-cxx=/opt/local/bin/openmpicxx > --with-mpiexec=/opt/local/bin/openmpiexec > --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscClassRegLogGetClass() line 282 in > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.2-p5/src/sys/plog/utils/classLog.c > [0]PETSC ERROR: PetscLogObjCreateDefault() line 310 in > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.2-p5/src/sys/plog/utils/classLog.c > [0]PETSC ERROR: EPSCreate() line 436 in src/eps/interface/basic.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: ! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 > 13:45:54 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving > Tue Dec 27 19:07:59 2011 > [0]PETSC ERROR: Libraries linked from > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib > [0]PETSC ERROR: Configure run at Mon Nov 21 14:00:28 2011 > [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python > --with-debugging=0 --with-c-support=1 --with-c++-support=1 > --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 > --PETSC_ARCH=darwin > --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc > --with-cc=/opt/local/bin/openmpicc > --with-cxx=/opt/local/bin/openmpicxx > --with-mpiexec=/opt/local/bin/openmpiexec > --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: sparse_symmetric_spectrum() line 55 in petsc/slepc.cpp From dalcinl at gmail.com Wed Dec 28 07:51:55 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 28 Dec 2011 10:51:55 -0300 Subject: [petsc-users] invalid object classid 0 between PETSc and SLEPc In-Reply-To: References: Message-ID: On 28 December 2011 01:16, Geoffrey Irving wrote: > Hello, > > I've installed PETSc 3.2 and SLEPc 3.2 on Mac OS 10.7 through MacPorts > 2.0.3. ?PETSc works fine by itself, but SLEPc fails on the initial > EPSCreate call. ?The error follows. > > This is presumably a MacPorts misconfiguration issue, but thought I'd > check here first to see if anyone has seen this before in a similar > situation. ?Only static libraries exist, so as far as I know the > suggested explanation doesn't apply. ?The conf directory is at > http://naml.us/~irving/random/petsc-conf.tar.gz if it helps. > Just in case, please run: $ otool -L ./your_executable Do you have any other PETSc/SLEPc build lying around? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From irving at naml.us Wed Dec 28 14:48:37 2011 From: irving at naml.us (Geoffrey Irving) Date: Wed, 28 Dec 2011 11:48:37 -0900 Subject: [petsc-users] invalid object classid 0 between PETSc and SLEPc In-Reply-To: References: Message-ID: On Wed, Dec 28, 2011 at 4:51 AM, Lisandro Dalcin wrote: > On 28 December 2011 01:16, Geoffrey Irving wrote: >> Hello, >> >> I've installed PETSc 3.2 and SLEPc 3.2 on Mac OS 10.7 through MacPorts >> 2.0.3. ?PETSc works fine by itself, but SLEPc fails on the initial >> EPSCreate call. ?The error follows. >> >> This is presumably a MacPorts misconfiguration issue, but thought I'd >> check here first to see if anyone has seen this before in a similar >> situation. ?Only static libraries exist, so as far as I know the >> suggested explanation doesn't apply. ?The conf directory is at >> http://naml.us/~irving/random/petsc-conf.tar.gz if it helps. >> > > Just in case, please run: > > $ otool -L ./your_executable Seems clean. tile:petsc% otool -L ../install/flavor/lib/libother_petsc.* ../install/flavor/lib/libother_petsc.dylib: /Users/irving/otherlab/other/install/debug/lib/libother_petsc.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/local/lib/OpenMesh/libOpenMeshCore.2.0.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/local/lib/OpenMesh/libOpenMeshTools.2.0.dylib (compatibility version 2.0.0, current version 2.0.0) /opt/local/lib/libmpi_cxx.1.dylib (compatibility version 2.0.0, current version 2.1.0) /opt/local/lib/libmpi.1.dylib (compatibility version 2.0.0, current version 2.2.0) /opt/local/lib/libboost_iostreams.dylib (compatibility version 0.0.0, current version 0.0.0) /opt/local/lib/libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0) /opt/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0) /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.6) /opt/local/lib/libX11.6.dylib (compatibility version 10.0.0, current version 10.0.0) /Users/irving/otherlab/other/install/debug/lib/libother_core.dylib (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0) /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Python (compatibility version 2.6.0, current version 2.6.0) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) ../install/flavor/lib/libother_petsc.so: /Users/irving/otherlab/other/install/debug/lib/libother_petsc.dylib (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0) /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Python (compatibility version 2.6.0, current version 2.6.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) I forgot to mention that I'm linking both petsc and slepc into an .dylib and then into an .so which is then dynamically loaded as a python module. Since both petsc and slepc are linked as static libraries into the .dylib but not the .so, there shouldn't be any issues with shared library boundaries, but I could be wrong about this. The link commands are g++ -o build/nocona/release/petsc/libother_petsc.dylib -install_name /Users/irving/otherlab/other/install/release/lib/libother_petsc.dylib -dead_strip -fvisibility=hidden -dynamiclib build/nocona/release/petsc/ksp.os build/nocona/release/petsc/module.os build/nocona/release/petsc/slepc.os build/nocona/release/petsc/util.os -L/opt/local/lib/petsc/lib -L/usr/local/lib/OpenMesh -L/opt/local/lib -Linstall/release/lib -lOpenMeshCore -lOpenMeshTools -lmpi_cxx -lmpi -lslepc -lboost_iostreams -lboost_filesystem -lboost_system -lz -lbz2 -lpetsc -lX11 -lother_core -F/opt/local/Library/Frameworks -framework Accelerate -framework Python gcc -o build/nocona/release/petsc/libother_petsc.so -dead_strip -fvisibility=hidden -bundle -L/opt/local/lib/petsc/lib -L/usr/local/lib/OpenMesh -L/opt/local/lib -Linstall/release/lib install/release/lib/libother_petsc.dylib -F/opt/local/Library/Frameworks -framework Accelerate -framework Python > Do you have any other PETSc/SLEPc build lying around? I do, but they didn't show up in my environment variables. Geoffrey From bsmith at mcs.anl.gov Wed Dec 28 14:54:39 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 28 Dec 2011 14:54:39 -0600 Subject: [petsc-users] invalid object classid 0 between PETSc and SLEPc In-Reply-To: References: Message-ID: <6F27BF7E-84DE-48F4-B6F2-1E8485C0B399@mcs.anl.gov> On Dec 28, 2011, at 2:48 PM, Geoffrey Irving wrote: > On Wed, Dec 28, 2011 at 4:51 AM, Lisandro Dalcin wrote: >> On 28 December 2011 01:16, Geoffrey Irving wrote: >>> Hello, >>> >>> I've installed PETSc 3.2 and SLEPc 3.2 on Mac OS 10.7 through MacPorts >>> 2.0.3. PETSc works fine by itself, but SLEPc fails on the initial >>> EPSCreate call. The error follows. >>> >>> This is presumably a MacPorts misconfiguration issue, but thought I'd >>> check here first to see if anyone has seen this before in a similar >>> situation. Only static libraries exist, so as far as I know the >>> suggested explanation doesn't apply. The conf directory is at >>> http://naml.us/~irving/random/petsc-conf.tar.gz if it helps. >>> >> >> Just in case, please run: >> >> $ otool -L ./your_executable > > Seems clean. > > tile:petsc% otool -L ../install/flavor/lib/libother_petsc.* > ../install/flavor/lib/libother_petsc.dylib: > /Users/irving/otherlab/other/install/debug/lib/libother_petsc.dylib > (compatibility version 0.0.0, current version 0.0.0) > /usr/local/lib/OpenMesh/libOpenMeshCore.2.0.dylib (compatibility > version 2.0.0, current version 2.0.0) > /usr/local/lib/OpenMesh/libOpenMeshTools.2.0.dylib (compatibility > version 2.0.0, current version 2.0.0) > /opt/local/lib/libmpi_cxx.1.dylib (compatibility version 2.0.0, > current version 2.1.0) > /opt/local/lib/libmpi.1.dylib (compatibility version 2.0.0, current > version 2.2.0) > /opt/local/lib/libboost_iostreams.dylib (compatibility version 0.0.0, > current version 0.0.0) > /opt/local/lib/libboost_filesystem.dylib (compatibility version > 0.0.0, current version 0.0.0) > /opt/local/lib/libboost_system.dylib (compatibility version 0.0.0, > current version 0.0.0) > /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current > version 1.2.5) > /opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current > version 1.0.6) > /opt/local/lib/libX11.6.dylib (compatibility version 10.0.0, current > version 10.0.0) > /Users/irving/otherlab/other/install/debug/lib/libother_core.dylib > (compatibility version 0.0.0, current version 0.0.0) > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate > (compatibility version 1.0.0, current version 4.0.0) > /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Python > (compatibility version 2.6.0, current version 2.6.0) > /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current > version 52.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 159.1.0) > ../install/flavor/lib/libother_petsc.so: > /Users/irving/otherlab/other/install/debug/lib/libother_petsc.dylib > (compatibility version 0.0.0, current version 0.0.0) > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate > (compatibility version 1.0.0, current version 4.0.0) > /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Python > (compatibility version 2.6.0, current version 2.6.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 159.1.0) > > I forgot to mention that I'm linking both petsc and slepc into an > .dylib and then into an .so which is then dynamically loaded as a > python module. Since both petsc and slepc are linked as static > libraries into the .dylib but not the .so, there shouldn't be any > issues with shared library boundaries, but I could be wrong about > this. The symptom you reported is indicative in this case of having two copies of the libraries "global variables". PETSc uses global variables for profile information and to mange the registration of classes etc. It could be that the way you generate the file .so results in two sets of the variables, one gets set properly in PetscInitialize() but the other one gets used later (like in SLEPc code) and since it is not set it generates an error. I've found that on the Mac it is not difficult to accidentally this problem when building up shared libraries. Barry > The link commands are > > g++ -o build/nocona/release/petsc/libother_petsc.dylib -install_name > /Users/irving/otherlab/other/install/release/lib/libother_petsc.dylib > -dead_strip -fvisibility=hidden -dynamiclib > build/nocona/release/petsc/ksp.os build/nocona/release/petsc/module.os > build/nocona/release/petsc/slepc.os build/nocona/release/petsc/util.os > -L/opt/local/lib/petsc/lib -L/usr/local/lib/OpenMesh -L/opt/local/lib > -Linstall/release/lib -lOpenMeshCore -lOpenMeshTools -lmpi_cxx -lmpi > -lslepc -lboost_iostreams -lboost_filesystem -lboost_system -lz -lbz2 > -lpetsc -lX11 -lother_core -F/opt/local/Library/Frameworks -framework > Accelerate -framework Python > gcc -o build/nocona/release/petsc/libother_petsc.so -dead_strip > -fvisibility=hidden -bundle -L/opt/local/lib/petsc/lib > -L/usr/local/lib/OpenMesh -L/opt/local/lib -Linstall/release/lib > install/release/lib/libother_petsc.dylib > -F/opt/local/Library/Frameworks -framework Accelerate -framework > Python > >> Do you have any other PETSc/SLEPc build lying around? > > I do, but they didn't show up in my environment variables. > > Geoffrey From praghanmor at gmail.com Thu Dec 29 01:27:52 2011 From: praghanmor at gmail.com (Rahul Praghanmor) Date: Thu, 29 Dec 2011 12:57:52 +0530 Subject: [petsc-users] Regarding global indexing Message-ID: Dear Sir, Thanks to make a such efficient library to handle the complex mathematics. I am Rahul and working on unstructured finite volume solver. The solver is able to solve both single zone and multi zone unstructured mesh using CSR Gauss Seidel. I successfully implement PETSc to solve on a single zone unstructured mesh i.e. serial implementation has done and tested. Now I want to implement a PETSc on a multi-zone mesh.The mesh partitioning in multi-zone is done using a separate tool, Metis. The multi-zone mesh is read in solver.I don't understand how to do the global indexing for multi-zone mesh to form a global matrix required for PETSc. Your valuable suggestion and guidance is cordially appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 29 07:08:42 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 29 Dec 2011 07:08:42 -0600 Subject: [petsc-users] Regarding global indexing In-Reply-To: References: Message-ID: On Thu, Dec 29, 2011 at 01:27, Rahul Praghanmor wrote: > I successfully implement PETSc to solve on a single zone unstructured mesh > i.e. serial implementation has done and tested. > Now I want to implement a PETSc on a multi-zone mesh.The mesh partitioning > in multi-zone is done using a separate tool, Metis. The multi-zone mesh is > read in solver.I don't understand how to do the global indexing for > multi-zone mesh to form a global matrix required for PETSc. > Start with a non-overlapping partition (usually what is provided by Metis). Distribute the mesh according to this partition, then compute the local size and use MPI_Scan() to compute the offset of your piece in the global ordering. You can learn the global index of ghosted points by sending it from owner to ghoster through the overlap. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfshow at gmail.com Thu Dec 29 08:10:07 2011 From: wolfshow at gmail.com (Fatcharm) Date: Thu, 29 Dec 2011 22:10:07 +0800 Subject: [petsc-users] Strongly nonlinear equation solved within the framework of PETSc Message-ID: 2011/12/28, petsc-users-request at mcs.anl.gov : > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Re: Strongly nonlinear equation solved within the framework > of PETSc (Jed Brown) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 27 Dec 2011 22:01:15 -0600 > From: Jed Brown > Subject: Re: [petsc-users] Strongly nonlinear equation solved within > the framework of PETSc > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Tue, Dec 27, 2011 at 19:58, Feng-Chao Wang wrote: > >> ** >> Dear all, >> >> I want to numerically solve a strongly nonlinear fourth-order equation >> (2-D), which is used to describe the dynamics of a liquid film. >> please find the form of the equation below or in the attachment, The >> thickness of the film, H(X,Y,T) is the function to be solved, C, G, A0 >> are >> constant parameters. >> >> >> I wrote a PETSc programs for this problem, using CN method in time. >> However, it does not work well. In some cases, there are some >> negativevalues in the solution (Obviously, the thickness of the liquid >> film >> could not be negative.) While in some other cases, the solution remains as >> the initial condition, looking like the governing equation doesn't work at >> all. >> > > 1. Make sure that the equations are actually being solved, for instance > with -snes_monitor -snes_converged_reason -ksp_converged_reason. > Dear Jed, Follow your suggestions, I examined my problem. For the cases that the equations are not being solved, I got these message: timestep 0: time 0, solution norm 10.201, max 0.00195106, min 4.89435e-05 0 SNES Function norm 2.772465906990e+07 Linear solve did not converge due to DIVERGED_DTOL iterations 3270 Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE timestep 1: time 0.0001, solution norm 10.201, max 0.00195106, min 4.89435e-05 0 SNES Function norm 2.772465906990e+07 Linear solve did not converge due to DIVERGED_DTOL iterations 3270 Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE timestep 2: time 0.0002, solution norm 10.201, max 0.00195106, min 4.89435e-05 0 SNES Function norm 2.772465906990e+07 Linear solve did not converge due to DIVERGED_DTOL iterations 3270 Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE We can see that the SNES Function norm is extremely large. I think it is because that the initial value for the unknown function H(X,T) is quite small and there is some terms like (1/H)(dH/dX) or (1/H^2)(dH/dX) in my equations. For "Linear solve did not converge due to DIVERGED_DTOL iterations 3270", does this mean I should change the ksp_type? Thank you very much! Feng-Chao > 2. Start with a time integration method that has a discrete maximum > principle. Implicit Euler has this property for many classes of systems, > but CN does not. > > 3. Consider writing the system as a variational inequality if positivity is > still a problem with the time steps you want to take. Then use SNESVIRS (or > SNESVIRSAUG or SNESVISS). > > >> >> I think it is because that the solution of this equation may form infinite >> fradient when the equation is developed. >> >> Some literatures reported that the similar problem was solved successfully >> using the ADI (Alternating Direction Implicit) method. Unfortunately, I >> found that the current PETSc doesn't support ADI. >> >> I also read a paper in which the similar equation was solved by using the >> CN scheme in time. Besides, the author employed the modified second-order >> upwind difference method to handle the nonlinear terms due to the >> inability >> of the centered differences in space. >> >> I prefer to use PETSc because this powerful toolkit enables easy parallel >> computation of PDE since I am not familar with MPI. Could anyone please >> give me some suggestions on how I can solve the above equation within the >> framework of PETSc? >> >> Thanks very much in advance! >> >> Feng-Chao Wang >> >> 2011-12-28 >> ------------------------------ >> >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/jpeg > Size: 52349 bytes > Desc: not available > URL: > > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 36, Issue 82 > ******************************************* > From jedbrown at mcs.anl.gov Thu Dec 29 08:21:08 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 29 Dec 2011 08:21:08 -0600 Subject: [petsc-users] Strongly nonlinear equation solved within the framework of PETSc In-Reply-To: References: Message-ID: On Thu, Dec 29, 2011 at 08:10, Fatcharm wrote: > We can see that the SNES Function norm is extremely large. I think it > is because that the initial value for the unknown function H(X,T) is > quite small and there is some terms like (1/H)(dH/dX) or > (1/H^2)(dH/dX) in my equations. > That may cause ill-conditioning, but you could still scale the equations so that the initial norm was of order 1. It shouldn't matter here though, because most methods are unaffected by scaling. Are you computing an analytic Jacobian or using finite differencing? > > For "Linear solve did not converge due to DIVERGED_DTOL iterations > 3270", does this mean I should change the ksp_type? > It's important to solve the linear systems before worrying about convergence rates for Newton methods. Try a direct solve on a small problem first, then read this http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging If you fix the linear solve issues, but SNES is still not converging, read http://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 29 13:02:12 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Dec 2011 13:02:12 -0600 Subject: [petsc-users] Regarding global indexing In-Reply-To: References: Message-ID: On Dec 29, 2011, at 7:08 AM, Jed Brown wrote: > On Thu, Dec 29, 2011 at 01:27, Rahul Praghanmor wrote: > I successfully implement PETSc to solve on a single zone unstructured mesh i.e. serial implementation has done and tested. > Now I want to implement a PETSc on a multi-zone mesh.The mesh partitioning in multi-zone is done using a separate tool, Metis. The multi-zone mesh is read in solver.I don't understand how to do the global indexing for multi-zone mesh to form a global matrix required for PETSc. > > Start with a non-overlapping partition (usually what is provided by Metis). Distribute the mesh according to this partition, then compute the local size and use MPI_Scan() to compute the offset of your piece in the global ordering. You can learn the global index of ghosted points by sending it from owner to ghoster through the overlap. You can optionally use the PetscLayout object (I've included the manual page for PetscLayoutCreate() below since I just improved it) to avoid having to use MPI_Scan() directly. You just set the local sizes based on your new number of grid points per MPI process and use PetscLayoutSetUp() and then you can query the global starting point (rstart) etc of each process using the other routines mentioned in the manual page. PetscLayoutCreate - Allocates PetscLayout space and sets the map contents to the default. Collective on MPI_Comm Input Parameters: + comm - the MPI communicator - map - pointer to the map Level: developer Notes: Typical calling sequence PetscLayoutCreate(MPI_Comm,PetscLayout *); PetscLayoutSetBlockSize(PetscLayout,1); PetscLayoutSetSize(PetscLayout,n) or PetscLayoutSetLocalSize(PetscLayout,N); PetscLayoutSetUp(PetscLayout); Optionally use any of the following: PetscLayoutGetSize(PetscLayout,PetscInt *); or PetscLayoutGetLocalSize(PetscLayout,PetscInt *;) PetscLayoutGetRange(PetscLayout,PetscInt *rstart,PetscInt *rend); or PetscLayoutGetRanges(PetscLayout,const PetscInt *range[]) PetscLayoutDestroy(PetscLayout); The PetscLayout object and methods are intended to be used in the PETSc Vec and Mat implementions; it is often not needed in user codes unless you really gain something in their use. Fortran Notes: Not available from Fortran .seealso: PetscLayoutSetLocalSize(), PetscLayoutSetSize(), PetscLayoutGetSize(), PetscLayoutGetLocalSize(), PetscLayout, PetscLayoutDestroy(), PetscLayoutGetRange(), PetscLayoutGetRanges(), PetscLayoutSetBlockSize(), PetscLayoutGetBlockSize(), PetscLayoutSetUp() From bsmith at mcs.anl.gov Thu Dec 29 13:06:10 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Dec 2011 13:06:10 -0600 Subject: [petsc-users] Strongly nonlinear equation solved within the framework of PETSc In-Reply-To: References: Message-ID: <1E858447-63AA-4C9F-8498-B532AC1A42EF@mcs.anl.gov> On Dec 29, 2011, at 8:21 AM, Jed Brown wrote: > On Thu, Dec 29, 2011 at 08:10, Fatcharm wrote: > We can see that the SNES Function norm is extremely large. I think it > is because that the initial value for the unknown function H(X,T) is > quite small and there is some terms like (1/H)(dH/dX) or > (1/H^2)(dH/dX) in my equations. > > That may cause ill-conditioning, but you could still scale the equations so that the initial norm was of order 1. It shouldn't matter here though, because most methods are unaffected by scaling. > > Are you computing an analytic Jacobian or using finite differencing? > > > For "Linear solve did not converge due to DIVERGED_DTOL iterations > 3270", does this mean I should change the ksp_type? > > It's important to solve the linear systems before worrying about convergence rates for Newton methods. Try a direct solve on a small problem first, This is done by running with the additional option -pc_type lu > then read this > > http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging > > If you fix the linear solve issues, but SNES is still not converging, read > > http://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging From junchao.zhang at gmail.com Thu Dec 29 13:28:49 2011 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 29 Dec 2011 13:28:49 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? Message-ID: Hello, I create a MPIAIJ matrix. How can I know the # of nonzeros stored locally? Furthermore, in case it is needed, how can I get access to the Compress Sparse Row(CSR) data structure of the local matrix, i.e., the row pointer array, column index array, value array? Thank you! -- Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 29 13:32:24 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 29 Dec 2011 13:32:24 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? In-Reply-To: References: Message-ID: On Thu, Dec 29, 2011 at 1:28 PM, Junchao Zhang wrote: > Hello, > I create a MPIAIJ matrix. How can I know the # of nonzeros stored > locally? > Furthermore, in case it is needed, how can I get access to the Compress > Sparse Row(CSR) data structure of the local matrix, i.e., the row pointer > array, column index array, value array? > We do nto recommend it, but you can use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetColumnIJ.html Matt > > Thank you! > -- Junchao Zhang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 29 13:34:57 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 29 Dec 2011 13:34:57 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? In-Reply-To: References: Message-ID: On Thu, Dec 29, 2011 at 13:32, Matthew Knepley wrote: > We do nto recommend it, but you can use > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetColumnIJ.html > Or, more likely, http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetRowIJ.html But the matrices are not actually stored this way, so calling this function involves a copy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 29 13:37:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Dec 2011 13:37:27 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? In-Reply-To: References: Message-ID: <6CC147C3-F161-42AB-85D7-3BEBD738713C@mcs.anl.gov> On Dec 29, 2011, at 1:28 PM, Junchao Zhang wrote: > Hello, > I create a MPIAIJ matrix. How can I know the # of nonzeros stored locally? MatGetInfo() > Furthermore, in case it is needed, how can I get access to the Compress Sparse Row(CSR) data structure of the local matrix, i.e., the row pointer array, column index array, value array? PETSc MPIAIJ matrices are not actually stored as a single CSR matrix on each process. They are actually stored as two CSR matrices: the "block diagonal local portion" and the "off-processor portion" (which is complicated by remapping the column indices). Thus it is basically impossible for normal PETSc users to muck directly with the matrix entries. If you are interested in developing new matrix entry specific solvers that work directly with MPIAIJ matrices then you need to include the private include files that define the data structures (you can find them in src/mat/impls/aij in subdirectories) I warn you that you need to be pretty good at C programming the muck with this stuff. Barry > > Thank you! > -- Junchao Zhang From junchao.zhang at gmail.com Thu Dec 29 13:42:50 2011 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 29 Dec 2011 13:42:50 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? In-Reply-To: References: Message-ID: Manual of MatGetRowIJ says "Returns the compressed row storage i and j indices for *sequential *matrices." Could I pass MPIAIJ matrix as an argument to it? On Thu, Dec 29, 2011 at 1:34 PM, Jed Brown wrote: > On Thu, Dec 29, 2011 at 13:32, Matthew Knepley wrote: > >> We do nto recommend it, but you can use >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetColumnIJ.html >> > > Or, more likely, > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetRowIJ.html > > > But the matrices are not actually stored this way, so calling this > function involves a copy. > -- Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Dec 29 14:10:24 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 29 Dec 2011 14:10:24 -0600 Subject: [petsc-users] How to get number of local nonzeros of a matrix? In-Reply-To: References: Message-ID: On Thu, Dec 29, 2011 at 13:42, Junchao Zhang wrote: > Manual of MatGetRowIJ says "Returns the compressed row storage i and j > indices for *sequential *matrices." > Could I pass MPIAIJ matrix as an argument to it? > No, but you can use MatGetRow() to access the rows one at a time. MPIAIJ matrices are not stored in a way where getting a single pointer makes sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 08:39:09 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 15:39:09 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! Message-ID: Hi, I am getting the below error with fgmres. I was wondering how to start debugging the problem. All the values in the matrix and rhs seem fine. Any suggestions are very welcome. Best regards, Dominik ||b|| = 3.102831e-05 Starting KSPSolve... 0 KSP Residual norm 2.071556923444e-05 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Infinite or not-a-number generated in norm! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/dsz/build/framework-debug/sandbox/cscs/bin/FluidSolver on a linux-gnu named tharsis by dsz Fri Dec 30 15:31:20 2011 [0]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.2-p5/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Wed Nov 23 13:57:33 2011 [0]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.2-p5 PETSC_ARCH=linux-gnu-c-debug --with-debugging=1 --with-x=0 --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --download-parmetis=1 --download-scalapack --download-blacs --download-mumps --with-hdf5 --with-hdf5-dir=/usr [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecNorm() line 167 in /home/dsz/pack/petsc-3.2-p5/src/vec/vec/interface/rvector.c [0]PETSC ERROR: FGMREScycle() line 132 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve_FGMRES() line 299 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve() line 423 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: ApplyPreconditioner() line 3721 in "unknowndirectory/"/home/dsz/src/framework/sandbox/dsz/cscs/solve/FluidSolver.cxx [0]PETSC ERROR: PCApply_Shell() line 121 in /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/impls/shell/shellpc.c [0]PETSC ERROR: PCApply() line 383 in /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: FGMREScycle() line 174 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve_FGMRES() line 299 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve() line 423 in /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/interface/itfunc.c From jedbrown at mcs.anl.gov Fri Dec 30 08:52:21 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 08:52:21 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 08:39, Dominik Szczerba wrote: > I am getting the below error with fgmres. I was wondering how to start > debugging the problem. All the values in the matrix and rhs seem fine. > Any suggestions are very welcome. > I suspect your custom preconditioner is computing NaN for the call to the inner solver. Run with -fp_trap -start_in_debugger to see where NaN was first generated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 10:34:29 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 17:34:29 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: > I suspect your custom preconditioner is computing NaN for the call to the > inner solver. Run with -fp_trap -start_in_debugger to see where NaN was > first generated. Very useful hint that with -fp_trap. I get trapped at: Program received signal SIGFPE, Arithmetic exception.0x00000000010fad4e in hypre_BoomerAMGRelax (A=0x1b56e9a0, f=0x195487b0,?? ? cf_marker=0x692b900, relax_type=6, relax_points=1, relax_weight=1,?? ? omega=1, l1_norms=0x0, u=0xb0f8aa0, Vtemp=0x18fd1900, Ztemp=0x0)? ? at par_relax.c:19251925 ? ? ?u_data[i] = res / A_diag_data[A_diag_i[i]]; with res = -2.4683099805590231e+303 and A_diag_data[A_diag_i[i]] = 9.287739686222747e-06. The full backtrace goes through many boomeramg lines to point back at the end to my code where (in a custom preconditioner) I solve a linear system with KSPGMRES and PCHYPRE/boomeramg. I guess I will proceed now with inspecting the values by hand, but if there still are some useful Petsc tricks I am eager to hear. From jedbrown at mcs.anl.gov Fri Dec 30 10:44:31 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 10:44:31 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 10:34, Dominik Szczerba wrote: > Very useful hint that with -fp_trap. I get trapped at: > > Program received signal SIGFPE, Arithmetic > exception.0x00000000010fad4e in hypre_BoomerAMGRelax (A=0x1b56e9a0, > f=0x195487b0, cf_marker=0x692b900, relax_type=6, relax_points=1, > relax_weight=1, omega=1, l1_norms=0x0, u=0xb0f8aa0, > Vtemp=0x18fd1900, Ztemp=0x0) at par_relax.c:19251925 > u_data[i] = res / A_diag_data[A_diag_i[i]]; > with res = -2.4683099805590231e+303 and A_diag_data[A_diag_i[i]] = > 9.287739686222747e-06. The full backtrace goes through many boomeramg > lines to point back at the end to my code where (in a custom > preconditioner) I solve a linear system with KSPGMRES and > PCHYPRE/boomeramg. I guess I will proceed now with inspecting the > values by hand, but if there still are some useful Petsc tricks I am > eager to hear. > Is your system singular? Or perhaps the BoomerAMG smoother is not stable for your problem. You didn't show the full stack. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 10:59:42 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 17:59:42 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: > Is your system singular? Or perhaps the BoomerAMG smoother is not stable for > your problem. You didn't show the full stack. Below a full stack. Yes, I am a bit malicious, solving for steady state where such is known not to exists (Karman vortex street past a cylinder). The very same problem (mesh, constraints, etc.) does not crash when solving with time stepping. However, I am not completely sure if this is the reason of the crash, because I am expecting to never be able to converge, but not to crash. Thanks Dominik Program received signal SIGFPE, Arithmetic exception. 0x00000000010fad4e in hypre_BoomerAMGRelax (A=0x1a1109a0, f=0x180ea7b0, cf_marker=0x54cd900, relax_type=6, relax_points=1, relax_weight=1, omega=1, l1_norms=0x0, u=0x9c9aaa0, Vtemp=0x17b73900, Ztemp=0x0) at par_relax.c:1925 1925 u_data[i] = res / A_diag_data[A_diag_i[i]]; (gdb) bt #0 0x00000000010fad4e in hypre_BoomerAMGRelax (A=0x1a1109a0, f=0x180ea7b0, cf_marker=0x54cd900, relax_type=6, relax_points=1, relax_weight=1, omega=1, l1_norms=0x0, u=0x9c9aaa0, Vtemp=0x17b73900, Ztemp=0x0) at par_relax.c:1925 #1 0x00000000010fff7a in hypre_BoomerAMGRelaxIF (A=0x1a1109a0, f=0x180ea7b0, cf_marker=0x54cd900, relax_type=6, relax_order=1, cycle_type=1, relax_weight=1, omega=1, l1_norms=0x0, u=0x9c9aaa0, Vtemp=0x17b73900, Ztemp=0x0) at par_relax_interface.c:59 #2 0x00000000010c0d38 in hypre_BoomerAMGCycle (amg_vdata=0x9930b60, F_array= 0x9cd8880, U_array=0x10f01640) at par_cycle.c:486 #3 0x00000000010acc47 in hypre_BoomerAMGSolve (amg_vdata=0x9930b60, A= 0x221dfa0, f=0x9d85be0, u=0x9a32710) at par_amg_solve.c:252 #4 0x000000000109ea0f in HYPRE_BoomerAMGSolve (solver=0x9930b60, A=0x221dfa0, b=0x9d85be0, x=0x9a32710) at HYPRE_parcsr_amg.c:76 #5 0x00000000007ef30b in PCApply_HYPRE (pc=0x10d8ff90, b=0x9cbed40, x=0x9cb5420) at /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/impls/hypre/hypre.c:159 #6 0x000000000081628c in PCApply (pc=0x10d8ff90, x=0x9cbed40, y=0x9cb5420) at /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/interface/precon.c:383 #7 0x0000000000953cdb in KSPSolve_PREONLY (ksp=0x9aa05d0) at /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/preonly/preonly.c:26 #8 0x00000000008e25a6 in KSPSolve (ksp=0x9aa05d0, b=0x9cbed40, x=0x9cb5420) at /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/interface/itfunc.c:423 ---Type to continue, or q to quit--- #9 0x0000000000504dfb in FluidSolver::ApplyPreconditioner (prec=0x10f013f0, x=0x10f12260, y=0x10f1bb40) at /home/dsz/src/framework/sandbox/dsz/cscs/solve/FluidSolver.cxx:3712 #10 0x000000000087c121 in PCApply_Shell (pc=0x10f013f0, x=0x10f12260, y=0x10f1bb40) at /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/impls/shell/shellpc.c:121 #11 0x000000000081628c in PCApply (pc=0x10f013f0, x=0x10f12260, y=0x10f1bb40) at /home/dsz/pack/petsc-3.2-p5/src/ksp/pc/interface/precon.c:383 #12 0x00000000009604d7 in FGMREScycle (itcount=0x7fff03d02cd4, ksp=0x10eff220) at /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:174 #13 0x00000000009611c0 in KSPSolve_FGMRES (ksp=0x10eff220) at /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:299 #14 0x00000000008e25a6 in KSPSolve (ksp=0x10eff220, b=0x9c9a750, x=0x17a31310) at /home/dsz/pack/petsc-3.2-p5/src/ksp/ksp/interface/itfunc.c:423 #15 0x00000000004ef53f in FluidSolver::Solve (this=0x1da0000) at /home/dsz/src/framework/sandbox/dsz/cscs/solve/FluidSolver.cxx:1764 #16 0x000000000053dc2b in main (argc=4, argv=0x7fff03d05888) at /home/dsz/src/framework/sandbox/dsz/cscs/solve/FluidSolverMain.cxx:316 From jedbrown at mcs.anl.gov Fri Dec 30 11:02:47 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 11:02:47 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 10:59, Dominik Szczerba wrote: > Yes, I am a bit malicious, solving for steady state where such is > known not to exists (Karman vortex street past a cylinder). The very > same problem (mesh, constraints, etc.) does not crash when solving > with time stepping. However, I am not completely sure if this is the > reason of the crash, because I am expecting to never be able to > converge, but not to crash. > You could very well be producing a singular linear system (with large unknown null space). I'm not sure why you want to do this. There are methods to solve for periodic behavior and you could choose a turbulence model in order to produce a steady state. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 11:25:10 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 18:25:10 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: > You could very well be producing a singular linear system (with large > unknown null space). I'm not sure why you want to do this. There are methods > to solve for periodic behavior and you could choose a turbulence model in > order to produce a steady state. Yes, I want to get a time averaged steady state. I am able to do this with another (pressure correction) solver, was just comparing. And you gave me a very good hint: I do not have a turbulence model, but I have streamline diffusion, which if increased, makes the crash go away, which indeed seams to indicate a singular system. Thanks, Dominik From jedbrown at mcs.anl.gov Fri Dec 30 11:38:02 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 11:38:02 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 11:25, Dominik Szczerba wrote: > Yes, I want to get a time averaged steady state. I am able to do this > with another (pressure correction) solver, was just comparing. > What makes you think that the point where some nonlinear solve *stagnates* has anything to do with correct time-averaged statistics? And you gave me a very good hint: I do not have a turbulence model, > but I have streamline diffusion, which if increased, makes the crash > go away, which indeed seams to indicate a singular system. > You should use a turbulence model. Even a zero-equation model like Smagorinsky would be better than simple streamline diffusion. What is your verification and validation process? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 11:49:12 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 18:49:12 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: > What makes you think that the point where some nonlinear solve stagnates has> anything to do with correct time-averaged statistics? > What is your verification and validation process? Strictly mathematically you are probably right. Intuitively physically, I do expect - and in fact get - some representative solution. I validated only qualitatively, looking at streamlines. I do get the expected vortices at the expected locations by eye looking very similar to the ones in published papers. > You should use a turbulence model. Even a zero-equation model like > Smagorinsky would be better than simple streamline diffusion. You are right, but I will not be able to implement it in the nearest future. And the focus is transient von Karman problem, steady state was just a side jump to see what it would do. Streamline diffusion just proved your guess right about the singular system. Still I am not happy it crashes instead of diverging. Thanks Dominik From jedbrown at mcs.anl.gov Fri Dec 30 11:58:40 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 11:58:40 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 11:49, Dominik Szczerba wrote: > You are right, but I will not be able to implement it in the nearest > future. > Smagorinsky is very simple and at least slightly justified. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 12:03:11 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 19:03:11 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: > Smagorinsky is very simple and at least slightly justified. Thanks for the hint, I will look into it. Regards, Dominik From bsmith at mcs.anl.gov Fri Dec 30 16:11:16 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 30 Dec 2011 16:11:16 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: Message-ID: <238BB94B-704C-4F87-BD2F-3F49C2A11DAA@mcs.anl.gov> I have added to Petsc-dev VecValidValues() that will check that input and output vector values for many commonly used routines such as PCApply() have only valid entries to simplify tracking down these kinds of problems. Barry On Dec 30, 2011, at 8:52 AM, Jed Brown wrote: > On Fri, Dec 30, 2011 at 08:39, Dominik Szczerba wrote: > I am getting the below error with fgmres. I was wondering how to start > debugging the problem. All the values in the matrix and rhs seem fine. > Any suggestions are very welcome. > > I suspect your custom preconditioner is computing NaN for the call to the inner solver. Run with -fp_trap -start_in_debugger to see where NaN was first generated. From junchao.zhang at gmail.com Fri Dec 30 16:30:30 2011 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 30 Dec 2011 16:30:30 -0600 Subject: [petsc-users] Flops pf MatMult is less than expected Message-ID: Hello, I have a sparse matrix A. Its size is 494 x 494. It has 1666 nonzeros. I call MatMult(A, x, y). From the profiling result (got by -log_summary), I find MatMult's flops is 2838. I think the flops should be nnz x 2 = 1666 x 2 = 3332. I don't know how PETSc got this number (2838). I also observed the phenomena with other matrices. It seems PETSc always give smaller flops. Why? Thank you! -- Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Dec 30 16:35:53 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 30 Dec 2011 16:35:53 -0600 Subject: [petsc-users] Flops pf MatMult is less than expected In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 4:30 PM, Junchao Zhang wrote: > Hello, > I have a sparse matrix A. Its size is 494 x 494. It has 1666 nonzeros. > I call MatMult(A, x, y). From the profiling result (got by > -log_summary), I find MatMult's flops is 2838. > I think the flops should be nnz x 2 = 1666 x 2 = 3332. I don't know how > PETSc got this number (2838). > I also observed the phenomena with other matrices. It seems PETSc always > give smaller flops. Why? > There is 1 fewer adds than multiplies in every row: 3332-494 = 2838 Matt > > Thank you! > -- Junchao Zhang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Dec 30 16:38:33 2011 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 30 Dec 2011 16:38:33 -0600 Subject: [petsc-users] Flops pf MatMult is less than expected In-Reply-To: References: Message-ID: Because 2838 + 494 = 3332, I believe PETSc calculates flops of an inner product as n-1, where n is the vector length. On Fri, Dec 30, 2011 at 4:30 PM, Junchao Zhang wrote: > Hello, > I have a sparse matrix A. Its size is 494 x 494. It has 1666 nonzeros. > I call MatMult(A, x, y). From the profiling result (got by > -log_summary), I find MatMult's flops is 2838. > I think the flops should be nnz x 2 = 1666 x 2 = 3332. I don't know how > PETSc got this number (2838). > I also observed the phenomena with other matrices. It seems PETSc always > give smaller flops. Why? > > Thank you! > -- Junchao Zhang > -- Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Dec 30 16:43:48 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 30 Dec 2011 23:43:48 +0100 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: <238BB94B-704C-4F87-BD2F-3F49C2A11DAA@mcs.anl.gov> References: <238BB94B-704C-4F87-BD2F-3F49C2A11DAA@mcs.anl.gov> Message-ID: Sounds very useful, will check it out once the next patch is out, thanks! Dominik On Fri, Dec 30, 2011 at 11:11 PM, Barry Smith wrote: > > ? I have added to Petsc-dev VecValidValues() that will check that input and output vector values for many commonly used routines such as PCApply() have only valid entries to simplify tracking down these kinds of problems. > > ? ?Barry > > On Dec 30, 2011, at 8:52 AM, Jed Brown wrote: > >> On Fri, Dec 30, 2011 at 08:39, Dominik Szczerba wrote: >> I am getting the below error with fgmres. I was wondering how to start >> debugging the problem. All the values in the matrix and rhs seem fine. >> Any suggestions are very welcome. >> >> I suspect your custom preconditioner is computing NaN for the call to the inner solver. Run with -fp_trap -start_in_debugger to see where NaN was first generated. > From jedbrown at mcs.anl.gov Fri Dec 30 16:45:52 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Dec 2011 16:45:52 -0600 Subject: [petsc-users] Infinite or not-a-number generated in norm! In-Reply-To: References: <238BB94B-704C-4F87-BD2F-3F49C2A11DAA@mcs.anl.gov> Message-ID: On Fri, Dec 30, 2011 at 16:43, Dominik Szczerba wrote: > Sounds very useful, will check it out once the next patch is out, thanks! It's not going to be ported to petsc-3.2. Use petsc-dev if you want this. (There aren't many changes yet, so it's easy to migrate from petsc-3.2 to petsc-dev: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html) -------------- next part -------------- An HTML attachment was scrubbed... URL: