From lb2653 at columbia.edu  Sun Mar  1 00:28:38 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Sun, 01 Mar 2015 01:28:38 -0500
Subject: [petsc-users] PetscLogStage functions not working in fortran
Message-ID: <54F2B196.5070100@columbi.edu>

Hi all,
I am trying to fine gain a little my program profiling but cannot get 
PetscLogStageRegiset, PetscLogStagePush and PetscLogStagePop to work in 
my Fortran code.
Here is my fortran subroutine:

       subroutine SetPetscStage(StageNum)

       implicit  none

#     include  "finclude/petsc.h"

       integer        StageNum
       character      StageName*5
       PetscErrorCode ierr
       PetscInt       Stage

       if(StageNum.eq.1) then
          Stage = 1
          PetscLogStageRegister("Jacobian and residual assembly",
      &        Stage,ierr)
          PetscLogStagePush(Stage,ierr)
       elseif(StageNum.eq.11) then
          PetscLogStagePop()
       endif

       end

I get the following compiler error when I try to make my program:

SetPetscStage.F:14.9:

          PetscLogStageRegister("Jacobian and residual assembly",
          1
Error: Unclassifiable statement at (1)
SetPetscStage.F:16.9:

PetscLogStagePush(Stage,ierr)
          1
Error: Unclassifiable statement at (1)
SetPetscStage.F:18.9:

PetscLogStagePop()
          1
Error: Unclassifiable statement at (1)

Any idea on what is be happening?

-- 
Best,
Luc



From hus003 at ucsd.edu  Sun Mar  1 00:30:35 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sun, 1 Mar 2015 06:30:35 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you Barry. I have yet two more questions:

1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector? 

2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial? 

By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA. 

Best, 
Hui

________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Saturday, February 28, 2015 9:35 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>
> Is it true that
> 1) Only DMDA can use mg?

   No this is not true

> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?

   No you do not have to

> We cannot create a matrix and add it to KSP if we want to use mg?

    Yes you can.

   There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.

More details

> 1) Only DMDA can use mg?

   Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).

> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?

   Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).

> We cannot create a matrix and add it to KSP if we want to use mg?

    As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).

   If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg

  Barry

>
> Best,
> Hui
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Friday, February 27, 2015 5:11 PM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>
>  Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>
>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>
>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>
>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>
>  No it handles dof > 1 fine.
>
>  Send your code.
>
>  Barry
>
>>
>> Thank you!

From knepley at gmail.com  Sun Mar  1 08:02:00 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Mar 2015 08:02:00 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <CAMYG4GnotCRxB1J0DAMnT03CF883pDSG70TLpq=X24KEMtzCYg@mail.gmail.com>

On Sun, Mar 1, 2015 at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:

> Thank you Barry. I have yet two more questions:
>
> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS
> to set up matrices and rhs, and I use geometric mg, what if I want to
> change my rhs many times? Should I write many KSPSetComputeRHS, and
> register them with ksp? Or is there a simple way to just register the rhs
> with ksp as a vector?
>

If you just give a vector, you could use -pc_mg_galerkin to have it
projected. Same with the matrix.


> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and
> whose rows are serial?
>

Sparse matrices are always divided by rows across processors.

  Thanks,

    Matt


> By the way, I've figured out and fixed the bugs in my code concerning
> using mg with DMDA having 4 dof. It has to do with the interpolations. Now
> I can see mg works well with 4 dof DMDA.
>
> Best,
> Hui
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Saturday, February 28, 2015 9:35 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
> > On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> >
> > Thank you Barry. Another question: I observe that in those ksp examples,
> whenever multigrid is used, DMDA is also used, besides,
> KSPSetComputeOperators and KSPSetComputeRHS are also used.
> >
> > Is it true that
> > 1) Only DMDA can use mg?
>
>    No this is not true
>
> > 2) We have to set up matrices and rhs using KSPSetComputeOperators  and
> KSPSetComputeRHS?
>
>    No you do not have to
>
> > We cannot create a matrix and add it to KSP if we want to use mg?
>
>     Yes you can.
>
>    There are many many variants of multigrid one can do with PETSc; we
> don't have the time to have examples of all the possibilities.
>
> More details
>
> > 1) Only DMDA can use mg?
>
>    Because DMDA provides structured grids with easy interpolation between
> levels and it is easy for users to write Jacobians we have many examples
> that use the DMDA. However, so long as YOU (or something) can provide
> interpolation between the multigrid levels you can use multigrid. For
> example PCGAMG uses algebraic multigrid to generate the interpolations. If
> you have your own interpolations you can provide them with
> PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially
> handles those details automatically for you).
>
> > 2) We have to set up matrices and rhs using KSPSetComputeOperators  and
> KSPSetComputeRHS?
>
>    Normally with geometric multigrid one discretizes the operator on each
> level of the grid. Thus the user has to provide several matrices (one for
> each level). KSPSetComputeOperators() is ONE way that the user can provide
> them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and
> then call KSPSetOperators(ksp,...) for each of the levels
> (KSPSetComputeOperators() essentially does the book keeping for you).
>
> > We cannot create a matrix and add it to KSP if we want to use mg?
>
>     As I said in 2 normally multigrid requires you to provide a
> discretized operator at each level. But with Galerkin coarse grids (which
> is what algebraic multigrid users and can also be used by geometric
> multigrid) the user does not provide coarser grid operators instead the
> code computes them automatically from the formula R*A*P where R is the
> restriction operator used in multigrid and P is the interpolation operator
> (usually the transpose of P).
>
>    If you are looking for a simple automatic multigrid then you want to
> use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you
> provide interpolations or coarser operators. However algebraic multigrid
> doesn't work for all problems; though it does work for many. Try it with
> -pc_type gamg
>
>   Barry
>
> >
> > Best,
> > Hui
> >
> > ________________________________________
> > From: Barry Smith [bsmith at mcs.anl.gov]
> > Sent: Friday, February 27, 2015 5:11 PM
> > To: Sun, Hui
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> >
> >> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> >>
> >> I'm trying to work on 4 Poisson's equations defined on a DMDA grid,
> Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil
> width to be 4, and stencil type to be star.
> >
> >  Use a stencil width of 1, not 4. The stencil width is defined in terms
> of dof.
> >>
> >> If I run the code with -pc_type ilu and -ksp_type gmres, it works
> alright.
> >>
> >> However, if I run with pc_type mg, it gives me an error saying that
> when it is doing MatSetValues, the argument is out of range, and there is a
> new nonzero at (60,64) in the matrix. However, that new nonzero is expected
> to be there, the row number 60 corresponds to i=15 and c=0 in x direction,
> and the column number 64 corresponds to i=16 and c=0 in x direction. So
> they are next to each other, and the star stencil with width 1 should
> include that. I have also checked with the memory allocations, and I'm
> found no problem.
> >>
> >> So I'm wondering if there is any problem of using multigrid on a DMDA
> with dof greater than 1?
> >
> >  No it handles dof > 1 fine.
> >
> >  Send your code.
> >
> >  Barry
> >
> >>
> >> Thank you!
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/e6e5af71/attachment-0001.html>

From knepley at gmail.com  Sun Mar  1 08:06:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Mar 2015 08:06:01 -0600
Subject: [petsc-users] PetscLogStage functions not working in fortran
In-Reply-To: <54F2B196.5070100@columbi.edu>
References: <54F2B196.5070100@columbi.edu>
Message-ID: <CAMYG4GmSrxvuJycsxSBtu8=yMzd08gRoTro+6sXmBSJFcjGY8Q@mail.gmail.com>

On Sun, Mar 1, 2015 at 12:28 AM, Luc Berger-Vergiat <lb2653 at columbia.edu>
wrote:

> Hi all,
> I am trying to fine gain a little my program profiling but cannot get
> PetscLogStageRegiset, PetscLogStagePush and PetscLogStagePop to work in my
> Fortran code.
> Here is my fortran subroutine:
>

They are certainly there:


https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118

What do you get for

  nm $PETSC_ARCH/lib/libpetsc.a | grep -i PetscLogStageRegister

     Matt


>       subroutine SetPetscStage(StageNum)
>
>       implicit  none
>
> #     include  "finclude/petsc.h"
>
>       integer        StageNum
>       character      StageName*5
>       PetscErrorCode ierr
>       PetscInt       Stage
>
>       if(StageNum.eq.1) then
>          Stage = 1
>          PetscLogStageRegister("Jacobian and residual assembly",
>      &        Stage,ierr)
>          PetscLogStagePush(Stage,ierr)
>       elseif(StageNum.eq.11) then
>          PetscLogStagePop()
>       endif
>
>       end
>
> I get the following compiler error when I try to make my program:
>
> SetPetscStage.F:14.9:
>
>          PetscLogStageRegister("Jacobian and residual assembly",
>          1
> Error: Unclassifiable statement at (1)
> SetPetscStage.F:16.9:
>
> PetscLogStagePush(Stage,ierr)
>          1
> Error: Unclassifiable statement at (1)
> SetPetscStage.F:18.9:
>
> PetscLogStagePop()
>          1
> Error: Unclassifiable statement at (1)
>
> Any idea on what is be happening?
>
> --
> Best,
> Luc
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/2a04edee/attachment.html>

From lb2653 at columbia.edu  Sun Mar  1 08:09:08 2015
From: lb2653 at columbia.edu (luc berger-vergiat)
Date: Sun, 1 Mar 2015 09:09:08 -0500
Subject: [petsc-users] PetscLogStage functions not working in fortran
In-Reply-To: <CAMYG4GmSrxvuJycsxSBtu8=yMzd08gRoTro+6sXmBSJFcjGY8Q@mail.gmail.com>
References: <54F2B196.5070100@columbi.edu>
	<CAMYG4GmSrxvuJycsxSBtu8=yMzd08gRoTro+6sXmBSJFcjGY8Q@mail.gmail.com>
Message-ID: <947F92D5-0B5A-43F9-B1B7-AF52619F4104@columbia.edu>

I get the following:

nm /home/luc/research/petsc-3.5.2/arch-opt/lib/libpetsc.a | grep -i PetscLogStageRegister
0000000000000338 T PetscLogStageRegister
00000000000004ba T petsclogstageregister_
                 U PetscLogStageRegister
                 U PetscLogStageRegister

> On Mar 1, 2015, at 9:06 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sun, Mar 1, 2015 at 12:28 AM, Luc Berger-Vergiat <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
> Hi all,
> I am trying to fine gain a little my program profiling but cannot get PetscLogStageRegiset, PetscLogStagePush and PetscLogStagePop to work in my Fortran code.
> Here is my fortran subroutine:
> 
> They are certainly there:
> 
>   https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118 <https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118>
> 
> What do you get for
> 
>   nm $PETSC_ARCH/lib/libpetsc.a | grep -i PetscLogStageRegister
> 
>      Matt
>  
>       subroutine SetPetscStage(StageNum)
> 
>       implicit  none
> 
> #     include  "finclude/petsc.h"
> 
>       integer        StageNum
>       character      StageName*5
>       PetscErrorCode ierr
>       PetscInt       Stage
> 
>       if(StageNum.eq.1) then
>          Stage = 1
>          PetscLogStageRegister("Jacobian and residual assembly",
>      &        Stage,ierr)
>          PetscLogStagePush(Stage,ierr)
>       elseif(StageNum.eq.11) then
>          PetscLogStagePop()
>       endif
> 
>       end
> 
> I get the following compiler error when I try to make my program:
> 
> SetPetscStage.F:14.9:
> 
>          PetscLogStageRegister("Jacobian and residual assembly",
>          1
> Error: Unclassifiable statement at (1)
> SetPetscStage.F:16.9:
> 
> PetscLogStagePush(Stage,ierr)
>          1
> Error: Unclassifiable statement at (1)
> SetPetscStage.F:18.9:
> 
> PetscLogStagePop()
>          1
> Error: Unclassifiable statement at (1)
> 
> Any idea on what is be happening?
> 
> -- 
> Best,
> Luc
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/d3285cb9/attachment.html>

From knepley at gmail.com  Sun Mar  1 08:15:27 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Mar 2015 08:15:27 -0600
Subject: [petsc-users] PetscLogStage functions not working in fortran
In-Reply-To: <947F92D5-0B5A-43F9-B1B7-AF52619F4104@columbia.edu>
References: <54F2B196.5070100@columbi.edu>
	<CAMYG4GmSrxvuJycsxSBtu8=yMzd08gRoTro+6sXmBSJFcjGY8Q@mail.gmail.com>
	<947F92D5-0B5A-43F9-B1B7-AF52619F4104@columbia.edu>
Message-ID: <CAMYG4GkOmTa0-MmDKWkMxacS=5Df+RvMn4wOFe0h74i_uoq2jw@mail.gmail.com>

Don't you need a 'call' first?

   Matt

On Sun, Mar 1, 2015 at 8:09 AM, luc berger-vergiat <lb2653 at columbia.edu>
wrote:

> I get the following:
>
> nm /home/luc/research/petsc-3.5.2/arch-opt/lib/libpetsc.a | grep -i
> PetscLogStageRegister
> 0000000000000338 T PetscLogStageRegister
> 00000000000004ba T petsclogstageregister_
>                  U PetscLogStageRegister
>                  U PetscLogStageRegister
>
> On Mar 1, 2015, at 9:06 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sun, Mar 1, 2015 at 12:28 AM, Luc Berger-Vergiat <lb2653 at columbia.edu>
> wrote:
>
>> Hi all,
>> I am trying to fine gain a little my program profiling but cannot get
>> PetscLogStageRegiset, PetscLogStagePush and PetscLogStagePop to work in my
>> Fortran code.
>> Here is my fortran subroutine:
>>
>
> They are certainly there:
>
>
> https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118
>
> What do you get for
>
>   nm $PETSC_ARCH/lib/libpetsc.a | grep -i PetscLogStageRegister
>
>      Matt
>
>
>>       subroutine SetPetscStage(StageNum)
>>
>>       implicit  none
>>
>> #     include  "finclude/petsc.h"
>>
>>       integer        StageNum
>>       character      StageName*5
>>       PetscErrorCode ierr
>>       PetscInt       Stage
>>
>>       if(StageNum.eq.1) then
>>          Stage = 1
>>          PetscLogStageRegister("Jacobian and residual assembly",
>>      &        Stage,ierr)
>>          PetscLogStagePush(Stage,ierr)
>>       elseif(StageNum.eq.11) then
>>          PetscLogStagePop()
>>       endif
>>
>>       end
>>
>> I get the following compiler error when I try to make my program:
>>
>> SetPetscStage.F:14.9:
>>
>>          PetscLogStageRegister("Jacobian and residual assembly",
>>          1
>> Error: Unclassifiable statement at (1)
>> SetPetscStage.F:16.9:
>>
>> PetscLogStagePush(Stage,ierr)
>>          1
>> Error: Unclassifiable statement at (1)
>> SetPetscStage.F:18.9:
>>
>> PetscLogStagePop()
>>          1
>> Error: Unclassifiable statement at (1)
>>
>> Any idea on what is be happening?
>>
>> --
>> Best,
>> Luc
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/51dedd9b/attachment.html>

From lb2653 at columbia.edu  Sun Mar  1 08:20:08 2015
From: lb2653 at columbia.edu (luc berger-vergiat)
Date: Sun, 1 Mar 2015 09:20:08 -0500
Subject: [petsc-users] PetscLogStage functions not working in fortran
In-Reply-To: <CAMYG4GkOmTa0-MmDKWkMxacS=5Df+RvMn4wOFe0h74i_uoq2jw@mail.gmail.com>
References: <54F2B196.5070100@columbi.edu>
	<CAMYG4GmSrxvuJycsxSBtu8=yMzd08gRoTro+6sXmBSJFcjGY8Q@mail.gmail.com>
	<947F92D5-0B5A-43F9-B1B7-AF52619F4104@columbia.edu>
	<CAMYG4GkOmTa0-MmDKWkMxacS=5Df+RvMn4wOFe0h74i_uoq2jw@mail.gmail.com>
Message-ID: <9D4D0212-85A6-466F-A732-0C6E1CD1DD7D@columbia.edu>

True,
that?ll teach me to wait until the next morning before mailing the user-list when I program at 1am?
Thanks a lot for taking the time to help anyway!

Best,
Luc

> On Mar 1, 2015, at 9:15 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> Don't you need a 'call' first?
> 
>    Matt
> 
> On Sun, Mar 1, 2015 at 8:09 AM, luc berger-vergiat <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
> I get the following:
> 
> nm /home/luc/research/petsc-3.5.2/arch-opt/lib/libpetsc.a | grep -i PetscLogStageRegister
> 0000000000000338 T PetscLogStageRegister
> 00000000000004ba T petsclogstageregister_
>                  U PetscLogStageRegister
>                  U PetscLogStageRegister
> 
>> On Mar 1, 2015, at 9:06 AM, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Sun, Mar 1, 2015 at 12:28 AM, Luc Berger-Vergiat <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
>> Hi all,
>> I am trying to fine gain a little my program profiling but cannot get PetscLogStageRegiset, PetscLogStagePush and PetscLogStagePop to work in my Fortran code.
>> Here is my fortran subroutine:
>> 
>> They are certainly there:
>> 
>>   https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118 <https://bitbucket.org/petsc/petsc/src/5620d6dc8304f23d8d30f97758f32d1dd71aa0ff/src/sys/logging/ftn-custom/zplogf.c?at=master#cl-118>
>> 
>> What do you get for
>> 
>>   nm $PETSC_ARCH/lib/libpetsc.a | grep -i PetscLogStageRegister
>> 
>>      Matt
>>  
>>       subroutine SetPetscStage(StageNum)
>> 
>>       implicit  none
>> 
>> #     include  "finclude/petsc.h"
>> 
>>       integer        StageNum
>>       character      StageName*5
>>       PetscErrorCode ierr
>>       PetscInt       Stage
>> 
>>       if(StageNum.eq.1) then
>>          Stage = 1
>>          PetscLogStageRegister("Jacobian and residual assembly",
>>      &        Stage,ierr)
>>          PetscLogStagePush(Stage,ierr)
>>       elseif(StageNum.eq.11) then
>>          PetscLogStagePop()
>>       endif
>> 
>>       end
>> 
>> I get the following compiler error when I try to make my program:
>> 
>> SetPetscStage.F:14.9:
>> 
>>          PetscLogStageRegister("Jacobian and residual assembly",
>>          1
>> Error: Unclassifiable statement at (1)
>> SetPetscStage.F:16.9:
>> 
>> PetscLogStagePush(Stage,ierr)
>>          1
>> Error: Unclassifiable statement at (1)
>> SetPetscStage.F:18.9:
>> 
>> PetscLogStagePop()
>>          1
>> Error: Unclassifiable statement at (1)
>> 
>> Any idea on what is be happening?
>> 
>> -- 
>> Best,
>> Luc
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/71c1fbb6/attachment-0001.html>

From hus003 at ucsd.edu  Sun Mar  1 10:09:12 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sun, 1 Mar 2015 16:09:12 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <CAMYG4GnotCRxB1J0DAMnT03CF883pDSG70TLpq=X24KEMtzCYg@mail.gmail.com>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<CAMYG4GnotCRxB1J0DAMnT03CF883pDSG70TLpq=X24KEMtzCYg@mail.gmail.com>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9CE9@XMAIL-MBX-BH1.AD.UCSD.EDU>

Hi Matt, I don't think what you said concerning using -pc_mg_galerkin to have it projected is what I'm looking for. What I try to do is the following:
I already have ComputeRHS and ComputeMatrix, and hence:

    ierr = KSPSetComputeRHS(ksp,ComputeRHS,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp,ComputeMatrix,NULL);CHKERRQ(ierr);

    ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);

    ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);

Now I have a vector v, let's say:
ierr = KSPGetSolution(ksp,&v);CHKERRQ(ierr);

After that, I want so set v to be my rhs, and solve the new system. It doesn't seem that something like this would work:

     ierr = KSPSolve(ksp,&x,&x);CHKERRQ(ierr);

So how do I set x to the rhs? It does not seem to me that -pc_mg_galerkin would set x as the rhs of the system.


________________________________
From: Matthew Knepley [knepley at gmail.com]
Sent: Sunday, March 01, 2015 6:02 AM
To: Sun, Hui
Cc: Barry Smith; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

On Sun, Mar 1, 2015 at 12:30 AM, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
Thank you Barry. I have yet two more questions:

1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?

If you just give a vector, you could use -pc_mg_galerkin to have it projected. Same with the matrix.

2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?

Sparse matrices are always divided by rows across processors.

  Thanks,

    Matt

By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.

Best,
Hui

________________________________________
From: Barry Smith [bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>]
Sent: Saturday, February 28, 2015 9:35 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
>
> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>
> Is it true that
> 1) Only DMDA can use mg?

   No this is not true

> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?

   No you do not have to

> We cannot create a matrix and add it to KSP if we want to use mg?

    Yes you can.

   There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.

More details

> 1) Only DMDA can use mg?

   Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).

> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?

   Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).

> We cannot create a matrix and add it to KSP if we want to use mg?

    As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).

   If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg

  Barry

>
> Best,
> Hui
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>]
> Sent: Friday, February 27, 2015 5:11 PM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
>>
>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>
>  Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>
>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>
>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>
>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>
>  No it handles dof > 1 fine.
>
>  Send your code.
>
>  Barry
>
>>
>> Thank you!



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/73b0eabe/attachment.html>

From bsmith at mcs.anl.gov  Sun Mar  1 11:22:08 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 11:22:08 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9CE9@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,
	<CAMYG4GnotCRxB1J0DAMnT03CF883pDSG70TLpq=X24KEMtzCYg@mail.gmail.com>
	<>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CE9@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <FDBC3544-6A8F-459B-994A-DD865AE1A5BF@mcs.anl.gov>


> On Mar 1, 2015, at 10:09 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Hi Matt, I don't think what you said concerning using -pc_mg_galerkin to have it projected is what I'm looking for. What I try to do is the following:
> I already have ComputeRHS and ComputeMatrix, and hence:
>     ierr = KSPSetComputeRHS(ksp,ComputeRHS,NULL);CHKERRQ(ierr);
> 
>     ierr = KSPSetComputeOperators(ksp,ComputeMatrix,NULL);CHKERRQ(ierr);
> 
>     ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
> 
>     ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);
> 
> 
> Now I have a vector v, let's say:
> ierr = KSPGetSolution(ksp,&v);CHKERRQ(ierr);
> 
> After that, I want so set v to be my rhs, and solve the new system. It doesn't seem that something like this would work: 
>      ierr = KSPSolve(ksp,&x,&x);CHKERRQ(ierr);

    ierr = KSPSolve(ksp,v,x); CHKERRQ(ierr); should work. You do not need to do use KSPSetComputeRHS()

  Barry

> 
> 
> So how do I set x to the rhs? It does not seem to me that -pc_mg_galerkin would set x as the rhs of the system. 
> 
> 
> From: Matthew Knepley [knepley at gmail.com]
> Sent: Sunday, March 01, 2015 6:02 AM
> To: Sun, Hui
> Cc: Barry Smith; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
> On Sun, Mar 1, 2015 at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
> Thank you Barry. I have yet two more questions:
> 
> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
> 
> If you just give a vector, you could use -pc_mg_galerkin to have it projected. Same with the matrix.
>  
> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
> 
> Sparse matrices are always divided by rows across processors.
> 
>   Thanks,
> 
>     Matt
>  
> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
> 
> Best,
> Hui
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Saturday, February 28, 2015 9:35 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
> > On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> >
> > Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
> >
> > Is it true that
> > 1) Only DMDA can use mg?
> 
>    No this is not true
> 
> > 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
> 
>    No you do not have to
> 
> > We cannot create a matrix and add it to KSP if we want to use mg?
> 
>     Yes you can.
> 
>    There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
> 
> More details
> 
> > 1) Only DMDA can use mg?
> 
>    Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
> 
> > 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
> 
>    Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
> 
> > We cannot create a matrix and add it to KSP if we want to use mg?
> 
>     As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
> 
>    If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
> 
>   Barry
> 
> >
> > Best,
> > Hui
> >
> > ________________________________________
> > From: Barry Smith [bsmith at mcs.anl.gov]
> > Sent: Friday, February 27, 2015 5:11 PM
> > To: Sun, Hui
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> >
> >> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> >>
> >> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
> >
> >  Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
> >>
> >> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
> >>
> >> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
> >>
> >> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
> >
> >  No it handles dof > 1 fine.
> >
> >  Send your code.
> >
> >  Barry
> >
> >>
> >> Thank you!
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From bsmith at mcs.anl.gov  Sun Mar  1 11:24:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 11:24:00 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov>


> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Thank you Barry. I have yet two more questions:
> 
> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector? 
> 
> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial? 

  Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.

> 
> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA. 
> 
> Best, 
> Hui
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Saturday, February 28, 2015 9:35 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>> 
>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>> 
>> Is it true that
>> 1) Only DMDA can use mg?
> 
>   No this is not true
> 
>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
> 
>   No you do not have to
> 
>> We cannot create a matrix and add it to KSP if we want to use mg?
> 
>    Yes you can.
> 
>   There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
> 
> More details
> 
>> 1) Only DMDA can use mg?
> 
>   Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
> 
>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
> 
>   Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
> 
>> We cannot create a matrix and add it to KSP if we want to use mg?
> 
>    As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
> 
>   If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
> 
>  Barry
> 
>> 
>> Best,
>> Hui
>> 
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Friday, February 27, 2015 5:11 PM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>> 
>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>> 
>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>> 
>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>> 
>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>> 
>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>> 
>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>> 
>> No it handles dof > 1 fine.
>> 
>> Send your code.
>> 
>> Barry
>> 
>>> 
>>> Thank you!


From stali at geology.wisc.edu  Sun Mar  1 11:42:20 2015
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Sun, 01 Mar 2015 11:42:20 -0600
Subject: [petsc-users] working with petsc-dev
Message-ID: <54F34F7C.8010101@geology.wisc.edu>

Hello

When working with the development version of PETSc what exactly should I 
do each time I perform "git pull"?

Can I simply run make all or do I have to run make distclean and 
reconfigure/rebuild everything from scratch (so that external packages 
are also updated as needed).

Thanks in advance.

T

From bsmith at mcs.anl.gov  Sun Mar  1 11:49:04 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 11:49:04 -0600
Subject: [petsc-users] working with petsc-dev
In-Reply-To: <54F34F7C.8010101@geology.wisc.edu>
References: <54F34F7C.8010101@geology.wisc.edu>
Message-ID: <42FAFF97-9C0A-4EE4-B9DE-D798E161563E@mcs.anl.gov>


  It depends on how much has been changed in the repository since you pulled last. It may be a simple make is enough or it may be that a complete clear and rebuild is required. Unfortunately it is not possible to know, thus it is only safe to do a complete reconfigure. Note that unless you are pulling to get a specific new feature that was added there probably isn't a reason to git pull all that often (certainly not daily) maybe every couple of weeks or even once a month.

   Barry

> On Mar 1, 2015, at 11:42 AM, Tabrez Ali <stali at geology.wisc.edu> wrote:
> 
> Hello
> 
> When working with the development version of PETSc what exactly should I do each time I perform "git pull"?
> 
> Can I simply run make all or do I have to run make distclean and reconfigure/rebuild everything from scratch (so that external packages are also updated as needed).
> 
> Thanks in advance.
> 
> T


From stali at geology.wisc.edu  Sun Mar  1 12:09:19 2015
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Sun, 01 Mar 2015 12:09:19 -0600
Subject: [petsc-users] working with petsc-dev
In-Reply-To: <42FAFF97-9C0A-4EE4-B9DE-D798E161563E@mcs.anl.gov>
References: <54F34F7C.8010101@geology.wisc.edu>
	<42FAFF97-9C0A-4EE4-B9DE-D798E161563E@mcs.anl.gov>
Message-ID: <54F355CF.7080303@geology.wisc.edu>

OK Thanks. Yes, I wont be pulling often. Once a month should be more 
than enough. I just wanted to make sure about the need for a complete 
reconf.

Regards,
T


On 03/01/2015 11:49 AM, Barry Smith wrote:
>    It depends on how much has been changed in the repository since you pulled last. It may be a simple make is enough or it may be that a complete clear and rebuild is required. Unfortunately it is not possible to know, thus it is only safe to do a complete reconfigure. Note that unless you are pulling to get a specific new feature that was added there probably isn't a reason to git pull all that often (certainly not daily) maybe every couple of weeks or even once a month.
>
>     Barry
>
>> On Mar 1, 2015, at 11:42 AM, Tabrez Ali <stali at geology.wisc.edu> wrote:
>>
>> Hello
>>
>> When working with the development version of PETSc what exactly should I do each time I perform "git pull"?
>>
>> Can I simply run make all or do I have to run make distclean and reconfigure/rebuild everything from scratch (so that external packages are also updated as needed).
>>
>> Thanks in advance.
>>
>> T


From fd.kong at siat.ac.cn  Sun Mar  1 13:47:22 2015
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 1 Mar 2015 12:47:22 -0700
Subject: [petsc-users] how to switch ILU to its block version?
Message-ID: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>

Hi all,

I have a code with ILU as subdomain solver, and it works well. Now I want
to try a block version of ILU, there are some easy ways to implement this?

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/e5d20ae4/attachment.html>

From stali at geology.wisc.edu  Sun Mar  1 14:05:57 2015
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Sun, 01 Mar 2015 14:05:57 -0600
Subject: [petsc-users] petsc-dev and Fortran include files
Message-ID: <54F37125.6050901@geology.wisc.edu>

Hello again

This is probably not directly related to petsc but what is the simplest 
way to handle the change in the path of the include files (in petsc-dev) 
while using Fortran?

I guess I can do the following:

#include <petscversion.h>
#if (PETSC_VERSION_RELEASE==0)
#include <petsc-finclude/petsc.h90>
#else
#include <finclude/petsc.h90>
#endif

but then I have to repeat it a number of times in each 
module/subroutine. Is there a simpler way? I am not sure if I can use a 
preprocessor variable in the include directive.

T

From bsmith at mcs.anl.gov  Sun Mar  1 15:30:55 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 15:30:55 -0600
Subject: [petsc-users] petsc-dev and Fortran include files
In-Reply-To: <54F37125.6050901@geology.wisc.edu>
References: <54F37125.6050901@geology.wisc.edu>
Message-ID: <7F9338D8-98B9-4C57-9388-8C52056330B1@mcs.anl.gov>


  Make a symbolic link in the file system from the old finclude directory to the new petsc-finclude then you can always use the new style or make a link from the new to the old then you can always use the old style.

  Barry

> On Mar 1, 2015, at 2:05 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:
> 
> Hello again
> 
> This is probably not directly related to petsc but what is the simplest way to handle the change in the path of the include files (in petsc-dev) while using Fortran?
> 
> I guess I can do the following:
> 
> #include <petscversion.h>
> #if (PETSC_VERSION_RELEASE==0)
> #include <petsc-finclude/petsc.h90>
> #else
> #include <finclude/petsc.h90>
> #endif
> 
> but then I have to repeat it a number of times in each module/subroutine. Is there a simpler way? I am not sure if I can use a preprocessor variable in the include directive.
> 
> T


From bsmith at mcs.anl.gov  Sun Mar  1 15:36:30 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 15:36:30 -0600
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
Message-ID: <44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>


  Fande,

    The MATSEQBAIJ matrix automatically uses a point block ILU; with the block size of the matrix. So if your matrix naturally has a block size bs if you use the MATSEQBAIJ matrix type it will be point block ILU of block size bs.

    We don't have a variable point block ILU in PETSc for SeqAIJ. It is possible to write but would be rather involved.

   Barry

> On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> 
> Hi all,
> 
> I have a code with ILU as subdomain solver, and it works well. Now I want to try a block version of ILU, there are some easy ways to implement this?
> 
> Fande,


From bsmith at mcs.anl.gov  Sun Mar  1 16:26:30 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 16:26:30 -0600
Subject: [petsc-users] do SNESVI objects support multigrid?
In-Reply-To: <CAOHboJ-rs3RD12_Snu96-oyMbdWGaSeYPnAHe_Dfg3LZaoQ8Xw@mail.gmail.com>
References: <CAOHboJ-Ov9LyqOQ35-upDYK=xQramd9R-U-VykuxyC+Bi_S6nA@mail.gmail.com>
	<2F686956-82E5-43BC-AF3C-0E14D3952373@mcs.anl.gov>
	<CAOHboJ-rs3RD12_Snu96-oyMbdWGaSeYPnAHe_Dfg3LZaoQ8Xw@mail.gmail.com>
Message-ID: <BD7BE5D7-6FCF-48D0-9A77-2094255B8AFD@mcs.anl.gov>


   Ed,

   I have fixed ex9 so it works super nicely with multigrid in the branch pr276/elbueler/fix-ex9/master and next

$ ./ex9 -da_refine 5 -pc_type mg -snes_monitor  -ksp_monitor_true_residual -snes_vi_monitor
setup done: grid  Mx,My = 321,321  with spacing  dx,dy = 0.013,0.013
  0 SNES Function norm 1.691947292888e+01 
  0 SNES VI Function norm 1.691947292888e+01 Active lower constraints 19981/20079 upper constraints 0/0 Percent of total 0.193913 Percent of bounded 0
    0 KSP preconditioned resid norm 2.107314279885e+01 true resid norm 1.691947292888e+01 ||r(i)||/||b|| 1.000000000000e+00
    1 KSP preconditioned resid norm 2.771357241740e-01 true resid norm 2.720618543577e-01 ||r(i)||/||b|| 1.607980671155e-02
    2 KSP preconditioned resid norm 4.702355133797e-03 true resid norm 8.162965549257e-03 ||r(i)||/||b|| 4.824598014118e-04
    3 KSP preconditioned resid norm 1.104885629534e-04 true resid norm 2.674390325875e-04 ||r(i)||/||b|| 1.580658178370e-05

  The problem was your scaling of the Dirichlet boundary conditions made geometric multigrid not work. I first turned off the VI part of the example and got terrible multigrid convergence and this led me to the fix. So it shows while debugging with constraints it is always good to debug without them first.


  Barry


> On Feb 28, 2015, at 9:00 PM, Ed Bueler <elbueler at alaska.edu> wrote:
> 
> Barry --
> 
> >  If you would like I could take a look at trying to get it working again as it did before.
> 
> I would like that.
> 
> I think the availability of robust SNESVI is the key to hugely improving how we model conservation in thin layers (e.g. ice sheets, sea ice, surface hydrology, ...); see http://www2.gi.alaska.edu/snowice/glaciers/iceflow/bueler-liwg2015.pdf, a short and easy talk.  (View with acroread for the movies.)
> 
> Currently everyone in ice sheets (specifically) is doing explicit time stepping, in substantial part because that's how they think about the free boundary of the ice sheet, where the melt exceeds snowfall+flow.  By changing to thinking of each time-step as a VI you get to add "thickness of ice is nonnegative", as a constraint, to the problem, instead of ad hoc time-splitting discrete ways of moving the free boundary.
> 
> Ed
> 
> 
> On Sat, Feb 28, 2015 at 10:40 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Ed,
> 
>    At one time it did. However it has suffered from neglect and bit rot over several years. Part of the reason is that we have had big plans to "do it much better" and hence neglected the old code; while we never "did it much better".
> 
>    So algorithmically what we implemented as an active set method for the VI with multigrid solving and I feel algorithmically it is a pretty good method (for problems where multigrid works well without the VI). My implementation was a bit ad hoc and didn't mesh great with the rest of PETSc which is why it hasn't been maintained properly. If you would like I could take a look at trying to get it working again as it did before.
> 
>   Barry
> 
> 
> 
> > On Feb 27, 2015, at 10:28 PM, Ed Bueler <elbueler at alaska.edu> wrote:
> >
> > Dear Petsc --
> >
> > I am confused on whether  -pc_type mg  is supported for SNESVI solvers.  The error messages sure aren't helping.
> >
> > I am using petsc-maint.
> >
> > First I looked at couple of simplest examples in src/snes/examples/tutorials, namely ex9 and ex54.  I got mixed results (below), namely crashes with 'nonconforming object sizes' or 'zero pivot' errors in 3 of 4 cases.  But I am not sure from the very limited docs what is supposed to work.
> >
> > The documented ex54 run itself (see line 12 of ex54.c) just hangs/stagnates for me:
> >
> > $ ./ex54 -pc_type mg -pc_mg_galerkin -T .01 -da_grid_x 65 -da_grid_y 65 -pc_mg_levels 4 -ksp_type fgmres -snes_atol 1.e-14 -mat_no_inode -snes_vi_monitor -snes_monitor
> >   0 SNES Function norm 6.177810506000e-03
> >   0 SNES VI Function norm 6.177810506000e-03 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0 Percent of bounded 0
> >   0 SNES Function norm 6.177810506000e-03
> >   0 SNES VI Function norm 6.177810506000e-03 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0 Percent of bounded 0
> >   0 SNES Function norm 6.177810506000e-03
> >   0 SNES VI Function norm 6.177810506000e-03 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0 Percent of bounded 0
> >   0 SNES Function norm 6.177810506000e-03
> >   0 SNES VI Function norm 6.177810506000e-03 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0 Percent of bounded 0
> >   0 SNES Function norm 6.177810506000e-03
> >   0 SNES VI Function norm 6.177810506000e-03 Active lower constraints 0/0 upper constraints 0/0 Percent of total 0 Percent of bounded 0
> > ...
> >
> > The wordy documented run of ex55.c stagnates and then seg faults for me (see attachment):
> >
> > ~/petsc-maint/src/snes/examples/tutorials[maint*]$ ./ex55 -ksp_type fgmres -pc_type mg -mg_levels_ksp_type fgmres -mg_levels_pc_type fieldsplit -mg_levels_pc_fieldsplit_detect_saddle_point -mg_levels_pc_fieldsplit_type schur -mg_levels_pc_fieldsplit_factorization_type full -mg_levels_pc_fieldsplit_schur_precondition user -mg_levels_fieldsplit_1_ksp_type gmres -mg_levels_fieldsplit_1_pc_type none -mg_levels_fieldsplit_0_ksp_type preonly -mg_levels_fieldsplit_0_pc_type sor -mg_levels_fieldsplit_0_pc_sor_forward -snes_vi_monitor -ksp_monitor_true_residual -pc_mg_levels 5 -pc_mg_galerkin -mg_levels_ksp_monitor -mg_levels_fieldsplit_ksp_monitor -mg_levels_ksp_max_it 2 -mg_levels_fieldsplit_ksp_max_it 5 -snes_atol 1.e-11 -mg_coarse_ksp_type preonly -mg_coarse_pc_type svd -da_grid_x 65 -da_grid_y 65 -ksp_rtol 1.e-8 &> out.ex55err
> >
> > These examples all seem to perform o.k. with default (and non-multigrid) options.
> >
> > Thanks for help!
> >
> > Ed
> >
> >
> > errors from simple ex9, ex54 runs below:
> >
> > ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -da_refine 2 -snes_type vinewtonrsls -pc_type mg
> > setup done: square       side length = 4.000
> >             grid               Mx,My = 41,41
> >             spacing            dx,dy = 0.100,0.100
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Nonconforming object sizes
> > [0]PETSC ERROR: Mat mat,Vec x: global dim 1388 1681
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> > [0]PETSC ERROR: ./ex9 on a linux-c-opt named bueler-leopard by ed Fri Feb 27 21:02:20 2015
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=0
> > [0]PETSC ERROR: #1 MatMultTranspose() line 2232 in /home/ed/petsc-maint/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #2 MatRestrict() line 7529 in /home/ed/petsc-maint/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #3 DMRestrictHook_SNESVecSol() line 480 in /home/ed/petsc-maint/src/snes/interface/snes.c
> > [0]PETSC ERROR: #4 DMRestrict() line 2022 in /home/ed/petsc-maint/src/dm/interface/dm.c
> > [0]PETSC ERROR: #5 PCSetUp_MG() line 699 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/ed/petsc-maint/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #7 KSPSetUp() line 306 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #8 SNESSolve_VINEWTONRSLS() line 506 in /home/ed/petsc-maint/src/snes/impls/vi/rs/virs.c
> > [0]PETSC ERROR: #9 SNESSolve() line 3743 in /home/ed/petsc-maint/src/snes/interface/snes.c
> > [0]PETSC ERROR: #10 main() line 122 in /home/ed/petsc-maint/src/snes/examples/tutorials/ex9.c
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 60) - process 0
> > [unset]: aborting job:
> > application called MPI_Abort(MPI_COMM_WORLD, 60) - process 0
> >
> >
> > ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -da_refine 2 -snes_type vinewtonssls -pc_type mg
> > setup done: square       side length = 4.000
> >             grid               Mx,My = 41,41
> >             spacing            dx,dy = 0.100,0.100
> > number of Newton iterations = 8; result = CONVERGED_FNORM_RELATIVE
> > errors:    av |u-uexact|  = 2.909e-04
> >            |u-uexact|_inf = 1.896e-03
> >
> >
> > ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex54 -snes_monitor -da_refine 2 -snes_type vinewtonssls -pc_type mg
> >   0 SNES Function norm 3.156635589354e-02
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
> > [0]PETSC ERROR: Zero pivot, row 0
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> > [0]PETSC ERROR: ./ex54 on a linux-c-opt named bueler-leopard by ed Fri Feb 27 21:02:43 2015
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=0
> > [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_2() line 50 in /home/ed/petsc-maint/src/mat/impls/baij/seq/dgefa2.c
> > [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2806 in /home/ed/petsc-maint/src/mat/impls/aij/seq/inode.c
> > [0]PETSC ERROR: #3 MatSOR() line 3643 in /home/ed/petsc-maint/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #4 PCApply_SOR() line 35 in /home/ed/petsc-maint/src/ksp/pc/impls/sor/sor.c
> > [0]PETSC ERROR: #5 PCApply() line 440 in /home/ed/petsc-maint/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #6 KSP_PCApply() line 230 in /home/ed/petsc-maint/include/petsc-private/kspimpl.h
> > [0]PETSC ERROR: #7 KSPInitialResidual() line 56 in /home/ed/petsc-maint/src/ksp/ksp/interface/itres.c
> > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 234 in /home/ed/petsc-maint/src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: #9 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 368 in /home/ed/petsc-maint/src/ksp/ksp/impls/cheby/cheby.c
> > [0]PETSC ERROR: #11 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #14 PCApply_MG() line 337 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #15 PCApply() line 440 in /home/ed/petsc-maint/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #16 KSP_PCApply() line 230 in /home/ed/petsc-maint/include/petsc-private/kspimpl.h
> > [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in /home/ed/petsc-maint/src/ksp/ksp/interface/itres.c
> > [0]PETSC ERROR: #18 KSPSolve_GMRES() line 234 in /home/ed/petsc-maint/src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: #19 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #20 SNESSolve_VINEWTONSSLS() line 317 in /home/ed/petsc-maint/src/snes/impls/vi/ss/viss.c
> > [0]PETSC ERROR: #21 SNESSolve() line 3743 in /home/ed/petsc-maint/src/snes/interface/snes.c
> > [0]PETSC ERROR: #22 main() line 98 in /home/ed/petsc-maint/src/snes/examples/tutorials/ex54.c
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> > [unset]: aborting job:
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> >
> >
> > ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex54 -snes_monitor -da_refine 2 -snes_type vinewtonrsls -pc_type mg
> >   0 SNES Function norm 3.160548858489e-02
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
> > [0]PETSC ERROR: Zero pivot, row 0
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
> > [0]PETSC ERROR: ./ex54 on a linux-c-opt named bueler-leopard by ed Fri Feb 27 21:02:48 2015
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=0
> > [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_2() line 50 in /home/ed/petsc-maint/src/mat/impls/baij/seq/dgefa2.c
> > [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2806 in /home/ed/petsc-maint/src/mat/impls/aij/seq/inode.c
> > [0]PETSC ERROR: #3 MatSOR() line 3643 in /home/ed/petsc-maint/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #4 PCApply_SOR() line 35 in /home/ed/petsc-maint/src/ksp/pc/impls/sor/sor.c
> > [0]PETSC ERROR: #5 PCApply() line 440 in /home/ed/petsc-maint/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #6 KSP_PCApply() line 230 in /home/ed/petsc-maint/include/petsc-private/kspimpl.h
> > [0]PETSC ERROR: #7 KSPInitialResidual() line 56 in /home/ed/petsc-maint/src/ksp/ksp/interface/itres.c
> > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 234 in /home/ed/petsc-maint/src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: #9 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 368 in /home/ed/petsc-maint/src/ksp/ksp/impls/cheby/cheby.c
> > [0]PETSC ERROR: #11 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #14 PCApply_MG() line 337 in /home/ed/petsc-maint/src/ksp/pc/impls/mg/mg.c
> > [0]PETSC ERROR: #15 PCApply() line 440 in /home/ed/petsc-maint/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #16 KSP_PCApply() line 230 in /home/ed/petsc-maint/include/petsc-private/kspimpl.h
> > [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in /home/ed/petsc-maint/src/ksp/ksp/interface/itres.c
> > [0]PETSC ERROR: #18 KSPSolve_GMRES() line 234 in /home/ed/petsc-maint/src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: #19 KSPSolve() line 460 in /home/ed/petsc-maint/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #20 SNESSolve_VINEWTONRSLS() line 536 in /home/ed/petsc-maint/src/snes/impls/vi/rs/virs.c
> > [0]PETSC ERROR: #21 SNESSolve() line 3743 in /home/ed/petsc-maint/src/snes/interface/snes.c
> > [0]PETSC ERROR: #22 main() line 98 in /home/ed/petsc-maint/src/snes/examples/tutorials/ex54.c
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> > [unset]: aborting job:
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> >
> >
> > --
> > Ed Bueler
> > Dept of Math and Stat and Geophysical Institute
> > University of Alaska Fairbanks
> > Fairbanks, AK 99775-6660
> > 301C Chapman and 410D Elvey
> > 907 474-7693 and 907 474-7199  (fax 907 474-5394)
> > <out.ex55err>
> 
> 
> 
> 
> -- 
> Ed Bueler
> Dept of Math and Stat and Geophysical Institute
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 301C Chapman and 410D Elvey
> 907 474-7693 and 907 474-7199  (fax 907 474-5394)


From fd.kong at siat.ac.cn  Sun Mar  1 16:55:58 2015
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 1 Mar 2015 15:55:58 -0700
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
Message-ID: <CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>

Thanks, Barry.

More questions. Is bs usually the number of unknowns associated with a grid
point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?

Fande,

On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Fande,
>
>     The MATSEQBAIJ matrix automatically uses a point block ILU; with the
> block size of the matrix. So if your matrix naturally has a block size bs
> if you use the MATSEQBAIJ matrix type it will be point block ILU of block
> size bs.
>
>     We don't have a variable point block ILU in PETSc for SeqAIJ. It is
> possible to write but would be rather involved.
>
>    Barry
>
> > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> >
> > Hi all,
> >
> > I have a code with ILU as subdomain solver, and it works well. Now I
> want to try a block version of ILU, there are some easy ways to implement
> this?
> >
> > Fande,
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/3d1026da/attachment-0001.html>

From knepley at gmail.com  Sun Mar  1 16:57:02 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Mar 2015 16:57:02 -0600
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
Message-ID: <CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>

On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:

> Thanks, Barry.
>
> More questions. Is bs usually the number of unknowns associated with a
> grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
>

No, but it does enable you to use MatSetValuesBlock() if you want.

   Matt


> Fande,
>
> On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Fande,
>>
>>     The MATSEQBAIJ matrix automatically uses a point block ILU; with the
>> block size of the matrix. So if your matrix naturally has a block size bs
>> if you use the MATSEQBAIJ matrix type it will be point block ILU of block
>> size bs.
>>
>>     We don't have a variable point block ILU in PETSc for SeqAIJ. It is
>> possible to write but would be rather involved.
>>
>>    Barry
>>
>> > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>> >
>> > Hi all,
>> >
>> > I have a code with ILU as subdomain solver, and it works well. Now I
>> want to try a block version of ILU, there are some easy ways to implement
>> this?
>> >
>> > Fande,
>>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/7a9a9aac/attachment.html>

From fd.kong at siat.ac.cn  Sun Mar  1 17:03:18 2015
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 1 Mar 2015 16:03:18 -0700
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
Message-ID: <CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>

Thanks, Matt.

Could bs be different from rows. I am solving a fluid-structure problem,
which has different bs in fluid and solid parts.

Fande,

On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>
>> Thanks, Barry.
>>
>> More questions. Is bs usually the number of unknowns associated with a
>> grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
>>
>
> No, but it does enable you to use MatSetValuesBlock() if you want.
>
>    Matt
>
>
>> Fande,
>>
>> On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Fande,
>>>
>>>     The MATSEQBAIJ matrix automatically uses a point block ILU; with the
>>> block size of the matrix. So if your matrix naturally has a block size bs
>>> if you use the MATSEQBAIJ matrix type it will be point block ILU of block
>>> size bs.
>>>
>>>     We don't have a variable point block ILU in PETSc for SeqAIJ. It is
>>> possible to write but would be rather involved.
>>>
>>>    Barry
>>>
>>> > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > I have a code with ILU as subdomain solver, and it works well. Now I
>>> want to try a block version of ILU, there are some easy ways to implement
>>> this?
>>> >
>>> > Fande,
>>>
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/41a8180c/attachment.html>

From knepley at gmail.com  Sun Mar  1 17:43:54 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Mar 2015 17:43:54 -0600
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
	<CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
Message-ID: <CAMYG4GmdpjG6C2uOiM6EBJ7xJEB=Rn2ACL7C8RaB-OuPjPEBSw@mail.gmail.com>

On Sun, Mar 1, 2015 at 5:03 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:

> Thanks, Matt.
>
> Could bs be different from rows. I am solving a fluid-structure problem,
> which has different bs in fluid and solid parts.
>

I think you are asking about variable block size in the matrix. We do not
support that.

  Thanks,

     Matt


> Fande,
>
> On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>>
>>> Thanks, Barry.
>>>
>>> More questions. Is bs usually the number of unknowns associated with a
>>> grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
>>>
>>
>> No, but it does enable you to use MatSetValuesBlock() if you want.
>>
>>    Matt
>>
>>
>>> Fande,
>>>
>>> On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>>   Fande,
>>>>
>>>>     The MATSEQBAIJ matrix automatically uses a point block ILU; with
>>>> the block size of the matrix. So if your matrix naturally has a block size
>>>> bs if you use the MATSEQBAIJ matrix type it will be point block ILU of
>>>> block size bs.
>>>>
>>>>     We don't have a variable point block ILU in PETSc for SeqAIJ. It is
>>>> possible to write but would be rather involved.
>>>>
>>>>    Barry
>>>>
>>>> > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I have a code with ILU as subdomain solver, and it works well. Now I
>>>> want to try a block version of ILU, there are some easy ways to implement
>>>> this?
>>>> >
>>>> > Fande,
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/c1966df2/attachment.html>

From bsmith at mcs.anl.gov  Sun Mar  1 17:48:26 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 17:48:26 -0600
Subject: [petsc-users] GMRES stability
In-Reply-To: <CAOYX9gzSfgswMAPnxPMNHab8As9MdkOj3g-9ZJGqPc6Rr8M4pQ@mail.gmail.com>
References: <CAOYX9gzxrd7-tHWLS6RZZK2AaO_O7tVbDw1s1b6vF=YewwCSuQ@mail.gmail.com>
	<0A826E90-DD1A-450B-BDE2-D5A4867B682B@mcs.anl.gov>
	<CAOYX9gx5e=TigrB3A0hE8G-qXJX9h83mRB+GSYwD3YE9oVATtg@mail.gmail.com>
	<239AD9F8-8DBE-42D6-B185-7BF246FC4900@mcs.anl.gov>
	<CAOYX9gzSfgswMAPnxPMNHab8As9MdkOj3g-9ZJGqPc6Rr8M4pQ@mail.gmail.com>
Message-ID: <041B1984-4BFB-4A6C-92F1-8CC8DBBF0B07@mcs.anl.gov>


  Ok, we need to understand what is algorithmically different between what your code is doing and PETSc is doing.

  From below PETSc is running a symmetric point block SOR with a block size of 5. Are you using a point or point block GS? Are you using any particular ordering of the unknowns in your GS, or just natural ordering? Are you sure the matrices are the same between the two codes? Print them out for a tiny grid and compare.

  Barry

> On Mar 1, 2015, at 4:33 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> 
> I don't run in parallel. I tried -ksp_richardson -pc_type but still
> cannot get over CFL=5 (with my own GS code CFL=40 was converging).
> Also, lowering damping factor does not help. Default number of
> iterations (10000) does not help as well. -ksp_view outputs:
> 
> KSP Object: 1 MPI processes
>  type: richardson
>    Richardson: damping factor=0.5
>  maximum iterations=10, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>  type: sor
>    SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
>  linear system matrix = precond matrix:
>  Mat Object:   1 MPI processes
>    type: seqbaij
>    rows=13250, cols=13250, bs=5
>    total: nonzeros=261150, allocated nonzeros=1.325e+06
>    total number of mallocs used during MatSetValues calls =0
>        block size is 5
> 
> On Fri, Feb 27, 2015 at 4:30 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>  Ok, please provide the rest of the information I asked for.
>> 
>>  Barry
>> 
>>> On Feb 27, 2015, at 2:33 AM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
>>> 
>>> No. It does not converge at all or iow it diverges.
>>> 
>>> On Thu, Feb 26, 2015 at 9:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> By stability I assume you mean the the GMRES does not converge (or converges too slowly)?
>>>> 
>>>> The way to improve GMRES convergence is with a preconditioner better suited to your problem. By default PETSc uses GMRES with a block Jacobi preconditioner with one block per process and ILU(0) on each block. For some problems this is fine, but for many problems it will give bad convergence.
>>>> 
>>>>  What do you get for -ksp_view  (are you using the default?) Are you running yet in parallel?
>>>> 
>>>> As a test on one process you can use GS in PETSc as the preconditioner and make sure you get similar convergence to your code. For example -ksp_richardson -pc_type sor on one processor will give you a GS solver.
>>>> 
>>>>  Once we know a bit more about your problem we can suggest better preconditioners.
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> On Feb 26, 2015, at 10:25 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> I tried to solve Ax=b with my own Gauss-Seidel code and Petsc's GMRES.
>>>>> With my GS, for a steady state problem I can set CFL=40 and for
>>>>> unsteady case can set dt=0.1. However, for GMRES I can't set CFL more
>>>>> than 5 and for unsteady case dt more than 0.00001. I need GMRES for
>>>>> parallel computations so I cannot use GS for this purpose. Is there a
>>>>> way to improve the stability of GMRES?
>>>> 
>> 


From bsmith at mcs.anl.gov  Sun Mar  1 17:53:16 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 17:53:16 -0600
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
	<CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
Message-ID: <3C100FEC-AA03-400C-A52D-890F3BAC8FD9@mcs.anl.gov>


  Probably you want a better preconditioner than ILU (point or point block) for your problem. Just blindly using ILU on a matrix arising from complex physics is not very efficient and is fragile.

   You might start with the PCFIELDSPLIT where you have one "field" for the fluid and one for the "solid" part. Then you can use whatever appropriate preconditioners on each part; for example on the "solid" part you want to use PCGAMG (make sure you set the block size and the near null space for that one). The fluid part you might again use PCFIELDSPLIT on (yes it can be used recursively).

   Barry

> On Mar 1, 2015, at 5:03 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> 
> Thanks, Matt.
> 
> Could bs be different from rows. I am solving a fluid-structure problem, which has different bs in fluid and solid parts.
> 
> Fande,
> 
> On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> Thanks, Barry.
> 
> More questions. Is bs usually the number of unknowns associated with a grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
> 
> No, but it does enable you to use MatSetValuesBlock() if you want.
> 
>    Matt
>  
> Fande,
> 
> On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Fande,
> 
>     The MATSEQBAIJ matrix automatically uses a point block ILU; with the block size of the matrix. So if your matrix naturally has a block size bs if you use the MATSEQBAIJ matrix type it will be point block ILU of block size bs.
> 
>     We don't have a variable point block ILU in PETSc for SeqAIJ. It is possible to write but would be rather involved.
> 
>    Barry
> 
> > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> >
> > Hi all,
> >
> > I have a code with ILU as subdomain solver, and it works well. Now I want to try a block version of ILU, there are some easy ways to implement this?
> >
> > Fande,
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From fd.kong at siat.ac.cn  Sun Mar  1 21:58:50 2015
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 1 Mar 2015 20:58:50 -0700
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <3C100FEC-AA03-400C-A52D-890F3BAC8FD9@mcs.anl.gov>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
	<CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
	<3C100FEC-AA03-400C-A52D-890F3BAC8FD9@mcs.anl.gov>
Message-ID: <CAN5Wd-LgMDq_0mZWSVL-x1_j4a9Kn4-j0cCBvSkdSk9eVLYQRQ@mail.gmail.com>

Thanks. The fluid-structure problem has been implemented via a monolithic
coupling method. I am using geometric multigrid method as a preconditioner,
where on each level additive Schwarz method is used. And ILU(1) is used as
a subdomain solver. It does work well.  I just was wondering if a block
version of ILU can help improve the performance further since I have a few
degrees per grid point.



On Sun, Mar 1, 2015 at 4:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Probably you want a better preconditioner than ILU (point or point
> block) for your problem. Just blindly using ILU on a matrix arising from
> complex physics is not very efficient and is fragile.
>
>    You might start with the PCFIELDSPLIT where you have one "field" for
> the fluid and one for the "solid" part. Then you can use whatever
> appropriate preconditioners on each part; for example on the "solid" part
> you want to use PCGAMG (make sure you set the block size and the near null
> space for that one). The fluid part you might again use PCFIELDSPLIT on
> (yes it can be used recursively).
>
>    Barry
>
> > On Mar 1, 2015, at 5:03 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> >
> > Thanks, Matt.
> >
> > Could bs be different from rows. I am solving a fluid-structure problem,
> which has different bs in fluid and solid parts.
> >
> > Fande,
> >
> > On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > Thanks, Barry.
> >
> > More questions. Is bs usually the number of unknowns associated with a
> grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
> >
> > No, but it does enable you to use MatSetValuesBlock() if you want.
> >
> >    Matt
> >
> > Fande,
> >
> > On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Fande,
> >
> >     The MATSEQBAIJ matrix automatically uses a point block ILU; with the
> block size of the matrix. So if your matrix naturally has a block size bs
> if you use the MATSEQBAIJ matrix type it will be point block ILU of block
> size bs.
> >
> >     We don't have a variable point block ILU in PETSc for SeqAIJ. It is
> possible to write but would be rather involved.
> >
> >    Barry
> >
> > > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > >
> > > Hi all,
> > >
> > > I have a code with ILU as subdomain solver, and it works well. Now I
> want to try a block version of ILU, there are some easy ways to implement
> this?
> > >
> > > Fande,
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/9f06a01b/attachment-0001.html>

From bsmith at mcs.anl.gov  Sun Mar  1 22:08:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 1 Mar 2015 22:08:23 -0600
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <CAN5Wd-LgMDq_0mZWSVL-x1_j4a9Kn4-j0cCBvSkdSk9eVLYQRQ@mail.gmail.com>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
	<CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
	<3C100FEC-AA03-400C-A52D-890F3BAC8FD9@mcs.anl.gov>
	<CAN5Wd-LgMDq_0mZWSVL-x1_j4a9Kn4-j0cCBvSkdSk9eVLYQRQ@mail.gmail.com>
Message-ID: <D986CAF6-8723-4B26-8E21-D87CD277C39E@mcs.anl.gov>


> On Mar 1, 2015, at 9:58 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> 
> Thanks. The fluid-structure problem has been implemented via a monolithic coupling method. I am using geometric multigrid method as a preconditioner, where on each level additive Schwarz method is used. And ILU(1) is used as a subdomain solver. It does work well.  I just was wondering if a block version of ILU can help improve the performance further since I have a few degrees per grid point.

  How many KSP iterations is your multigrid solver requiring per linear solve?

  I would first check the convergence rate if you use LU on each domain. This will give you an upper bound on the best convergence rate you can get with additive Schwarz preconditioner. If it is not much better than ILU(1) then it is not worth using a "stronger" block solver. 

  Barry

> 
> 
> 
> On Sun, Mar 1, 2015 at 4:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Probably you want a better preconditioner than ILU (point or point block) for your problem. Just blindly using ILU on a matrix arising from complex physics is not very efficient and is fragile.
> 
>    You might start with the PCFIELDSPLIT where you have one "field" for the fluid and one for the "solid" part. Then you can use whatever appropriate preconditioners on each part; for example on the "solid" part you want to use PCGAMG (make sure you set the block size and the near null space for that one). The fluid part you might again use PCFIELDSPLIT on (yes it can be used recursively).
> 
>    Barry
> 
> > On Mar 1, 2015, at 5:03 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> >
> > Thanks, Matt.
> >
> > Could bs be different from rows. I am solving a fluid-structure problem, which has different bs in fluid and solid parts.
> >
> > Fande,
> >
> > On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > Thanks, Barry.
> >
> > More questions. Is bs usually the number of unknowns associated with a grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
> >
> > No, but it does enable you to use MatSetValuesBlock() if you want.
> >
> >    Matt
> >
> > Fande,
> >
> > On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Fande,
> >
> >     The MATSEQBAIJ matrix automatically uses a point block ILU; with the block size of the matrix. So if your matrix naturally has a block size bs if you use the MATSEQBAIJ matrix type it will be point block ILU of block size bs.
> >
> >     We don't have a variable point block ILU in PETSc for SeqAIJ. It is possible to write but would be rather involved.
> >
> >    Barry
> >
> > > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > >
> > > Hi all,
> > >
> > > I have a code with ILU as subdomain solver, and it works well. Now I want to try a block version of ILU, there are some easy ways to implement this?
> > >
> > > Fande,
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> 
> 
> 


From fd.kong at siat.ac.cn  Sun Mar  1 22:52:59 2015
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 1 Mar 2015 21:52:59 -0700
Subject: [petsc-users] how to switch ILU to its block version?
In-Reply-To: <D986CAF6-8723-4B26-8E21-D87CD277C39E@mcs.anl.gov>
References: <CAN5Wd-J1m5NBLm5Ou8Sa0fFDtHwFrKJ=+Ef4AiECi23gkor3NQ@mail.gmail.com>
	<44208A3B-F23B-4EB1-BADB-9F05BBB20F29@mcs.anl.gov>
	<CAN5Wd-+Buj8HxV86BLrS8pEudKW+-uG9tLwVk+MXYYzKrk5-RA@mail.gmail.com>
	<CAMYG4GmNgBiVtJZcAWbdMTuA8pphgCYH8E7kBTzvqKqw=UuO7w@mail.gmail.com>
	<CAN5Wd-K8jBHbU-518wCqe+RFZjyOpTqbqe8=GtK22ogWy0sEuQ@mail.gmail.com>
	<3C100FEC-AA03-400C-A52D-890F3BAC8FD9@mcs.anl.gov>
	<CAN5Wd-LgMDq_0mZWSVL-x1_j4a9Kn4-j0cCBvSkdSk9eVLYQRQ@mail.gmail.com>
	<D986CAF6-8723-4B26-8E21-D87CD277C39E@mcs.anl.gov>
Message-ID: <CAN5Wd-KHFfhNyKADmZzQJke+QVkGnwWSbgM6-po9UebzXSUJUQ@mail.gmail.com>

Thanks. I got a clue.

On Sun, Mar 1, 2015 at 9:08 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 1, 2015, at 9:58 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> >
> > Thanks. The fluid-structure problem has been implemented via a
> monolithic coupling method. I am using geometric multigrid method as a
> preconditioner, where on each level additive Schwarz method is used. And
> ILU(1) is used as a subdomain solver. It does work well.  I just was
> wondering if a block version of ILU can help improve the performance
> further since I have a few degrees per grid point.
>
>   How many KSP iterations is your multigrid solver requiring per linear
> solve?
>
>   I would first check the convergence rate if you use LU on each domain.
> This will give you an upper bound on the best convergence rate you can get
> with additive Schwarz preconditioner. If it is not much better than ILU(1)
> then it is not worth using a "stronger" block solver.
>
>   Barry
>
> >
> >
> >
> > On Sun, Mar 1, 2015 at 4:53 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Probably you want a better preconditioner than ILU (point or point
> block) for your problem. Just blindly using ILU on a matrix arising from
> complex physics is not very efficient and is fragile.
> >
> >    You might start with the PCFIELDSPLIT where you have one "field" for
> the fluid and one for the "solid" part. Then you can use whatever
> appropriate preconditioners on each part; for example on the "solid" part
> you want to use PCGAMG (make sure you set the block size and the near null
> space for that one). The fluid part you might again use PCFIELDSPLIT on
> (yes it can be used recursively).
> >
> >    Barry
> >
> > > On Mar 1, 2015, at 5:03 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > >
> > > Thanks, Matt.
> > >
> > > Could bs be different from rows. I am solving a fluid-structure
> problem, which has different bs in fluid and solid parts.
> > >
> > > Fande,
> > >
> > > On Sun, Mar 1, 2015 at 3:57 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > On Sun, Mar 1, 2015 at 4:55 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > > Thanks, Barry.
> > >
> > > More questions. Is bs usually the number of unknowns associated with a
> grid point. Do I need to change MatSetValues if use a MATSEQBAIJ matrix?
> > >
> > > No, but it does enable you to use MatSetValuesBlock() if you want.
> > >
> > >    Matt
> > >
> > > Fande,
> > >
> > > On Sun, Mar 1, 2015 at 2:36 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >   Fande,
> > >
> > >     The MATSEQBAIJ matrix automatically uses a point block ILU; with
> the block size of the matrix. So if your matrix naturally has a block size
> bs if you use the MATSEQBAIJ matrix type it will be point block ILU of
> block size bs.
> > >
> > >     We don't have a variable point block ILU in PETSc for SeqAIJ. It
> is possible to write but would be rather involved.
> > >
> > >    Barry
> > >
> > > > On Mar 1, 2015, at 1:47 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I have a code with ILU as subdomain solver, and it works well. Now I
> want to try a block version of ILU, there are some easy ways to implement
> this?
> > > >
> > > > Fande,
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> > >
> >
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150301/ea233c1f/attachment.html>

From gabel.fabian at gmail.com  Mon Mar  2 17:26:18 2015
From: gabel.fabian at gmail.com (Fabian Gabel)
Date: Tue, 03 Mar 2015 00:26:18 +0100
Subject: [petsc-users] Multigrid preconditioning of entire linear systems
 for discretized coupled multiphysics problems
Message-ID: <1425338778.27237.18.camel@gmail.com>

Dear PETSc Team,

I came across the following paragraph in your publication "Composable
Linear Solvers for Multiphysics" (2012):

"Rather than splitting the matrix into large blocks and
forming a preconditioner from solvers (for example, multi-
grid) on each block, one can perform multigrid on the entire
system, basing the smoother on solves coming from the tiny
blocks coupling the degrees of freedom at a single point (or
small number of points). This approach is also handled in
PETSc, but we will not elaborate on it here."

How would I use a multigrid preconditioner (GAMG) from PETSc on linear
systems of the form (after reordering the variables):

[A_uu   0     0   A_up  A_uT]
[0    A_vv    0   A_vp  A_vT]
[0      0   A_ww  A_up  A_wT]
[A_pu A_pv  A_pw  A_pp   0  ]
[A_Tu A_Tv  A_Tw  A_Tp  A_TT]

where each of the block matrices A_ij, with i,j \in {u,v,w,p,T}, results
directly from a FVM discretization of the incompressible Navier-Stokes
equations and the temperature equation. The fifth row and column are
optional, depending on the method I choose to couple the temperature.
The Matrix is stored as one AIJ Matrix.

Regards,
Fabian Gabel



From jed at jedbrown.org  Mon Mar  2 17:29:49 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 02 Mar 2015 16:29:49 -0700
Subject: [petsc-users] Multigrid preconditioning of entire linear
	systems for discretized coupled multiphysics problems
In-Reply-To: <1425338778.27237.18.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com>
Message-ID: <87pp8r9dw2.fsf@jedbrown.org>

Fabian Gabel <gabel.fabian at gmail.com> writes:

> Dear PETSc Team,
>
> I came across the following paragraph in your publication "Composable
> Linear Solvers for Multiphysics" (2012):
>
> "Rather than splitting the matrix into large blocks and
> forming a preconditioner from solvers (for example, multi-
> grid) on each block, one can perform multigrid on the entire
> system, basing the smoother on solves coming from the tiny
> blocks coupling the degrees of freedom at a single point (or
> small number of points). This approach is also handled in
> PETSc, but we will not elaborate on it here."
>
> How would I use a multigrid preconditioner (GAMG) 

The heuristics in GAMG are not appropriate for indefinite/saddle-point
systems such as arise from Navier-Stokes.  You can use geometric
multigrid and use the fieldsplit techniques described in the paper as a
smoother, for example.

> from PETSc on linear systems of the form (after reordering the
> variables):
>
> [A_uu   0     0   A_up  A_uT]
> [0    A_vv    0   A_vp  A_vT]
> [0      0   A_ww  A_up  A_wT]
> [A_pu A_pv  A_pw  A_pp   0  ]
> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
>
> where each of the block matrices A_ij, with i,j \in {u,v,w,p,T}, results
> directly from a FVM discretization of the incompressible Navier-Stokes
> equations and the temperature equation. The fifth row and column are
> optional, depending on the method I choose to couple the temperature.
> The Matrix is stored as one AIJ Matrix.
>
> Regards,
> Fabian Gabel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150302/1de99dff/attachment.pgp>

From knepley at gmail.com  Mon Mar  2 17:32:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Mar 2015 17:32:09 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <87pp8r9dw2.fsf@jedbrown.org>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
Message-ID: <CAMYG4G=pod6A-KcdTkAwFMrReZHx1jh_0pH9JmRg0Jbirrq-Ew@mail.gmail.com>

On Mon, Mar 2, 2015 at 5:29 PM, Jed Brown <jed at jedbrown.org> wrote:

> Fabian Gabel <gabel.fabian at gmail.com> writes:
>
> > Dear PETSc Team,
> >
> > I came across the following paragraph in your publication "Composable
> > Linear Solvers for Multiphysics" (2012):
> >
> > "Rather than splitting the matrix into large blocks and
> > forming a preconditioner from solvers (for example, multi-
> > grid) on each block, one can perform multigrid on the entire
> > system, basing the smoother on solves coming from the tiny
> > blocks coupling the degrees of freedom at a single point (or
> > small number of points). This approach is also handled in
> > PETSc, but we will not elaborate on it here."
> >
> > How would I use a multigrid preconditioner (GAMG)
>
> The heuristics in GAMG are not appropriate for indefinite/saddle-point
> systems such as arise from Navier-Stokes.  You can use geometric
> multigrid and use the fieldsplit techniques described in the paper as a
> smoother, for example.
>
> > from PETSc on linear systems of the form (after reordering the
> > variables):
> >
> > [A_uu   0     0   A_up  A_uT]
> > [0    A_vv    0   A_vp  A_vT]
> > [0      0   A_ww  A_up  A_wT]
> > [A_pu A_pv  A_pw  A_pp   0  ]
> > [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> >
> > where each of the block matrices A_ij, with i,j \in {u,v,w,p,T}, results
> > directly from a FVM discretization of the incompressible Navier-Stokes
> > equations and the temperature equation. The fifth row and column are
> > optional, depending on the method I choose to couple the temperature.
> > The Matrix is stored as one AIJ Matrix.
>

If you have an unstructured grid, we have examples using regular refinement
and GMG. We are working towards having irregular refinement using the
Pragmatic library from ICL.

  Thanks,

     Matt


> > Regards,
> > Fabian Gabel
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150302/27cff010/attachment.html>

From gabel.fabian at gmail.com  Mon Mar  2 18:39:34 2015
From: gabel.fabian at gmail.com (Fabian Gabel)
Date: Tue, 03 Mar 2015 01:39:34 +0100
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <87pp8r9dw2.fsf@jedbrown.org>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
Message-ID: <1425343174.27237.60.camel@gmail.com>

On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
> Fabian Gabel <gabel.fabian at gmail.com> writes:
> 
> > Dear PETSc Team,
> >
> > I came across the following paragraph in your publication "Composable
> > Linear Solvers for Multiphysics" (2012):
> >
> > "Rather than splitting the matrix into large blocks and
> > forming a preconditioner from solvers (for example, multi-
> > grid) on each block, one can perform multigrid on the entire
> > system, basing the smoother on solves coming from the tiny
> > blocks coupling the degrees of freedom at a single point (or
> > small number of points). This approach is also handled in
> > PETSc, but we will not elaborate on it here."
> >
> > How would I use a multigrid preconditioner (GAMG) 
> 
> The heuristics in GAMG are not appropriate for indefinite/saddle-point
> systems such as arise from Navier-Stokes.  You can use geometric
> multigrid and use the fieldsplit techniques described in the paper as a
> smoother, for example.

I sadly don't have a solid background on multigrid methods, but as
mentioned in a previous thread

http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html

AMG has apparently been used (successfully?) for fully-coupled
finite-volume discretizations of Navier-Stokes:

http://dx.doi.org/10.1080/10407790.2014.894448
http://dx.doi.org/10.1016/j.jcp.2008.08.027

I was hoping to achieve something similar with the right configuration
of the PETSc preconditioners. So far I have only been using GAMG in a
straightforward manner, without providing any details on the structure
of the linear system. I attached the output of a test run with GAMG.

> 
> > from PETSc on linear systems of the form (after reordering the
> > variables):
> >
> > [A_uu   0     0   A_up  A_uT]
> > [0    A_vv    0   A_vp  A_vT]
> > [0      0   A_ww  A_up  A_wT]
> > [A_pu A_pv  A_pw  A_pp   0  ]
> > [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> >
> > where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
> > directly from a FVM discretization of the incompressible Navier-Stokes
> > equations and the temperature equation. The fifth row and column are
> > optional, depending on the method I choose to couple the temperature.
> > The Matrix is stored as one AIJ Matrix.
> >
> > Regards,
> > Fabian Gabel

-------------- next part --------------
Sender: LSF System <lsfadmin at hpa0678>
Subject: Job 578677: <coupling_cpld_scalability_openmpi> in cluster <lichtenberg> Done

Job <coupling_cpld_scalability_openmpi> was submitted from host <hla0003> by user <gu08vomo> in cluster <lichtenberg>.
Job was executed on host(s) <16*hpa0678>, in queue <short>, as user <gu08vomo> in cluster <lichtenberg>.
                            <16*hpa0611>
                            <16*hpa0665>
                            <16*hpa0649>
                            <16*hpa0616>
                            <16*hpa0559>
                            <16*hpa0577>
                            <16*hpa0618>
</home/gu08vomo> was used as the home directory.
</work/scratch/gu08vomo/thesis/coupling/128_1024_cpld> was used as the working directory.
Started at Tue Mar  3 00:06:32 2015
Results reported at Tue Mar  3 00:08:14 2015

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
#! /bin/sh

#BSUB -J coupling_cpld_scalability_openmpi

#BSUB -o /home/gu08vomo/thesis/coupling_128/scalability_openmpi_mpi1/cpld_0128.out.%J

#BSUB -n 0128
#BSUB -W 0:10
#BSUB -x

##BSUB -q test_mpi2

#BSUB -a openmpi

module load openmpi/intel/1.8.2
export PETSC_DIR=/home/gu08vomo/soft/petsc/3.5.2/build/arch-openmpi-opt-intel-hlr
export MYWORKDIR=/work/scratch/gu08vomo/thesis/coupling/128_1024_cpld/128_1024_0128/
export OUTPUTDIR=/home/gu08vomo/thesis/coupling
export PETSC_OPS="-options_file ../ops.gamg"

cat ../ops.gamg

echo "SCALABILITY MEASUREMENT ON MPI1 WITH OPENMPI AND FIELDSPLIT PRECONDITIONING"

echo "PETSC_DIR="$PETSC_DIR
echo "MYWORKDIR="$MYWORKDIR
cd $MYWORKDIR
mpirun -report-bindings -map-by core -bind-to core -n 0128 ./openmpi.caffa3d.cpld.lnx ${PETSC_OPS}



------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :               8314.24 sec.
    Max Memory :             9883 MB
    Average Memory :         3650.71 MB
    Total Requested Memory : -
    Delta Memory :           -
    (Delta: the difference between total requested memory and actual max usage.)
    Max Swap :               27578 MB

    Max Processes :          55
    Max Threads :            201

The output (if any) follows:

Modules: loading openmpi/intel/1.8.2
cat: ../ops.gamg: No such file or directory
SCALABILITY MEASUREMENT ON MPI1 WITH OPENMPI AND FIELDSPLIT PRECONDITIONING
PETSC_DIR=/home/gu08vomo/soft/petsc/3.5.2/build/arch-openmpi-opt-intel-hlr
MYWORKDIR=/work/scratch/gu08vomo/thesis/coupling/128_1024_cpld/128_1024_0128/
[hpa0678:13880] MCW rank 3 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0678:13880] MCW rank 4 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0678:13880] MCW rank 5 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0678:13880] MCW rank 6 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0678:13880] MCW rank 7 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0678:13880] MCW rank 8 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0678:13880] MCW rank 9 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0678:13880] MCW rank 10 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0678:13880] MCW rank 11 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0678:13880] MCW rank 12 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0678:13880] MCW rank 13 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0678:13880] MCW rank 14 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0678:13880] MCW rank 15 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0678:13880] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0678:13880] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0678:13880] MCW rank 2 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0611:03178] MCW rank 23 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0611:03178] MCW rank 24 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0611:03178] MCW rank 25 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0611:03178] MCW rank 26 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0611:03178] MCW rank 27 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0611:03178] MCW rank 28 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0611:03178] MCW rank 29 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0611:03178] MCW rank 30 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0611:03178] MCW rank 31 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0611:03178] MCW rank 16 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0611:03178] MCW rank 17 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0611:03178] MCW rank 18 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0611:03178] MCW rank 19 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0611:03178] MCW rank 20 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0611:03178] MCW rank 21 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0611:03178] MCW rank 22 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0616:32359] MCW rank 72 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0616:32359] MCW rank 73 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0616:32359] MCW rank 74 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0616:32359] MCW rank 75 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0616:32359] MCW rank 76 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0616:32359] MCW rank 77 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0616:32359] MCW rank 78 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0616:32359] MCW rank 79 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0616:32359] MCW rank 64 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0616:32359] MCW rank 65 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0616:32359] MCW rank 66 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0616:32359] MCW rank 67 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0616:32359] MCW rank 68 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0616:32359] MCW rank 69 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0616:32359] MCW rank 70 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0616:32359] MCW rank 71 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0665:23912] MCW rank 46 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0665:23912] MCW rank 47 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0665:23912] MCW rank 32 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0665:23912] MCW rank 33 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0665:23912] MCW rank 34 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0665:23912] MCW rank 35 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0665:23912] MCW rank 36 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0665:23912] MCW rank 37 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0665:23912] MCW rank 38 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0665:23912] MCW rank 39 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0665:23912] MCW rank 40 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0665:23912] MCW rank 41 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0665:23912] MCW rank 42 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0665:23912] MCW rank 43 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0665:23912] MCW rank 44 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0665:23912] MCW rank 45 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0559:03858] MCW rank 87 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0559:03858] MCW rank 88 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0559:03858] MCW rank 89 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0559:03858] MCW rank 90 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0559:03858] MCW rank 91 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0559:03858] MCW rank 92 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0559:03858] MCW rank 93 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0559:03858] MCW rank 94 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0559:03858] MCW rank 95 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0559:03858] MCW rank 80 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0559:03858] MCW rank 81 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0559:03858] MCW rank 82 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0559:03858] MCW rank 83 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0559:03858] MCW rank 84 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0559:03858] MCW rank 85 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0559:03858] MCW rank 86 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0649:21015] MCW rank 63 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0649:21015] MCW rank 48 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0649:21015] MCW rank 49 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0649:21015] MCW rank 50 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0649:21015] MCW rank 51 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0649:21015] MCW rank 52 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0649:21015] MCW rank 53 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0649:21015] MCW rank 54 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0649:21015] MCW rank 55 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0649:21015] MCW rank 56 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0649:21015] MCW rank 57 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0649:21015] MCW rank 58 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0649:21015] MCW rank 59 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0649:21015] MCW rank 60 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0649:21015] MCW rank 61 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0649:21015] MCW rank 62 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0577:03197] MCW rank 103 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0577:03197] MCW rank 104 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0577:03197] MCW rank 105 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0577:03197] MCW rank 106 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0577:03197] MCW rank 107 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0577:03197] MCW rank 108 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0577:03197] MCW rank 109 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0577:03197] MCW rank 110 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0577:03197] MCW rank 111 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0577:03197] MCW rank 96 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0577:03197] MCW rank 97 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0577:03197] MCW rank 98 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0577:03197] MCW rank 99 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0577:03197] MCW rank 100 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0577:03197] MCW rank 101 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0577:03197] MCW rank 102 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
[hpa0618:15697] MCW rank 119 bound to socket 0[core 7[hwt 0]]: [./././././././B][./././././././.]
[hpa0618:15697] MCW rank 120 bound to socket 1[core 8[hwt 0]]: [./././././././.][B/././././././.]
[hpa0618:15697] MCW rank 121 bound to socket 1[core 9[hwt 0]]: [./././././././.][./B/./././././.]
[hpa0618:15697] MCW rank 122 bound to socket 1[core 10[hwt 0]]: [./././././././.][././B/././././.]
[hpa0618:15697] MCW rank 123 bound to socket 1[core 11[hwt 0]]: [./././././././.][./././B/./././.]
[hpa0618:15697] MCW rank 124 bound to socket 1[core 12[hwt 0]]: [./././././././.][././././B/././.]
[hpa0618:15697] MCW rank 125 bound to socket 1[core 13[hwt 0]]: [./././././././.][./././././B/./.]
[hpa0618:15697] MCW rank 126 bound to socket 1[core 14[hwt 0]]: [./././././././.][././././././B/.]
[hpa0618:15697] MCW rank 127 bound to socket 1[core 15[hwt 0]]: [./././././././.][./././././././B]
[hpa0618:15697] MCW rank 112 bound to socket 0[core 0[hwt 0]]: [B/././././././.][./././././././.]
[hpa0618:15697] MCW rank 113 bound to socket 0[core 1[hwt 0]]: [./B/./././././.][./././././././.]
[hpa0618:15697] MCW rank 114 bound to socket 0[core 2[hwt 0]]: [././B/././././.][./././././././.]
[hpa0618:15697] MCW rank 115 bound to socket 0[core 3[hwt 0]]: [./././B/./././.][./././././././.]
[hpa0618:15697] MCW rank 116 bound to socket 0[core 4[hwt 0]]: [././././B/././.][./././././././.]
[hpa0618:15697] MCW rank 117 bound to socket 0[core 5[hwt 0]]: [./././././B/./.][./././././././.]
[hpa0618:15697] MCW rank 118 bound to socket 0[core 6[hwt 0]]: [././././././B/.][./././././././.]
  ENTER PROBLEM NAME (SIX CHARACTERS):  
 ****************************************************
 NAME OF PROBLEM SOLVED control
 
 ****************************************************
 ***************************************************
 CONTROL SETTINGS
 ***************************************************
 LREAD,LWRITE,LPOST,LTEST,LOUTS,LOUTE,LTIME,LGRAD
 F F T F F F F F
  IMON, JMON, KMON, MMON, RMON,  IPR,  JPR,  KPR,  MPR,NPCOR,NIGRAD 
     2     2     2     1     0     2     2     3     1     1     1
  SORMAX,     SLARGE,     ALFA
  0.1000E-07  0.1000E+31  0.9200E+00
 (URF(I),I=1,6)
  0.1000E+01  0.1000E+01  0.1000E+01  0.1000E+01  0.1000E+01  0.1000E+01
 (SOR(I),I=1,6)
  0.1000E+00  0.1000E+00  0.1000E+00  0.1000E+00  0.1000E+00  0.1000E+00
 (GDS(I),I=1,6) - BLENDING (CDS-UDS)
  0.1000E+01  0.1000E+01  0.1000E+01  0.1000E+01  0.7000E+00  0.0000E+00
 LSG
100000
 ***************************************************
 START COUPLED ALGORITHM
 ***************************************************
Linear solve converged due to CONVERGED_ATOL iterations 6
KSP Object:(coupledsolve_) 128 MPI processes
  type: gmres
    GMRES: restart=100, using Modified Gram-Schmidt Orthogonalization
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-90, absolute=1.10423, divergence=10000
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object:(coupledsolve_) 128 MPI processes
  type: gamg
    MG: type is MULTIPLICATIVE, levels=4 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (coupledsolve_mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_coarse_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (coupledsolve_mg_coarse_sub_)       1 MPI processes
        type: preonly
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (coupledsolve_mg_coarse_sub_)       1 MPI processes
        type: sor
          SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
        linear system matrix = precond matrix:
        Mat Object:         1 MPI processes
          type: seqaij
          rows=27, cols=27
          total: nonzeros=653, allocated nonzeros=653
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Mat Object:       128 MPI processes
        type: mpiaij
        rows=27, cols=27
        total: nonzeros=653, allocated nonzeros=653
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_1_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_1_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       128 MPI processes
        type: mpiaij
        rows=2918, cols=2918
        total: nonzeros=221560, allocated nonzeros=221560
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_2_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_2_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       128 MPI processes
        type: mpiaij
        rows=488741, cols=488741
        total: nonzeros=4.42626e+07, allocated nonzeros=4.42626e+07
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_3_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      has attached null space
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_3_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       128 MPI processes
        type: mpiaij
        rows=13271040, cols=13271040
        total: nonzeros=1.507e+08, allocated nonzeros=1.507e+08
        total number of mallocs used during MatSetValues calls =0
          has attached null space
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object:   128 MPI processes
    type: mpiaij
    rows=13271040, cols=13271040
    total: nonzeros=1.507e+08, allocated nonzeros=1.507e+08
    total number of mallocs used during MatSetValues calls =0
      has attached null space
      not using I-node (on process 0) routines
 0000001  0.1000E+01  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 9
 0000002  0.2440E+00  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 14
 0000003  0.4575E-01  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 19
 0000004  0.1901E-01  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 41
 0000005  0.4321E-02  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 30
 0000006  0.1885E-02  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 41
 0000007  0.4674E-03  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 20
 0000008  0.2057E-03  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 43
 0000009  0.5536E-04  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 37
 0000010  0.2371E-04  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 43
 0000011  0.7186E-05  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 36
 0000012  0.2985E-05  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 44
 0000013  0.1013E-05  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 31
 0000014  0.4192E-06  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 38
 0000015  0.1592E-06  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 25
 0000016  0.6564E-07  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 39
 0000017  0.2712E-07  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 36
 0000018  0.1140E-07  0.0000E+00
Linear solve converged due to CONVERGED_ATOL iterations 39
 0000019  0.4980E-08  0.0000E+00
TIME FOR CALCULATION:  0.5863E+02
 L2-NORM ERROR U    VELOCITY  2.803885678347621E-005
 L2-NORM ERROR V    VELOCITY  2.790913623092557E-005
 L2-NORM ERROR W    VELOCITY  2.917203293110774E-005
 L2-NORM ERROR ABS. VELOCITY  3.168713181612872E-005
 L2-NORM ERROR      PRESSURE  1.392940412005762E-003
      *** CALCULATION FINISHED - SEE RESULTS ***
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./openmpi.caffa3d.cpld.lnx on a arch-openmpi-opt-intel-hlr-ext named hpa0678 with 128 processors, by gu08vomo Tue Mar  3 00:08:13 2015
Using Petsc Release Version 3.5.3, Jan, 31, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           9.008e+01      1.00025   9.007e+01
Objects:              3.215e+03      1.00000   3.215e+03
Flops:                2.373e+10      1.02696   2.361e+10  3.022e+12
Flops/sec:            2.634e+08      1.02671   2.621e+08  3.355e+10
MPI Messages:         1.232e+05      6.18545   6.987e+04  8.943e+06
MPI Message Lengths:  9.556e+08      2.06082   1.237e+04  1.107e+11
MPI Reductions:       1.491e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 3.3561e+01  37.3%  3.9813e+07   0.0%  2.149e+05   2.4%  5.702e+02        4.6%  2.590e+02   1.7% 
 1:        CPLD_SOL: 5.6514e+01  62.7%  3.0223e+12 100.0%  8.728e+06  97.6%  1.180e+04       95.4%  1.465e+04  98.3% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

ThreadCommRunKer      81 1.0 2.2925e+00221.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecNorm                1 1.0 4.6450e-01 1.0 2.07e+05 1.0 0.0e+00 0.0e+00 1.0e+00  1  0  0  0  0   1 67  0  0  0    57
VecScale               1 1.0 9.2983e-05 1.5 1.04e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0 33  0  0  0 142725
VecSet               676 1.0 3.8981e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin      736 1.0 2.0966e-02 3.6 0.00e+00 0.0 1.8e+05 1.8e+04 0.0e+00  0  0  2  3  0   0  0 84 63  0     0
VecScatterEnd        736 1.0 1.3496e+01105.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  4  0  0  0  0  10  0  0  0  0     0
VecNormalize           1 1.0 9.4175e-05 1.5 1.04e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0 33  0  0  0 140918
MatAssemblyBegin      38 1.0 1.6660e-01 2.5 0.00e+00 0.0 2.6e+04 4.1e+04 7.6e+01  0  0  0  1  1   0  0 12 21 29     0
MatAssemblyEnd        38 1.0 1.4369e-01 1.1 0.00e+00 0.0 9.3e+02 1.2e+04 8.3e+01  0  0  0  0  1   0  0  0  0 32     0
MatZeroEntries        19 1.0 8.4610e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetGraph            19 1.0 1.5828e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFReduceBegin         19 1.0 1.5753e-01 1.7 0.00e+00 0.0 6.1e+03 1.2e+05 0.0e+00  0  0  0  1  0   0  0  3 15  0     0
SFReduceEnd           19 1.0 1.3356e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: CPLD_SOL

VecDot             10797 1.0 3.5414e+00 1.1 2.24e+09 1.0 0.0e+00 0.0e+00 1.1e+04  4  9  0  0 72   6  9  0  0 74 80921
VecMDot             2481 1.0 4.0228e+00 1.8 5.20e+08 1.0 0.0e+00 0.0e+00 2.5e+03  3  2  0  0 17   5  2  0  0 17 16548
VecNorm              680 1.0 1.2228e-01 1.4 1.37e+08 1.0 0.0e+00 0.0e+00 6.8e+02  0  1  0  0  5   0  1  0  0  5 142909
VecScale            4303 1.0 4.1210e-02 1.1 8.04e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 243285
VecCopy              689 1.0 2.5491e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              7391 1.0 1.2242e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            13296 1.0 1.3723e+00 1.1 2.76e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1 12  0  0  0   2 12  0  0  0 257131
VecAYPX             3660 1.0 8.1738e-01 1.2 2.55e+08 1.0 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0 39983
VecMAXPY            2503 1.0 4.0565e-01 1.2 6.45e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  3  0  0  0   1  3  0  0  0 203451
VecAssemblyBegin      23 1.0 1.4377e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 6.3e+01  0  0  0  0  0   0  0  0  0  0     0
VecAssemblyEnd        23 1.0 3.2425e-05 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      33 1.0 5.8870e-03 1.2 1.18e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 25716
VecScatterBegin    11664 1.0 5.3135e-01 2.3 0.00e+00 0.0 8.5e+06 1.1e+04 0.0e+00  0  0 95 86  0   1  0 97 90  0     0
VecScatterEnd      11664 1.0 2.9502e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   4  0  0  0  0     0
VecSetRandom           3 1.0 2.2552e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize         643 1.0 1.4068e-01 1.3 1.93e+08 1.0 0.0e+00 0.0e+00 6.4e+02  0  1  0  0  4   0  1  0  0  4 175859
MatMult             4318 1.0 1.3883e+01 1.1 7.40e+09 1.0 2.6e+06 2.9e+04 0.0e+00 15 31 29 69  0  24 31 30 72  0 67832
MatMultAdd          1830 1.0 2.2636e+00 1.7 6.18e+08 1.1 1.1e+06 2.0e+03 0.0e+00  2  3 13  2  0   3  3 13  2  0 34556
MatMultTranspose    1830 1.0 1.7695e+00 1.5 6.18e+08 1.1 1.1e+06 2.0e+03 0.0e+00  2  3 13  2  0   2  3 13  2  0 44205
MatSOR              5490 1.0 2.0926e+01 1.1 7.55e+09 1.0 3.6e+06 4.0e+03 0.0e+00 23 32 40 13  0  36 32 41 13  0 45947
MatConvert             3 1.0 1.4187e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatScale               9 1.0 1.7555e-02 1.2 4.09e+06 1.1 2.4e+03 1.3e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0 29542
MatResidual         1830 1.0 3.5397e+00 1.2 1.88e+09 1.1 1.5e+06 1.3e+04 0.0e+00  4  8 16 17  0   6  8 17 18  0 67274
MatAssemblyBegin     113 1.0 5.5142e-01 4.3 0.00e+00 0.0 5.5e+03 2.8e+03 1.6e+02  0  0  0  0  1   0  0  0  0  1     0
MatAssemblyEnd       113 1.0 1.4980e+00 1.0 0.00e+00 0.0 4.0e+04 1.5e+03 1.8e+02  2  0  0  0  1   3  0  0  0  1     0
MatGetRow         430720 1.0 4.4299e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             3 1.0 1.1389e-01 1.0 0.00e+00 0.0 9.1e+04 4.4e+03 4.3e+01  0  0  1  0  0   0  0  1  0  0     0
MatView                6 1.2 1.6303e+001777.9 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00  1  0  0  0  0   2  0  0  0  0     0
MatAXPY                3 1.0 3.0613e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             3 1.0 1.7952e-01 1.0 3.00e+06 1.1 1.5e+04 6.2e+03 4.8e+01  0  0  0  0  0   0  0  0  0  0  2120
MatMatMultSym          3 1.0 1.4599e-01 1.0 0.00e+00 0.0 1.3e+04 4.9e+03 4.2e+01  0  0  0  0  0   0  0  0  0  0     0
MatMatMultNum          3 1.0 3.3665e-02 1.0 3.00e+06 1.1 2.4e+03 1.3e+04 6.0e+00  0  0  0  0  0   0  0  0  0  0 11306
MatPtAP               57 1.0 5.3013e+00 1.0 8.50e+08 1.2 1.2e+05 6.1e+04 2.0e+02  6  4  1  7  1   9  4  1  7  1 20055
MatPtAPSymbolic        6 1.0 3.7654e-01 1.0 0.00e+00 0.0 2.8e+04 2.9e+04 4.2e+01  0  0  0  1  0   1  0  0  1  0     0
MatPtAPNumeric        57 1.0 4.9250e+00 1.0 8.50e+08 1.2 9.0e+04 7.1e+04 1.6e+02  5  4  1  6  1   9  4  1  6  1 21587
MatTrnMatMult          3 1.0 1.8678e+00 1.0 6.86e+07 1.3 1.9e+04 1.5e+05 5.7e+01  2  0  0  3  0   3  0  0  3  0  4509
MatTrnMatMultSym       3 1.0 1.1316e+00 1.0 0.00e+00 0.0 1.7e+04 6.0e+04 5.1e+01  1  0  0  1  0   2  0  0  1  0     0
MatTrnMatMultNum       3 1.0 7.3659e-01 1.0 6.86e+07 1.3 2.4e+03 7.7e+05 6.0e+00  1  0  0  2  0   1  0  0  2  0 11432
MatGetLocalMat        69 1.0 8.1363e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol         63 1.0 1.6193e-01 2.3 0.00e+00 0.0 6.5e+04 6.9e+04 0.0e+00  0  0  1  4  0   0  0  1  4  0     0
MatGetSymTrans        12 1.0 6.9423e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetGraph             3 1.0 3.2842e-03 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin          49 1.0 2.1371e-02 1.5 0.00e+00 0.0 9.1e+04 4.4e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
SFBcastEnd            49 1.0 3.3243e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog       621 1.0 4.4305e+00 1.1 4.50e+09 1.0 0.0e+00 0.0e+00 1.1e+04  5 19  0  0 73   7 19  0  0 74 130047
KSPSetUp             136 1.0 6.1779e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve              19 1.0 5.4701e+01 1.0 2.37e+10 1.0 8.7e+06 1.2e+04 1.5e+04 61100 97 95 98  97100100100100 55133
PCGAMGgraph_AGG        3 1.0 1.8440e-01 1.0 3.08e+06 1.1 7.2e+03 6.4e+03 4.2e+01  0  0  0  0  0   0  0  0  0  0  2117
PCGAMGcoarse_AGG       3 1.0 2.0266e+00 1.0 6.86e+07 1.3 1.3e+05 2.7e+04 1.5e+02  2  0  1  3  1   4  0  1  3  1  4155
PCGAMGProl_AGG         3 1.0 1.1422e-01 1.0 0.00e+00 0.0 1.9e+04 8.0e+03 7.2e+01  0  0  0  0  0   0  0  0  0  0     0
PCGAMGPOpt_AGG         3 1.0 3.6229e-01 1.0 6.91e+07 1.0 3.9e+04 1.0e+04 1.6e+02  0  0  0  0  1   1  0  0  0  1 24276
PCSetUp               38 1.0 8.7040e+00 1.0 9.91e+08 1.1 3.2e+05 3.6e+04 7.4e+02 10  4  4 10  5  15  4  4 11  5 14238
PCSetUpOnBlocks      610 1.0 4.7803e-04 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
PCApply              610 1.0 3.8941e+01 1.0 1.65e+10 1.0 8.1e+06 9.8e+03 2.4e+03 43 70 91 72 16  69 70 93 76 17 53976

--- Event Stage 2: Unknown

------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector   111            245     83646984     0
      Vector Scatter     5             10        10840     0
           Index Set    14             12       181536     0
   IS L to G Mapping     4              3      1045500     0
              Matrix     6             34     36248404     0
Star Forest Bipartite Graph    19             19        16568     0
       Krylov Solver     0              6       503176     0
      Preconditioner     0              6         6276     0

--- Event Stage 1: CPLD_SOL

              Vector  2806           2669     65491832     0
      Vector Scatter    27             21        22940     0
           Index Set    67             65       295500     0
              Matrix   106             78    133576640     0
      Matrix Coarsen     3              3         1932     0
   Matrix Null Space    19              0            0     0
Star Forest Bipartite Graph     3              3         2616     0
       Krylov Solver    10              4        91816     0
      Preconditioner    10              4         3792     0
         PetscRandom     3              3         1920     0
              Viewer     2              1          752     0

--- Event Stage 2: Unknown

========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 6.51836e-05
Average time for zero size MPI_Send(): 7.94418e-06
#PETSc Option Table entries:
-coupledsolve_ksp_converged_reason
-coupledsolve_ksp_gmres_modifiedgramschmidt
-coupledsolve_ksp_gmres_restart 100
-coupledsolve_ksp_norm_type unpreconditioned
-coupledsolve_ksp_type gmres
-coupledsolve_mg_coarse_sub_pc_type sor
-coupledsolve_mg_levels_ksp_rtol 1e-5
-coupledsolve_mg_levels_ksp_type richardson
-coupledsolve_pc_gamg_reuse_interpolation true
-coupledsolve_pc_type gamg
-log_summary
-on_error_abort
-options_left
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: PETSC_ARCH=arch-openmpi-opt-intel-hlr-ext PETSC_DIR=/home/gu08vomo/soft/petsc/3.5.3 -prefix=/home/gu08vomo/soft/petsc/3.5.3/build/arch-openmpi-opt-intel-hlr-ext --with-blas-lapack-dir=/shared/apps/intel/2015/composer_xe_2015/mkl/lib/intel64/ --with-mpi-dir=/shared/apps/openmpi/1.8.2_intel COPTFLAGS="-O3 -xHost" FOPTFLAGS="-O3 -xHost" CXXOPTFLAGS="-O3 -xHost" --with-debugging=0 --download-hypre --download-ml
-----------------------------------------
Libraries compiled on Sun Feb  1 16:09:22 2015 on hla0003 
Machine characteristics: Linux-3.0.101-0.40-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /home/gu08vomo/soft/petsc/3.5.3
Using PETSc arch: arch-openmpi-opt-intel-hlr-ext
-----------------------------------------

Using C compiler: /shared/apps/openmpi/1.8.2_intel/bin/mpicc  -fPIC -wd1572 -O3 -xHost  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /shared/apps/openmpi/1.8.2_intel/bin/mpif90  -fPIC -O3 -xHost   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/include -I/home/gu08vomo/soft/petsc/3.5.3/include -I/home/gu08vomo/soft/petsc/3.5.3/include -I/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/include -I/shared/apps/openmpi/1.8.2_intel/include
-----------------------------------------

Using C linker: /shared/apps/openmpi/1.8.2_intel/bin/mpicc
Using Fortran linker: /shared/apps/openmpi/1.8.2_intel/bin/mpif90
Using libraries: -Wl,-rpath,/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/lib -L/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/lib -lpetsc -Wl,-rpath,/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/lib -L/home/gu08vomo/soft/petsc/3.5.3/arch-openmpi-opt-intel-hlr-ext/lib -lHYPRE -Wl,-rpath,/shared/apps/openmpi/1.8.2_intel/lib -L/shared/apps/openmpi/1.8.2_intel/lib -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -L/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib64 -L/shared/apps/gcc/4.8.3/lib64 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib -L/shared/apps/gcc/4.8.3/lib -lmpi_cxx -lml -lmpi_cxx -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lpthread -lssl -lcrypto -lmpi_usempi_ignore_tkr -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -Wl,-rpath,/shared/apps/openmpi/1.8.2_intel/lib -L/shared/apps/openmpi/1.8.2_intel/lib -lmpi -Wl,-rpath,/shared/apps/openmpi/1.8.2_intel/lib -L/shared/apps/openmpi/1.8.2_intel/lib -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -L/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib64 -L/shared/apps/gcc/4.8.3/lib64 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib64 -L/shared/apps/gcc/4.8.3/lib64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -L/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib -L/shared/apps/gcc/4.8.3/lib -Wl,-rpath,/shared/apps/openmpi/1.8.2_intel/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -Wl,-rpath,/shared/apps/openmpi/1.8.2_intel/lib -L/shared/apps/openmpi/1.8.2_intel/lib -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -L/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib64 -L/shared/apps/gcc/4.8.3/lib64 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib64 -L/shared/apps/gcc/4.8.3/lib64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/ipp/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -L/shared/apps/intel/2015/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -L/shared/apps/intel/2015/composer_xe_2015.0.090/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/shared/apps/gcc/4.8.3/lib -L/shared/apps/gcc/4.8.3/lib -ldl  
-----------------------------------------

#PETSc Option Table entries:
-coupledsolve_ksp_converged_reason
-coupledsolve_ksp_gmres_modifiedgramschmidt
-coupledsolve_ksp_gmres_restart 100
-coupledsolve_ksp_norm_type unpreconditioned
-coupledsolve_ksp_type gmres
-coupledsolve_mg_coarse_sub_pc_type sor
-coupledsolve_mg_levels_ksp_rtol 1e-5
-coupledsolve_mg_levels_ksp_type richardson
-coupledsolve_pc_gamg_reuse_interpolation true
-coupledsolve_pc_type gamg
-log_summary
-on_error_abort
-options_left
#End of PETSc Option Table entries
There are no unused options.

From bsmith at mcs.anl.gov  Mon Mar  2 19:43:12 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 2 Mar 2015 19:43:12 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
	systems for discretized coupled multiphysics problems
In-Reply-To: <1425343174.27237.60.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
Message-ID: <9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>


  Do you really want tolerances:  relative=1e-90, absolute=1.10423, divergence=10000? That is an absolute tolerance of 1.1? Normally that would be huge.

  You can provide your matrix with a block size that GAMG will use with MatSetBlockSize().

  I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that it is using SOR for 27 points.

 So you must have provided a null space since it printed "has attached null space"

  Barry



> On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> 
> On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
>> Fabian Gabel <gabel.fabian at gmail.com> writes:
>> 
>>> Dear PETSc Team,
>>> 
>>> I came across the following paragraph in your publication "Composable
>>> Linear Solvers for Multiphysics" (2012):
>>> 
>>> "Rather than splitting the matrix into large blocks and
>>> forming a preconditioner from solvers (for example, multi-
>>> grid) on each block, one can perform multigrid on the entire
>>> system, basing the smoother on solves coming from the tiny
>>> blocks coupling the degrees of freedom at a single point (or
>>> small number of points). This approach is also handled in
>>> PETSc, but we will not elaborate on it here."
>>> 
>>> How would I use a multigrid preconditioner (GAMG) 
>> 
>> The heuristics in GAMG are not appropriate for indefinite/saddle-point
>> systems such as arise from Navier-Stokes.  You can use geometric
>> multigrid and use the fieldsplit techniques described in the paper as a
>> smoother, for example.
> 
> I sadly don't have a solid background on multigrid methods, but as
> mentioned in a previous thread
> 
> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
> 
> AMG has apparently been used (successfully?) for fully-coupled
> finite-volume discretizations of Navier-Stokes:
> 
> http://dx.doi.org/10.1080/10407790.2014.894448
> http://dx.doi.org/10.1016/j.jcp.2008.08.027
> 
> I was hoping to achieve something similar with the right configuration
> of the PETSc preconditioners. So far I have only been using GAMG in a
> straightforward manner, without providing any details on the structure
> of the linear system. I attached the output of a test run with GAMG.
> 
>> 
>>> from PETSc on linear systems of the form (after reordering the
>>> variables):
>>> 
>>> [A_uu   0     0   A_up  A_uT]
>>> [0    A_vv    0   A_vp  A_vT]
>>> [0      0   A_ww  A_up  A_wT]
>>> [A_pu A_pv  A_pw  A_pp   0  ]
>>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
>>> 
>>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
>>> directly from a FVM discretization of the incompressible Navier-Stokes
>>> equations and the temperature equation. The fifth row and column are
>>> optional, depending on the method I choose to couple the temperature.
>>> The Matrix is stored as one AIJ Matrix.
>>> 
>>> Regards,
>>> Fabian Gabel
> 
> <cpld_0128.out.578677>


From gabel.fabian at gmail.com  Mon Mar  2 21:19:28 2015
From: gabel.fabian at gmail.com (Fabian Gabel)
Date: Tue, 03 Mar 2015 04:19:28 +0100
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
References: <1425338778.27237.18.camel@gmail.com>
	<87pp8r9dw2.fsf@jedbrown.org> <1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
Message-ID: <1425352768.27237.88.camel@gmail.com>

On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
>   Do you really want tolerances:  relative=1e-90, absolute=1.10423, divergence=10000? That is an absolute tolerance of 1.1? Normally that would be huge.

I started using atol as convergence criterion with -ksp_norm_type
unpreconditioned. The value of atol gets updated every outer iteration.

>   You can provide your matrix with a block size that GAMG will use with MatSetBlockSize().

I think something went wrong. Setting the block size to 4 and solving
for (u,v,w,p) the convergence degraded significantly. I attached the
results for a smaller test case that shows the increase of the number of
needed inner iterations when setting the block size via
MatSetBlockSize().

> 
>   I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that it is using SOR for 27 points.
> 
>  So you must have provided a null space since it printed "has attached null space"

The system has indeed a one dimensional null space (from the pressure
equation with Neumann boundary conditions). But now that you mentioned
it: It seems that the outer GMRES doesn't notice that the matrix has an
attached nullspace. Replacing

	CALL MatSetNullSpace(CMAT,NULLSP,IERR)

with

	CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)

solves this. What is wrong with using MatSetNullSpace?


Fabian


> 
>   Barry
> 
> 
> 
> > On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> > 
> > On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
> >> Fabian Gabel <gabel.fabian at gmail.com> writes:
> >> 
> >>> Dear PETSc Team,
> >>> 
> >>> I came across the following paragraph in your publication "Composable
> >>> Linear Solvers for Multiphysics" (2012):
> >>> 
> >>> "Rather than splitting the matrix into large blocks and
> >>> forming a preconditioner from solvers (for example, multi-
> >>> grid) on each block, one can perform multigrid on the entire
> >>> system, basing the smoother on solves coming from the tiny
> >>> blocks coupling the degrees of freedom at a single point (or
> >>> small number of points). This approach is also handled in
> >>> PETSc, but we will not elaborate on it here."
> >>> 
> >>> How would I use a multigrid preconditioner (GAMG) 
> >> 
> >> The heuristics in GAMG are not appropriate for indefinite/saddle-point
> >> systems such as arise from Navier-Stokes.  You can use geometric
> >> multigrid and use the fieldsplit techniques described in the paper as a
> >> smoother, for example.
> > 
> > I sadly don't have a solid background on multigrid methods, but as
> > mentioned in a previous thread
> > 
> > http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
> > 
> > AMG has apparently been used (successfully?) for fully-coupled
> > finite-volume discretizations of Navier-Stokes:
> > 
> > http://dx.doi.org/10.1080/10407790.2014.894448
> > http://dx.doi.org/10.1016/j.jcp.2008.08.027
> > 
> > I was hoping to achieve something similar with the right configuration
> > of the PETSc preconditioners. So far I have only been using GAMG in a
> > straightforward manner, without providing any details on the structure
> > of the linear system. I attached the output of a test run with GAMG.
> > 
> >> 
> >>> from PETSc on linear systems of the form (after reordering the
> >>> variables):
> >>> 
> >>> [A_uu   0     0   A_up  A_uT]
> >>> [0    A_vv    0   A_vp  A_vT]
> >>> [0      0   A_ww  A_up  A_wT]
> >>> [A_pu A_pv  A_pw  A_pp   0  ]
> >>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> >>> 
> >>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
> >>> directly from a FVM discretization of the incompressible Navier-Stokes
> >>> equations and the temperature equation. The fifth row and column are
> >>> optional, depending on the method I choose to couple the temperature.
> >>> The Matrix is stored as one AIJ Matrix.
> >>> 
> >>> Regards,
> >>> Fabian Gabel
> > 
> > <cpld_0128.out.578677>
> 

-------------- next part --------------
  ENTER PROBLEM NAME (SIX CHARACTERS):  
 ****************************************************
 NAME OF PROBLEM SOLVED control
 
 ****************************************************
 ***************************************************
 CONTROL SETTINGS
 ***************************************************
 LREAD,LWRITE,LPOST,LTEST,LOUTS,LOUTE,LTIME,LGRAD
 F F F F F F F F
  IMON, JMON, KMON, MMON, RMON,  IPR,  JPR,  KPR,  MPR,NPCOR,NIGRAD 
     2     2     2     1     0     2     2     3     1     1     1
  SORMAX,     SLARGE,     ALFA
  0.1000E-07  0.1000E+31  0.9200E+00
 (URF(I),I=1,6)
  0.9000E+00  0.9000E+00  0.9000E+00  0.1000E+00  0.1000E+01  0.1000E+01
 (SOR(I),I=1,6)
  0.1000E+00  0.1000E+00  0.1000E+00  0.1000E-01  0.1000E+00  0.1000E+00
 (GDS(I),I=1,6) - BLENDING (CDS-UDS)
  0.1000E+01  0.1000E+01  0.1000E+01  0.1600E+01  0.1000E+01  0.0000E+00
 LSG
 10000
 ***************************************************
 START COUPLED ALGORITHM
 ***************************************************
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 7.680650991441e+01 
  1 KSP Residual norm 4.185899808477e+01 
  2 KSP Residual norm 2.202068303753e+01 
  3 KSP Residual norm 1.741802539838e+01 
  4 KSP Residual norm 1.364396531738e+01 
  5 KSP Residual norm 1.172273211505e+01 
  6 KSP Residual norm 1.162524589201e+01 
  7 KSP Residual norm 1.011907279997e+01 
  8 KSP Residual norm 8.685677152271e+00 
  9 KSP Residual norm 7.396125459146e+00 
 10 KSP Residual norm 7.296977130121e+00 
 11 KSP Residual norm 6.959200481951e+00 
 12 KSP Residual norm 6.905446814136e+00 
 13 KSP Residual norm 6.880720024552e+00 
 14 KSP Residual norm 6.824350801593e+00 
 15 KSP Residual norm 6.824016557799e+00 
 16 KSP Residual norm 4.724619270852e+00 
 17 KSP Residual norm 2.668642871524e+00 
 18 KSP Residual norm 2.191273302044e+00 
 19 KSP Residual norm 1.944296054954e+00 
 20 KSP Residual norm 1.900543170284e+00 
 21 KSP Residual norm 1.564163873693e+00 
 22 KSP Residual norm 1.506139846488e+00 
 23 KSP Residual norm 1.505810088310e+00 
 24 KSP Residual norm 1.503963939770e+00 
 25 KSP Residual norm 1.423291747222e+00 
 26 KSP Residual norm 1.396589263932e+00 
 27 KSP Residual norm 1.382496701568e+00 
 28 KSP Residual norm 1.111467646867e+00 
KSP Object:(coupledsolve_) 1 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-90, absolute=1.20082, divergence=10000
  right preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object:(coupledsolve_) 1 MPI processes
  type: gamg
    MG: type is MULTIPLICATIVE, levels=3 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (coupledsolve_mg_coarse_)     1 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_coarse_)     1 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 1
        Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object:        (coupledsolve_mg_coarse_sub_)         1 MPI processes
          type: preonly
          maximum iterations=1, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (coupledsolve_mg_coarse_sub_)         1 MPI processes
          type: sor
            SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
          linear system matrix = precond matrix:
          Mat Object:           1 MPI processes
            type: seqaij
            rows=224, cols=224, bs=4
            total: nonzeros=24768, allocated nonzeros=24768
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 56 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=224, cols=224, bs=4
        total: nonzeros=24768, allocated nonzeros=24768
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 56 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_1_)     1 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_1_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=12260, cols=12260, bs=4
        total: nonzeros=1.42581e+06, allocated nonzeros=1.42581e+06
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 3065 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_2_)     1 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_2_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=256000, cols=256000, bs=4
        total: nonzeros=2.35725e+06, allocated nonzeros=2.35725e+06
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object:   1 MPI processes
    type: seqaij
    rows=256000, cols=256000, bs=4
    total: nonzeros=2.35725e+06, allocated nonzeros=2.35725e+06
    total number of mallocs used during MatSetValues calls =0
      not using I-node routines
 0000001  0.1000E+01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.627622580920e+00 
  1 KSP Residual norm 2.618623510400e+00 
  2 KSP Residual norm 2.595681404299e+00 
  3 KSP Residual norm 2.573101727243e+00 
  4 KSP Residual norm 2.471252938050e+00 
  5 KSP Residual norm 2.428028929045e+00 
  6 KSP Residual norm 2.229168066194e+00 
  7 KSP Residual norm 2.085615414410e+00 
  8 KSP Residual norm 2.076282297913e+00 
  9 KSP Residual norm 2.069139125163e+00 
 10 KSP Residual norm 2.058700104333e+00 
 11 KSP Residual norm 2.058567796498e+00 
 12 KSP Residual norm 1.920870596572e+00 
 13 KSP Residual norm 1.777523400957e+00 
 14 KSP Residual norm 1.674384000336e+00 
 15 KSP Residual norm 1.642534139399e+00 
 16 KSP Residual norm 1.449777567903e+00 
 17 KSP Residual norm 1.440931159108e+00 
 18 KSP Residual norm 1.439213065379e+00 
 19 KSP Residual norm 1.382404265336e+00 
 20 KSP Residual norm 1.364215277267e+00 
 21 KSP Residual norm 1.362114300120e+00 
 22 KSP Residual norm 1.339978879706e+00 
 23 KSP Residual norm 1.322639467952e+00 
 24 KSP Residual norm 1.321384304678e+00 
 25 KSP Residual norm 1.321380759642e+00 
 26 KSP Residual norm 1.183186737875e+00 
 27 KSP Residual norm 8.659328686803e-01 
 28 KSP Residual norm 8.388085654581e-01 
 29 KSP Residual norm 8.346618418315e-01 
 30 KSP Residual norm 7.932132874906e-01 
 31 KSP Residual norm 7.931916049632e-01 
 32 KSP Residual norm 7.930070151834e-01 
 33 KSP Residual norm 7.927622711664e-01 
 34 KSP Residual norm 7.924352125863e-01 
 35 KSP Residual norm 7.923331320223e-01 
 36 KSP Residual norm 7.920591938247e-01 
 37 KSP Residual norm 7.914444071308e-01 
 38 KSP Residual norm 7.910824991274e-01 
 39 KSP Residual norm 7.910587420876e-01 
 40 KSP Residual norm 7.908788515020e-01 
 41 KSP Residual norm 7.895603769794e-01 
 42 KSP Residual norm 7.895589633648e-01 
 43 KSP Residual norm 7.891864300420e-01 
 44 KSP Residual norm 7.891326664165e-01 
 45 KSP Residual norm 7.888105721332e-01 
 46 KSP Residual norm 7.886174398811e-01 
 47 KSP Residual norm 7.869613760972e-01 
 48 KSP Residual norm 7.869597845056e-01 
 49 KSP Residual norm 7.861600492870e-01 
 50 KSP Residual norm 7.855646154910e-01 
 51 KSP Residual norm 7.854192275274e-01 
 52 KSP Residual norm 7.845302897412e-01 
 53 KSP Residual norm 7.845131211745e-01 
 54 KSP Residual norm 7.800888294949e-01 
 55 KSP Residual norm 7.782372196955e-01 
 56 KSP Residual norm 7.739281372664e-01 
 57 KSP Residual norm 7.703291682892e-01 
 58 KSP Residual norm 7.594343557276e-01 
 59 KSP Residual norm 7.428628814220e-01 
 60 KSP Residual norm 6.796108855423e-01 
 61 KSP Residual norm 6.792621836949e-01 
 62 KSP Residual norm 6.792599058243e-01 
 63 KSP Residual norm 6.790825225897e-01 
 64 KSP Residual norm 6.788952262666e-01 
 65 KSP Residual norm 6.775543713534e-01 
 66 KSP Residual norm 6.762003547942e-01 
 67 KSP Residual norm 6.750660083508e-01 
 68 KSP Residual norm 6.749056179696e-01 
 69 KSP Residual norm 6.745788595495e-01 
 70 KSP Residual norm 6.744831909936e-01 
 71 KSP Residual norm 6.744095168640e-01 
 72 KSP Residual norm 6.739619993802e-01 
 73 KSP Residual norm 6.733835789845e-01 
 74 KSP Residual norm 6.721515742494e-01 
 75 KSP Residual norm 6.717972280897e-01 
 76 KSP Residual norm 6.717926885501e-01 
 77 KSP Residual norm 6.711395282193e-01 
 78 KSP Residual norm 6.710189393745e-01 
 79 KSP Residual norm 6.708920353331e-01 
 80 KSP Residual norm 6.705065744993e-01 
 81 KSP Residual norm 6.702492254576e-01 
 82 KSP Residual norm 6.654066904612e-01 
 83 KSP Residual norm 6.633284243713e-01 
 84 KSP Residual norm 6.588735035312e-01 
 85 KSP Residual norm 6.588203269869e-01 
 86 KSP Residual norm 6.588021175950e-01 
 87 KSP Residual norm 6.582004015484e-01 
 88 KSP Residual norm 6.194837793643e-01 
 89 KSP Residual norm 5.961955972874e-01 
 90 KSP Residual norm 5.705567260811e-01 
 91 KSP Residual norm 5.705515967545e-01 
 92 KSP Residual norm 5.704003341423e-01 
 93 KSP Residual norm 5.703233684974e-01 
 94 KSP Residual norm 5.698144671527e-01 
 95 KSP Residual norm 5.698140906127e-01 
 96 KSP Residual norm 5.697148973128e-01 
 97 KSP Residual norm 5.685830320447e-01 
 98 KSP Residual norm 5.685420277423e-01 
 99 KSP Residual norm 5.683463895401e-01 
100 KSP Residual norm 5.683244725734e-01 
101 KSP Residual norm 5.681581335505e-01 
102 KSP Residual norm 5.676435098591e-01 
103 KSP Residual norm 5.664466011033e-01 
104 KSP Residual norm 5.654075348607e-01 
105 KSP Residual norm 5.653642207258e-01 
106 KSP Residual norm 5.639595704613e-01 
107 KSP Residual norm 5.619972750669e-01 
108 KSP Residual norm 5.605485908660e-01 
109 KSP Residual norm 5.604783703868e-01 
110 KSP Residual norm 5.576368973879e-01 
111 KSP Residual norm 5.576368717003e-01 
112 KSP Residual norm 5.571870008692e-01 
113 KSP Residual norm 5.546719813159e-01 
114 KSP Residual norm 5.545657244427e-01 
115 KSP Residual norm 5.537810648153e-01 
116 KSP Residual norm 5.533802839194e-01 
117 KSP Residual norm 5.520193889595e-01 
118 KSP Residual norm 5.093820108693e-01 
119 KSP Residual norm 4.531399469657e-01 
120 KSP Residual norm 2.957173921159e-01 
121 KSP Residual norm 2.956784310673e-01 
122 KSP Residual norm 2.950937537726e-01 
123 KSP Residual norm 2.867092781331e-01 
124 KSP Residual norm 2.863531544699e-01 
125 KSP Residual norm 2.847392979451e-01 
126 KSP Residual norm 2.833809327431e-01 
127 KSP Residual norm 2.829158304896e-01 
128 KSP Residual norm 2.822111361138e-01 
129 KSP Residual norm 2.785559895759e-01 
130 KSP Residual norm 2.595237994249e-01 
 0000002  0.2188E+00  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 5.804789475348e-01 
  1 KSP Residual norm 5.799778769779e-01 
  2 KSP Residual norm 5.641496919751e-01 
  3 KSP Residual norm 5.617959304693e-01 
  4 KSP Residual norm 5.614011665870e-01 
  5 KSP Residual norm 5.569451426114e-01 
  6 KSP Residual norm 5.556608778653e-01 
  7 KSP Residual norm 5.467421792576e-01 
  8 KSP Residual norm 5.467265438841e-01 
  9 KSP Residual norm 5.353090266132e-01 
 10 KSP Residual norm 4.746622832951e-01 
 11 KSP Residual norm 4.702818488973e-01 
 12 KSP Residual norm 4.406372118449e-01 
 13 KSP Residual norm 4.330662669124e-01 
 14 KSP Residual norm 4.288057798951e-01 
 15 KSP Residual norm 4.046274693057e-01 
 16 KSP Residual norm 4.046230837327e-01 
 17 KSP Residual norm 4.042511516735e-01 
 18 KSP Residual norm 4.037663498063e-01 
 19 KSP Residual norm 3.931216729466e-01 
 20 KSP Residual norm 3.921031432605e-01 
 21 KSP Residual norm 3.429800353658e-01 
 22 KSP Residual norm 2.979221673663e-01 
 23 KSP Residual norm 2.681645180400e-01 
 24 KSP Residual norm 2.495132152627e-01 
 25 KSP Residual norm 2.279615743396e-01 
 26 KSP Residual norm 2.186269030034e-01 
 27 KSP Residual norm 2.186147575889e-01 
 28 KSP Residual norm 2.184196067179e-01 
 29 KSP Residual norm 2.126159051784e-01 
 30 KSP Residual norm 2.069760657158e-01 
 31 KSP Residual norm 2.069746584589e-01 
 32 KSP Residual norm 2.069731656236e-01 
 33 KSP Residual norm 2.069707108122e-01 
 34 KSP Residual norm 2.069685614175e-01 
 35 KSP Residual norm 2.069685610728e-01 
 36 KSP Residual norm 2.069679499377e-01 
 37 KSP Residual norm 2.069667560718e-01 
 38 KSP Residual norm 2.069658387974e-01 
 39 KSP Residual norm 2.069656414990e-01 
 40 KSP Residual norm 2.069630365184e-01 
 41 KSP Residual norm 2.069630254407e-01 
 42 KSP Residual norm 2.069600524211e-01 
 43 KSP Residual norm 2.069528577985e-01 
 44 KSP Residual norm 2.069375150385e-01 
 45 KSP Residual norm 2.069214325540e-01 
 46 KSP Residual norm 2.069193513908e-01 
 47 KSP Residual norm 2.069027188948e-01 
 48 KSP Residual norm 2.069026823825e-01 
 49 KSP Residual norm 2.068953523635e-01 
 50 KSP Residual norm 2.068777919687e-01 
 51 KSP Residual norm 2.067685327874e-01 
 52 KSP Residual norm 2.067523582887e-01 
 53 KSP Residual norm 2.067099161030e-01 
 54 KSP Residual norm 2.065528777162e-01 
 55 KSP Residual norm 2.064349053912e-01 
 56 KSP Residual norm 2.062439881285e-01 
 57 KSP Residual norm 2.048486269207e-01 
 58 KSP Residual norm 2.032421705358e-01 
 59 KSP Residual norm 1.975396841907e-01 
 60 KSP Residual norm 1.795975675884e-01 
 61 KSP Residual norm 1.795934380605e-01 
 62 KSP Residual norm 1.795171646958e-01 
 63 KSP Residual norm 1.793585200727e-01 
 64 KSP Residual norm 1.791359937468e-01 
 65 KSP Residual norm 1.787155890634e-01 
 66 KSP Residual norm 1.787083647740e-01 
 67 KSP Residual norm 1.785741702253e-01 
 68 KSP Residual norm 1.782753198644e-01 
 69 KSP Residual norm 1.777962163575e-01 
 70 KSP Residual norm 1.775966385004e-01 
 71 KSP Residual norm 1.775381320053e-01 
 72 KSP Residual norm 1.775289200005e-01 
 73 KSP Residual norm 1.771227389228e-01 
 74 KSP Residual norm 1.769384491898e-01 
 75 KSP Residual norm 1.769231925523e-01 
 76 KSP Residual norm 1.767348805780e-01 
 77 KSP Residual norm 1.767316532455e-01 
 78 KSP Residual norm 1.766472311583e-01 
 79 KSP Residual norm 1.766439185473e-01 
 80 KSP Residual norm 1.760015558603e-01 
 81 KSP Residual norm 1.759613105921e-01 
 82 KSP Residual norm 1.759281035337e-01 
 83 KSP Residual norm 1.758447163326e-01 
 84 KSP Residual norm 1.753046343083e-01 
 85 KSP Residual norm 1.748441698476e-01 
 86 KSP Residual norm 1.748374509920e-01 
 87 KSP Residual norm 1.745938060587e-01 
 88 KSP Residual norm 1.735307552311e-01 
 89 KSP Residual norm 1.730258903974e-01 
 90 KSP Residual norm 1.697175326546e-01 
 91 KSP Residual norm 1.697173566959e-01 
 92 KSP Residual norm 1.697093289476e-01 
 93 KSP Residual norm 1.696982044568e-01 
 94 KSP Residual norm 1.696879274337e-01 
 95 KSP Residual norm 1.696855069974e-01 
 96 KSP Residual norm 1.696835982074e-01 
 97 KSP Residual norm 1.696710777240e-01 
 98 KSP Residual norm 1.696622260091e-01 
 99 KSP Residual norm 1.695969278837e-01 
100 KSP Residual norm 1.695836616942e-01 
101 KSP Residual norm 1.695834190045e-01 
102 KSP Residual norm 1.695812895531e-01 
103 KSP Residual norm 1.694567934722e-01 
104 KSP Residual norm 1.694195781071e-01 
105 KSP Residual norm 1.693701486949e-01 
106 KSP Residual norm 1.693482350903e-01 
107 KSP Residual norm 1.693142066308e-01 
108 KSP Residual norm 1.693139366048e-01 
109 KSP Residual norm 1.692893155871e-01 
110 KSP Residual norm 1.688223587231e-01 
111 KSP Residual norm 1.688197233297e-01 
112 KSP Residual norm 1.688163962887e-01 
113 KSP Residual norm 1.688049236283e-01 
114 KSP Residual norm 1.684781906985e-01 
115 KSP Residual norm 1.683539285586e-01 
116 KSP Residual norm 1.682149256972e-01 
117 KSP Residual norm 1.681685812286e-01 
118 KSP Residual norm 1.659367994851e-01 
119 KSP Residual norm 1.634765294983e-01 
120 KSP Residual norm 1.425866729058e-01 
121 KSP Residual norm 1.425841891153e-01 
122 KSP Residual norm 1.425662768676e-01 
123 KSP Residual norm 1.424811258758e-01 
124 KSP Residual norm 1.424663772468e-01 
125 KSP Residual norm 1.424392146345e-01 
126 KSP Residual norm 1.423428247695e-01 
127 KSP Residual norm 1.423379715786e-01 
128 KSP Residual norm 1.423161428637e-01 
129 KSP Residual norm 1.422826009846e-01 
130 KSP Residual norm 1.421362715567e-01 
131 KSP Residual norm 1.421297151769e-01 
132 KSP Residual norm 1.418970488464e-01 
133 KSP Residual norm 1.405045151205e-01 
134 KSP Residual norm 1.404234190366e-01 
135 KSP Residual norm 1.403729297668e-01 
136 KSP Residual norm 1.397587138297e-01 
137 KSP Residual norm 1.393043951726e-01 
138 KSP Residual norm 1.392643702290e-01 
139 KSP Residual norm 1.390947717975e-01 
140 KSP Residual norm 1.389847569971e-01 
141 KSP Residual norm 1.386182745937e-01 
142 KSP Residual norm 1.375593051658e-01 
143 KSP Residual norm 1.288200479354e-01 
144 KSP Residual norm 1.245208235838e-01 
145 KSP Residual norm 1.228085303126e-01 
146 KSP Residual norm 1.224380746235e-01 
147 KSP Residual norm 1.203742807462e-01 
148 KSP Residual norm 1.149199456902e-01 
149 KSP Residual norm 8.452247950761e-02 
150 KSP Residual norm 5.233620210445e-02 
 0000003  0.4834E-01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.195661259895e-01 
  1 KSP Residual norm 2.058279904671e-01 
  2 KSP Residual norm 2.046045866307e-01 
  3 KSP Residual norm 2.033797944751e-01 
  4 KSP Residual norm 2.012090188960e-01 
  5 KSP Residual norm 2.010479779691e-01 
  6 KSP Residual norm 1.953230724132e-01 
  7 KSP Residual norm 1.555726024661e-01 
  8 KSP Residual norm 1.502318724920e-01 
  9 KSP Residual norm 1.489692702783e-01 
 10 KSP Residual norm 1.460165310535e-01 
 11 KSP Residual norm 1.446268056783e-01 
 12 KSP Residual norm 1.407909493702e-01 
 13 KSP Residual norm 1.158026701992e-01 
 14 KSP Residual norm 1.157234255633e-01 
 15 KSP Residual norm 1.056728059699e-01 
 16 KSP Residual norm 9.476360181621e-02 
 17 KSP Residual norm 9.353972455501e-02 
 18 KSP Residual norm 9.353942709547e-02 
 19 KSP Residual norm 8.224859685736e-02 
 20 KSP Residual norm 8.200143406325e-02 
 21 KSP Residual norm 7.772887684678e-02 
 22 KSP Residual norm 7.700008764377e-02 
 23 KSP Residual norm 7.450258745835e-02 
 24 KSP Residual norm 6.528939332618e-02 
 25 KSP Residual norm 6.128315999136e-02 
 26 KSP Residual norm 6.015626596536e-02 
 27 KSP Residual norm 5.779857199939e-02 
 28 KSP Residual norm 5.688458134959e-02 
 29 KSP Residual norm 4.930918193192e-02 
 30 KSP Residual norm 4.413319071243e-02 
 31 KSP Residual norm 4.412509475939e-02 
 32 KSP Residual norm 4.393555086585e-02 
 33 KSP Residual norm 4.391898171710e-02 
 34 KSP Residual norm 4.391354959236e-02 
 35 KSP Residual norm 4.384162021232e-02 
 36 KSP Residual norm 4.325311444780e-02 
 37 KSP Residual norm 4.288365140046e-02 
 38 KSP Residual norm 4.287209988977e-02 
 39 KSP Residual norm 4.287206142933e-02 
 40 KSP Residual norm 4.286686909881e-02 
 41 KSP Residual norm 4.260594081690e-02 
 42 KSP Residual norm 4.183743120137e-02 
 43 KSP Residual norm 4.125657765887e-02 
 44 KSP Residual norm 4.083515451508e-02 
 45 KSP Residual norm 4.025039016469e-02 
 46 KSP Residual norm 4.014739619114e-02 
 47 KSP Residual norm 3.982092444641e-02 
 48 KSP Residual norm 3.958395232305e-02 
 49 KSP Residual norm 3.952931641207e-02 
 50 KSP Residual norm 3.912847460622e-02 
 51 KSP Residual norm 3.871947928625e-02 
 52 KSP Residual norm 3.867946020933e-02 
 53 KSP Residual norm 3.798425553173e-02 
 54 KSP Residual norm 3.748560407702e-02 
 55 KSP Residual norm 3.724127504646e-02 
 56 KSP Residual norm 3.440431176428e-02 
 57 KSP Residual norm 3.232527531585e-02 
 58 KSP Residual norm 2.559831144651e-02 
 59 KSP Residual norm 2.228223272302e-02 
 60 KSP Residual norm 1.914862438455e-02 
 0000004  0.1828E-01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 5.525635412447e-02 
  1 KSP Residual norm 5.523779213543e-02 
  2 KSP Residual norm 5.489254753372e-02 
  3 KSP Residual norm 5.449195079288e-02 
  4 KSP Residual norm 5.351045474716e-02 
  5 KSP Residual norm 5.350454984099e-02 
  6 KSP Residual norm 5.348879006758e-02 
  7 KSP Residual norm 5.131804406749e-02 
  8 KSP Residual norm 5.130634818964e-02 
  9 KSP Residual norm 5.087978009574e-02 
 10 KSP Residual norm 5.011790353306e-02 
 11 KSP Residual norm 4.533803591536e-02 
 12 KSP Residual norm 3.911373694053e-02 
 13 KSP Residual norm 3.846373237137e-02 
 14 KSP Residual norm 3.600651859095e-02 
 15 KSP Residual norm 3.367842601739e-02 
 16 KSP Residual norm 3.331417578347e-02 
 17 KSP Residual norm 3.232106678629e-02 
 18 KSP Residual norm 2.987776906332e-02 
 19 KSP Residual norm 2.987650978692e-02 
 20 KSP Residual norm 2.986461389971e-02 
 21 KSP Residual norm 2.986427312470e-02 
 22 KSP Residual norm 2.927086107956e-02 
 23 KSP Residual norm 2.915793209415e-02 
 24 KSP Residual norm 2.895908986430e-02 
 25 KSP Residual norm 2.482129709912e-02 
 26 KSP Residual norm 2.215601003618e-02 
 27 KSP Residual norm 1.852509765167e-02 
 28 KSP Residual norm 1.463909533226e-02 
 29 KSP Residual norm 1.182311494447e-02 
 30 KSP Residual norm 1.104520437591e-02 
 31 KSP Residual norm 1.104387877212e-02 
 32 KSP Residual norm 1.104147164556e-02 
 33 KSP Residual norm 1.103409135441e-02 
 34 KSP Residual norm 1.102286965812e-02 
 35 KSP Residual norm 1.102285643829e-02 
 36 KSP Residual norm 1.101641432221e-02 
 37 KSP Residual norm 1.101051494244e-02 
 38 KSP Residual norm 1.100036449340e-02 
 39 KSP Residual norm 1.099997781089e-02 
 40 KSP Residual norm 1.099778407905e-02 
 41 KSP Residual norm 1.099328811279e-02 
 42 KSP Residual norm 1.099279011358e-02 
 43 KSP Residual norm 1.094729805901e-02 
 44 KSP Residual norm 1.092560978245e-02 
 45 KSP Residual norm 1.085465993129e-02 
 46 KSP Residual norm 1.070530862736e-02 
 47 KSP Residual norm 1.069191452546e-02 
 48 KSP Residual norm 1.021260176837e-02 
 49 KSP Residual norm 1.016462435052e-02 
 50 KSP Residual norm 1.014117683302e-02 
 51 KSP Residual norm 1.009109769783e-02 
 52 KSP Residual norm 1.009014269115e-02 
 53 KSP Residual norm 1.008243622191e-02 
 54 KSP Residual norm 9.828263449640e-03 
 55 KSP Residual norm 9.644468912478e-03 
 56 KSP Residual norm 9.471903670996e-03 
 57 KSP Residual norm 9.424275132518e-03 
 58 KSP Residual norm 9.171514955945e-03 
 59 KSP Residual norm 9.129070363302e-03 
 60 KSP Residual norm 8.920084490772e-03 
 61 KSP Residual norm 8.920052881620e-03 
 62 KSP Residual norm 8.919017185449e-03 
 63 KSP Residual norm 8.917675663928e-03 
 64 KSP Residual norm 8.916816143903e-03 
 65 KSP Residual norm 8.916306367790e-03 
 66 KSP Residual norm 8.915411548718e-03 
 67 KSP Residual norm 8.914357204518e-03 
 68 KSP Residual norm 8.914325973013e-03 
 69 KSP Residual norm 8.913827907595e-03 
 70 KSP Residual norm 8.913823013719e-03 
 71 KSP Residual norm 8.912842938749e-03 
 72 KSP Residual norm 8.912759077344e-03 
 73 KSP Residual norm 8.908363036649e-03 
 74 KSP Residual norm 8.907594393040e-03 
 75 KSP Residual norm 8.903524069895e-03 
 76 KSP Residual norm 8.903262538003e-03 
 77 KSP Residual norm 8.901039140851e-03 
 78 KSP Residual norm 8.899996065837e-03 
 79 KSP Residual norm 8.899969808238e-03 
 80 KSP Residual norm 8.897816058103e-03 
 81 KSP Residual norm 8.891773515959e-03 
 82 KSP Residual norm 8.889867086082e-03 
 83 KSP Residual norm 8.889531100993e-03 
 84 KSP Residual norm 8.845142763665e-03 
 85 KSP Residual norm 8.812084759770e-03 
 86 KSP Residual norm 8.783032596521e-03 
 87 KSP Residual norm 8.707250471595e-03 
 88 KSP Residual norm 8.702180431461e-03 
 89 KSP Residual norm 8.635571960351e-03 
 90 KSP Residual norm 7.518785765931e-03 
 91 KSP Residual norm 7.517335145455e-03 
 92 KSP Residual norm 7.514172655556e-03 
 93 KSP Residual norm 7.506088977377e-03 
 94 KSP Residual norm 7.482746241802e-03 
 95 KSP Residual norm 7.475422608205e-03 
 96 KSP Residual norm 7.468838236535e-03 
 97 KSP Residual norm 7.455967440625e-03 
 98 KSP Residual norm 7.451468344334e-03 
 99 KSP Residual norm 7.444672451339e-03 
100 KSP Residual norm 7.427661463675e-03 
101 KSP Residual norm 7.427320924670e-03 
102 KSP Residual norm 7.373404855544e-03 
103 KSP Residual norm 7.367870254877e-03 
104 KSP Residual norm 7.365859749509e-03 
105 KSP Residual norm 7.244783986628e-03 
106 KSP Residual norm 7.242768509285e-03 
107 KSP Residual norm 7.242168100896e-03 
108 KSP Residual norm 7.209856849080e-03 
109 KSP Residual norm 7.193612184307e-03 
110 KSP Residual norm 7.192680079161e-03 
111 KSP Residual norm 7.191745723199e-03 
112 KSP Residual norm 7.132789915733e-03 
113 KSP Residual norm 7.102962868675e-03 
114 KSP Residual norm 7.093393180326e-03 
115 KSP Residual norm 7.050086753347e-03 
116 KSP Residual norm 7.005778938685e-03 
117 KSP Residual norm 6.923125763422e-03 
118 KSP Residual norm 6.845042283113e-03 
119 KSP Residual norm 6.782618558184e-03 
120 KSP Residual norm 6.725143356837e-03 
121 KSP Residual norm 6.725132622133e-03 
122 KSP Residual norm 6.724847388119e-03 
123 KSP Residual norm 6.724591741844e-03 
124 KSP Residual norm 6.724521076337e-03 
125 KSP Residual norm 6.724112824504e-03 
126 KSP Residual norm 6.724017992818e-03 
127 KSP Residual norm 6.723682776762e-03 
128 KSP Residual norm 6.722114858496e-03 
129 KSP Residual norm 6.720184071033e-03 
130 KSP Residual norm 6.719812160500e-03 
131 KSP Residual norm 6.719677445143e-03 
132 KSP Residual norm 6.719664963186e-03 
133 KSP Residual norm 6.715718760541e-03 
134 KSP Residual norm 6.712603576437e-03 
135 KSP Residual norm 6.712442649119e-03 
136 KSP Residual norm 6.709544343175e-03 
137 KSP Residual norm 6.708458187204e-03 
138 KSP Residual norm 6.708430822385e-03 
139 KSP Residual norm 6.708391457716e-03 
140 KSP Residual norm 6.708390792235e-03 
141 KSP Residual norm 6.707190601581e-03 
142 KSP Residual norm 6.703224755348e-03 
143 KSP Residual norm 6.703109804513e-03 
144 KSP Residual norm 6.702996838906e-03 
145 KSP Residual norm 6.699760767242e-03 
146 KSP Residual norm 6.692176493433e-03 
147 KSP Residual norm 6.662860649181e-03 
148 KSP Residual norm 6.628546635694e-03 
149 KSP Residual norm 6.530604292402e-03 
150 KSP Residual norm 6.161085281341e-03 
151 KSP Residual norm 6.160895992501e-03 
152 KSP Residual norm 6.160070254099e-03 
153 KSP Residual norm 6.159595725727e-03 
154 KSP Residual norm 6.157084924466e-03 
155 KSP Residual norm 6.156718079604e-03 
156 KSP Residual norm 6.155458765950e-03 
157 KSP Residual norm 6.155005765622e-03 
158 KSP Residual norm 6.154685720229e-03 
159 KSP Residual norm 6.149195573032e-03 
160 KSP Residual norm 6.147134547485e-03 
161 KSP Residual norm 6.147120074653e-03 
162 KSP Residual norm 6.146674768574e-03 
163 KSP Residual norm 6.122234963131e-03 
164 KSP Residual norm 6.110786892227e-03 
165 KSP Residual norm 6.109736500537e-03 
166 KSP Residual norm 6.109732765258e-03 
167 KSP Residual norm 6.108919009230e-03 
168 KSP Residual norm 6.099997144503e-03 
169 KSP Residual norm 6.089822903290e-03 
170 KSP Residual norm 6.089040042350e-03 
171 KSP Residual norm 6.086209099035e-03 
172 KSP Residual norm 6.079717411475e-03 
173 KSP Residual norm 6.068765961475e-03 
174 KSP Residual norm 6.062916731513e-03 
175 KSP Residual norm 6.054259223203e-03 
176 KSP Residual norm 6.053401430239e-03 
177 KSP Residual norm 6.053138364097e-03 
178 KSP Residual norm 6.003231859964e-03 
179 KSP Residual norm 5.867241012253e-03 
180 KSP Residual norm 5.222815812010e-03 
 0000005  0.4602E-02  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.181258502939e-02 
  1 KSP Residual norm 2.139160683993e-02 
  2 KSP Residual norm 2.111132578997e-02 
  3 KSP Residual norm 2.005041182942e-02 
  4 KSP Residual norm 1.752877255518e-02 
  5 KSP Residual norm 1.732245712837e-02 
  6 KSP Residual norm 1.579101376259e-02 
  7 KSP Residual norm 1.244042616193e-02 
  8 KSP Residual norm 1.239980884602e-02 
  9 KSP Residual norm 1.210096906811e-02 
 10 KSP Residual norm 1.184653943124e-02 
 11 KSP Residual norm 1.176320516373e-02 
 12 KSP Residual norm 1.170782473700e-02 
 13 KSP Residual norm 1.045298473249e-02 
 14 KSP Residual norm 1.039361338460e-02 
 15 KSP Residual norm 9.032884384939e-03 
 16 KSP Residual norm 8.447636755442e-03 
 17 KSP Residual norm 8.391101877814e-03 
 18 KSP Residual norm 8.283099686897e-03 
-------------- next part --------------
 ***************************************************
 CONTROL SETTINGS
 ***************************************************
 LREAD,LWRITE,LPOST,LTEST,LOUTS,LOUTE,LTIME,LGRAD
 F F F F F F F F
  IMON, JMON, KMON, MMON, RMON,  IPR,  JPR,  KPR,  MPR,NPCOR,NIGRAD 
     2     2     2     1     0     2     2     3     1     1     1
  SORMAX,     SLARGE,     ALFA
  0.1000E-07  0.1000E+31  0.9200E+00
 (URF(I),I=1,6)
  0.9000E+00  0.9000E+00  0.9000E+00  0.1000E+00  0.1000E+01  0.1000E+01
 (SOR(I),I=1,6)
  0.1000E+00  0.1000E+00  0.1000E+00  0.1000E-01  0.1000E+00  0.1000E+00
 (GDS(I),I=1,6) - BLENDING (CDS-UDS)
  0.1000E+01  0.1000E+01  0.1000E+01  0.1600E+01  0.1000E+01  0.0000E+00
 LSG
 10000
 ***************************************************
 START COUPLED ALGORITHM
 ***************************************************
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 7.680650991441e+01 
  1 KSP Residual norm 2.315112892713e+00 
  2 KSP Residual norm 7.080140933135e-01 
KSP Object:(coupledsolve_) 1 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-90, absolute=1.20082, divergence=10000
  right preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object:(coupledsolve_) 1 MPI processes
  type: gamg
    MG: type is MULTIPLICATIVE, levels=3 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (coupledsolve_mg_coarse_)     1 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_coarse_)     1 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 1
        Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object:        (coupledsolve_mg_coarse_sub_)         1 MPI processes
          type: preonly
          maximum iterations=1, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (coupledsolve_mg_coarse_sub_)         1 MPI processes
          type: sor
            SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
          linear system matrix = precond matrix:
          Mat Object:           1 MPI processes
            type: seqaij
            rows=56, cols=56
            total: nonzeros=1868, allocated nonzeros=1868
            total number of mallocs used during MatSetValues calls =0
              not using I-node routines
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=56, cols=56
        total: nonzeros=1868, allocated nonzeros=1868
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_1_)     1 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_1_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=7783, cols=7783
        total: nonzeros=627755, allocated nonzeros=627755
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (coupledsolve_mg_levels_2_)     1 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (coupledsolve_mg_levels_2_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=256000, cols=256000
        total: nonzeros=2.35725e+06, allocated nonzeros=2.35725e+06
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object:   1 MPI processes
    type: seqaij
    rows=256000, cols=256000
    total: nonzeros=2.35725e+06, allocated nonzeros=2.35725e+06
    total number of mallocs used during MatSetValues calls =0
      not using I-node routines
 0000001  0.1000E+01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.537524950128e+00 
  1 KSP Residual norm 4.800084858727e-01 
  2 KSP Residual norm 1.620821084959e-01 
 0000002  0.2113E+00  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 4.325900644323e-01 
  1 KSP Residual norm 1.317083025844e-01 
  2 KSP Residual norm 6.484567311390e-02 
  3 KSP Residual norm 3.810677528327e-02 
 0000003  0.3602E-01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 1.744152280000e-01 
  1 KSP Residual norm 3.901664369762e-02 
  2 KSP Residual norm 2.308489607380e-02 
  3 KSP Residual norm 1.282790886819e-02 
 0000004  0.1452E-01  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 3.792432575935e-02 
  1 KSP Residual norm 1.446893562868e-02 
  2 KSP Residual norm 1.170414412325e-02 
  3 KSP Residual norm 7.627017584906e-03 
  4 KSP Residual norm 4.004268384306e-03 
  5 KSP Residual norm 2.951612023714e-03 
 0000005  0.3158E-02  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 1.750225257756e-02 
  1 KSP Residual norm 3.287521188931e-03 
  2 KSP Residual norm 2.090054722070e-03 
  3 KSP Residual norm 1.313025221380e-03 
 0000006  0.1458E-02  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 4.304131293348e-03 
  1 KSP Residual norm 1.460779398297e-03 
  2 KSP Residual norm 1.094883111455e-03 
  3 KSP Residual norm 6.711726958470e-04 
  4 KSP Residual norm 4.235393183993e-04 
 0000007  0.3584E-03  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 1.934132941245e-03 
  1 KSP Residual norm 4.920885681151e-04 
  2 KSP Residual norm 3.608655529099e-04 
  3 KSP Residual norm 2.041208433378e-04 
  4 KSP Residual norm 1.516104297901e-04 
 0000008  0.1611E-03  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 5.204045108286e-04 
  1 KSP Residual norm 1.799866485727e-04 
  2 KSP Residual norm 1.337137496960e-04 
  3 KSP Residual norm 7.406048243690e-05 
  4 KSP Residual norm 5.741608689253e-05 
  5 KSP Residual norm 4.886315540918e-05 
 0000009  0.4334E-04  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.341009109387e-04 
  1 KSP Residual norm 4.097862927298e-05 
  2 KSP Residual norm 3.101636775409e-05 
  3 KSP Residual norm 2.303123486841e-05 
 0000010  0.1950E-04  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 6.980043269989e-05 
  1 KSP Residual norm 2.154354863441e-05 
  2 KSP Residual norm 1.515768659879e-05 
  3 KSP Residual norm 1.080676666607e-05 
  4 KSP Residual norm 9.930405731118e-06 
  5 KSP Residual norm 7.757848714880e-06 
  6 KSP Residual norm 4.165862290699e-06 
 0000011  0.5813E-05  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 2.922247023288e-05 
  1 KSP Residual norm 5.326163951339e-06 
  2 KSP Residual norm 2.915029700326e-06 
 0000012  0.2434E-05  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 9.202780292901e-06 
  1 KSP Residual norm 2.566838156317e-06 
  2 KSP Residual norm 1.748670689745e-06 
  3 KSP Residual norm 1.252703127678e-06 
  4 KSP Residual norm 1.129468798804e-06 
  5 KSP Residual norm 9.130304621369e-07 
 0000013  0.7664E-06  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 3.907612884416e-06 
  1 KSP Residual norm 8.982421498294e-07 
  2 KSP Residual norm 6.645851329385e-07 
  3 KSP Residual norm 5.497724351328e-07 
  4 KSP Residual norm 3.683792060416e-07 
 0000014  0.3254E-06  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 1.289190862676e-06 
  1 KSP Residual norm 4.434982001168e-07 
  2 KSP Residual norm 3.485444131419e-07 
  3 KSP Residual norm 2.346302677179e-07 
  4 KSP Residual norm 1.639403498152e-07 
  5 KSP Residual norm 1.359191571158e-07 
  6 KSP Residual norm 9.621188205348e-08 
 0000015  0.1074E-06  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 5.408086146100e-07 
  1 KSP Residual norm 9.223981776498e-08 
  2 KSP Residual norm 6.104650170618e-08 
  3 KSP Residual norm 4.981085059159e-08 
 0000016  0.4504E-07  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 1.935234341417e-07 
  1 KSP Residual norm 4.005921857358e-08 
  2 KSP Residual norm 3.259196479070e-08 
  3 KSP Residual norm 2.649660480243e-08 
  4 KSP Residual norm 2.335356743530e-08 
  5 KSP Residual norm 1.711730147849e-08 
 0000017  0.1612E-07  0.0000E+00
  Residual norms for coupledsolve_ solve.
  0 KSP Residual norm 8.100595588300e-08 
  1 KSP Residual norm 1.486341216211e-08 
  2 KSP Residual norm 1.322167068956e-08 
  3 KSP Residual norm 1.133936064325e-08 
  4 KSP Residual norm 6.626715597309e-09 
 0000018  0.6746E-08  0.0000E+00
TIME FOR CALCULATION:  0.7428E+02
 L2-NORM ERROR U    VELOCITY   4.6062628669300923E-004
 L2-NORM ERROR V    VELOCITY   4.4771789082304948E-004
 L2-NORM ERROR W    VELOCITY   4.7430150765984163E-004
 L2-NORM ERROR ABS. VELOCITY   5.2255204191562464E-004
 L2-NORM ERROR      PRESSURE   1.0967291493601079E-002
      *** CALCULATION FINISHED - SEE RESULTS ***

From hus003 at ucsd.edu  Tue Mar  3 01:22:08 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Tue, 3 Mar 2015 07:22:08 +0000
Subject: [petsc-users] A very strange bug about using multiple KSPs
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>

I set up 2 KSPs through KSPSetComputeOperators, with separate compute matrix functions.



    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

Within each computeMatrix function, I define the matrix, and I also print something to the screen. So


PetscErrorCode ComputeMatrix1(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"1\n");CHKERRQ(ierr);

    ......

}


PetscErrorCode ComputeMatrix2(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"2\n");CHKERRQ(ierr);

    ......

}


Now if I compile and execute the code, it only print 2, completely ignoring 1, which is not expected, since I have setup ksp1 to use ComputeMatrix1.

Does anyone know what might be wrong? Thank you. I have attached the short code here in this email as well.

Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/c72cedd5/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/c72cedd5/attachment.c>

From dave.mayhem23 at gmail.com  Tue Mar  3 03:11:33 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 3 Mar 2015 10:11:33 +0100
Subject: [petsc-users] A very strange bug about using multiple KSPs
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <CAJ98EDoesDFzxYQuU-GvnSNcn=f_NR6=b8_3kCasRQ4Qe06EQg@mail.gmail.com>

Try moving your print statement in ComputeMatrix2 to be after
PetscFunctionBeginUser;

You should really use a debugger to see if these functions are being called
or not - don't use print statements.

On Tuesday, 3 March 2015, Sun, Hui <hus003 at ucsd.edu> wrote:

>  I set up 2 KSPs through KSPSetComputeOperators, with separate compute
> matrix functions.
>
>
>
>     ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);
>
>     ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);
>
>  Within each computeMatrix function, I define the matrix, and I also
> print something to the screen. So
>
>   PetscErrorCode ComputeMatrix1(KSP ksp, Mat J,Mat jac, void *ctx)
>
> {
>
>     ......
>
>     ierr    = PetscPrintf(PETSC_COMM_WORLD,"1\n");CHKERRQ(ierr);
>
>     ......
>
> }
>
>
>  PetscErrorCode ComputeMatrix2(KSP ksp, Mat J,Mat jac, void *ctx)
>
> {
>
>     ......
>
>     ierr    = PetscPrintf(PETSC_COMM_WORLD,"2\n");CHKERRQ(ierr);
>
>     ......
>
> }
>
>
>  Now if I compile and execute the code, it only print 2, completely
> ignoring 1, which is not expected, since I have setup ksp1 to use
> ComputeMatrix1.
>
>  Does anyone know what might be wrong? Thank you. I have attached the
> short code here in this email as well.
>
>  Hui
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/01883ebc/attachment.html>

From gtheler at cites-gss.com  Tue Mar  3 06:57:28 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Tue, 3 Mar 2015 12:57:28 +0000
Subject: [petsc-users] dumping matrices structure into bitmap files
Message-ID: <1425387407.2523.2.camel@cites-gss.com>

Hi All

So far when I want to study the structure of my matrices I do something like

    PetscViewerDrawOpen(PETSC_COMM_WORLD,PETSC_NULL, "R", 100, 100, size, size, &viewer);
    MatView(R, viewer);


Is there a way to directly dump the image into a bitmap file in order to increase the scriptability of my codes?
Screen-to-png methods are accepted.


--
Germ?n Theler :: CTO Eng & IT

CITES ? Centro de Innovaci?n Tecnol?gica Empresarial y Social S.A.
Direcci?n General Sancor Seguros
Grupo Sancor Seguros
tel +54 3493 ?428 500 ? Int.: 3374
gtheler at cites-gss.com<mailto:ccipolatti at cites-gss.com>
www.cites-gss.com<http://www.cites-gss.com> - www.gruposancorseguros.com<http://www.gruposancorseguros.com>




________________________________
Imprima este mensaje s?lo si es absolutamente necesario.
Para imprimir, en lo posible utilice el papel de ambos lados.
El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.



************AVISO DE CONFIDENCIALIDAD************

El Grupo Sancor Seguros comunica que:

Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.

Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/6b12d6e0/attachment.html>

From knepley at gmail.com  Tue Mar  3 07:14:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 3 Mar 2015 07:14:01 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <1425352768.27237.88.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
	<1425352768.27237.88.camel@gmail.com>
Message-ID: <CAMYG4Gk1uVfz4pdNraEqa_OOn6zwF-miybTc-vbEWWYmDjRcfw@mail.gmail.com>

On Mon, Mar 2, 2015 at 9:19 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:

> On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
> >   Do you really want tolerances:  relative=1e-90, absolute=1.10423,
> divergence=10000? That is an absolute tolerance of 1.1? Normally that would
> be huge.
>
> I started using atol as convergence criterion with -ksp_norm_type
> unpreconditioned. The value of atol gets updated every outer iteration.
>
> >   You can provide your matrix with a block size that GAMG will use with
> MatSetBlockSize().
>
> I think something went wrong. Setting the block size to 4 and solving
> for (u,v,w,p) the convergence degraded significantly. I attached the
> results for a smaller test case that shows the increase of the number of
> needed inner iterations when setting the block size via
> MatSetBlockSize().
>
> >
> >   I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that
> it is using SOR for 27 points.
> >
> >  So you must have provided a null space since it printed "has attached
> null space"
>
> The system has indeed a one dimensional null space (from the pressure
> equation with Neumann boundary conditions). But now that you mentioned
> it: It seems that the outer GMRES doesn't notice that the matrix has an
> attached nullspace. Replacing
>
>         CALL MatSetNullSpace(CMAT,NULLSP,IERR)
>
> with
>
>         CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)
>
> solves this. What is wrong with using MatSetNullSpace?
>

That matrix must not be set as the system matrix for the KSP.

   Matt


> Fabian
>
>
> >
> >   Barry
> >
> >
> >
> > > On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com>
> wrote:
> > >
> > > On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
> > >> Fabian Gabel <gabel.fabian at gmail.com> writes:
> > >>
> > >>> Dear PETSc Team,
> > >>>
> > >>> I came across the following paragraph in your publication "Composable
> > >>> Linear Solvers for Multiphysics" (2012):
> > >>>
> > >>> "Rather than splitting the matrix into large blocks and
> > >>> forming a preconditioner from solvers (for example, multi-
> > >>> grid) on each block, one can perform multigrid on the entire
> > >>> system, basing the smoother on solves coming from the tiny
> > >>> blocks coupling the degrees of freedom at a single point (or
> > >>> small number of points). This approach is also handled in
> > >>> PETSc, but we will not elaborate on it here."
> > >>>
> > >>> How would I use a multigrid preconditioner (GAMG)
> > >>
> > >> The heuristics in GAMG are not appropriate for indefinite/saddle-point
> > >> systems such as arise from Navier-Stokes.  You can use geometric
> > >> multigrid and use the fieldsplit techniques described in the paper as
> a
> > >> smoother, for example.
> > >
> > > I sadly don't have a solid background on multigrid methods, but as
> > > mentioned in a previous thread
> > >
> > >
> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
> > >
> > > AMG has apparently been used (successfully?) for fully-coupled
> > > finite-volume discretizations of Navier-Stokes:
> > >
> > > http://dx.doi.org/10.1080/10407790.2014.894448
> > > http://dx.doi.org/10.1016/j.jcp.2008.08.027
> > >
> > > I was hoping to achieve something similar with the right configuration
> > > of the PETSc preconditioners. So far I have only been using GAMG in a
> > > straightforward manner, without providing any details on the structure
> > > of the linear system. I attached the output of a test run with GAMG.
> > >
> > >>
> > >>> from PETSc on linear systems of the form (after reordering the
> > >>> variables):
> > >>>
> > >>> [A_uu   0     0   A_up  A_uT]
> > >>> [0    A_vv    0   A_vp  A_vT]
> > >>> [0      0   A_ww  A_up  A_wT]
> > >>> [A_pu A_pv  A_pw  A_pp   0  ]
> > >>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> > >>>
> > >>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T},
> results
> > >>> directly from a FVM discretization of the incompressible
> Navier-Stokes
> > >>> equations and the temperature equation. The fifth row and column are
> > >>> optional, depending on the method I choose to couple the temperature.
> > >>> The Matrix is stored as one AIJ Matrix.
> > >>>
> > >>> Regards,
> > >>> Fabian Gabel
> > >
> > > <cpld_0128.out.578677>
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/f6c22273/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Mar  3 07:15:19 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Mar 2015 07:15:19 -0600
Subject: [petsc-users] dumping matrices structure into bitmap files
In-Reply-To: <1425387407.2523.2.camel@cites-gss.com>
References: <1425387407.2523.2.camel@cites-gss.com>
Message-ID: <8B829E09-2A27-48D3-AC24-74D0BEA36B90@mcs.anl.gov>


PetscDrawSetSave


> On Mar 3, 2015, at 6:57 AM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> Hi All
> 
> So far when I want to study the structure of my matrices I do something like
> 
>     PetscViewerDrawOpen(PETSC_COMM_WORLD,PETSC_NULL, "R", 100, 100, size, size, &viewer);
>     MatView(R, viewer);
> 
> 
> Is there a way to directly dump the image into a bitmap file in order to increase the scriptability of my codes?
> Screen-to-png methods are accepted.
> 
> 
> -- 
> Germ?n Theler :: CTO Eng & IT
> 
> CITES ? Centro de Innovaci?n Tecnol?gica Empresarial y Social S.A.
> Direcci?n General Sancor Seguros
> Grupo Sancor Seguros
> tel +54 3493 ?428 500 ? Int.: 3374
> gtheler at cites-gss.com
> www.cites-gss.com - www.gruposancorseguros.com
> 
> 
> 
> Imprima este mensaje s?lo si es absolutamente necesario.
> Para imprimir, en lo posible utilice el papel de ambos lados.
> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> 
> 
> ************AVISO DE CONFIDENCIALIDAD************
> 
> El Grupo Sancor Seguros comunica que:
> 
> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> 
> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.
> 


From grischa.jacobs at email.de  Tue Mar  3 09:49:45 2015
From: grischa.jacobs at email.de (Grischa Jacobs)
Date: Tue, 3 Mar 2015 16:49:45 +0100
Subject: [petsc-users] VTK Output / no file
Message-ID: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/ff630a50/attachment.html>

From hus003 at ucsd.edu  Tue Mar  3 10:15:23 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Tue, 3 Mar 2015 16:15:23 +0000
Subject: [petsc-users] A very strange bug about using multiple KSPs
In-Reply-To: <CAJ98EDoesDFzxYQuU-GvnSNcn=f_NR6=b8_3kCasRQ4Qe06EQg@mail.gmail.com>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<CAJ98EDoesDFzxYQuU-GvnSNcn=f_NR6=b8_3kCasRQ4Qe06EQg@mail.gmail.com>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E56@XMAIL-MBX-BH1.AD.UCSD.EDU>

Dave, thank you for your suggestions. I don't have a debugger, and I think without debugger we can still figure out if a function is called or not.

Putting the print commend before or after PetscFunctionBeginUser doesn't really matter. I just tried to put it after PetscFunctionBeginUser, but something doesn't print just doesn't print.

There are other ways to see that computeMatrix1 is not called. For example, I just change b1 to a vector of 1's, and b2 to a vector of 0.5's, and ComputeMatrix1 to give diagonal matrix with entries 1, and  ComputeMatrix2 to give diagonal matrix with entries 2.

Then I output the matrices and the vectors into a file, and visualize them in MATLAB. It does show me that the ksp1 and ksp2 both use ComputeMatrix2; while the ksp1 uses the rhs b1, and the ksp2 uses the rhs b2. This is not expected because I've set ksp1 to use ComputeMatrix1 before:


    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);


Now if I comment out the lines


    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

and


    ierr = KSPSolve(ksp2,b2,x2);CHKERRQ(ierr);


Then ksp1 start to use ComputeMatrix1, as can be visualized from both the output file and the terminal print. So it shows me that there are some crosstalk between ksp1 and ksp2. If I set

    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

ksp1 uses ComputeMatrix1. But if I set


    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

ksp1 uses ComputeMatrix2.



________________________________
From: Dave May [dave.mayhem23 at gmail.com]
Sent: Tuesday, March 03, 2015 1:11 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] A very strange bug about using multiple KSPs

Try moving your print statement in ComputeMatrix2 to be after PetscFunctionBeginUser;

You should really use a debugger to see if these functions are being called or not - don't use print statements.

On Tuesday, 3 March 2015, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
I set up 2 KSPs through KSPSetComputeOperators, with separate compute matrix functions.



    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

Within each computeMatrix function, I define the matrix, and I also print something to the screen. So


PetscErrorCode ComputeMatrix1(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"1\n");CHKERRQ(ierr);

    ......

}


PetscErrorCode ComputeMatrix2(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"2\n");CHKERRQ(ierr);

    ......

}


Now if I compile and execute the code, it only print 2, completely ignoring 1, which is not expected, since I have setup ksp1 to use ComputeMatrix1.

Does anyone know what might be wrong? Thank you. I have attached the short code here in this email as well.

Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/b16df11e/attachment.html>

From knepley at gmail.com  Tue Mar  3 10:26:49 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 3 Mar 2015 10:26:49 -0600
Subject: [petsc-users] A very strange bug about using multiple KSPs
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E56@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<CAJ98EDoesDFzxYQuU-GvnSNcn=f_NR6=b8_3kCasRQ4Qe06EQg@mail.gmail.com>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9E56@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <CAMYG4Gkv8FzMRHiy7EEY+0C62w6E0jtjJ9miy-Jh1Xyd_oQdOg@mail.gmail.com>

On Tue, Mar 3, 2015 at 10:15 AM, Sun, Hui <hus003 at ucsd.edu> wrote:

>  Dave, thank you for your suggestions. I don't have a debugger, and I
> think without debugger we can still figure out if a function is called or
> not.
>
>  Putting the print commend before or after PetscFunctionBeginUser doesn't
> really matter. I just tried to put it after PetscFunctionBeginUser, but
> something doesn't print just doesn't print.
>

The ComputeMatrix function is stored in the DM, or more precisely in the
DMKSP object. Since both KSPs have the
same DM, they have the same function.

I believe you can DMClone(da, &da2), and then this will work. Please tell
me if it does not.

  Thanks,

    Matt


> There are other ways to see that computeMatrix1 is not called. For
> example, I just change b1 to a vector of 1's, and b2 to a vector of 0.5's,
> and ComputeMatrix1 to give diagonal matrix with entries 1, and  ComputeMatrix2
> to give diagonal matrix with entries 2.
>
>  Then I output the matrices and the vectors into a file, and visualize
> them in MATLAB. It does show me that the ksp1 and ksp2 both use ComputeMatrix2;
> while the ksp1 uses the rhs b1, and the ksp2 uses the rhs b2. This is not
> expected because I've set ksp1 to use ComputeMatrix1 before:
>
>       ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL
> );CHKERRQ(ierr);
>
>     ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);
>
>
>  Now if I comment out the lines
>
>       ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL
> );CHKERRQ(ierr);
>
>  and
>
>       ierr = KSPSolve(ksp2,b2,x2);CHKERRQ(ierr);
>
>
>  Then ksp1 start to use ComputeMatrix1, as can be visualized from both
> the output file and the terminal print. So it shows me that there are some
> crosstalk between ksp1 and ksp2. If I set
>
>      ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL
> );CHKERRQ(ierr);
>
>  ksp1 uses ComputeMatrix1. But if I set
>
>       ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL
> );CHKERRQ(ierr);
>
>     ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);
>
>  ksp1 uses ComputeMatrix2.
>
>
>
>  ------------------------------
> *From:* Dave May [dave.mayhem23 at gmail.com]
> *Sent:* Tuesday, March 03, 2015 1:11 AM
> *To:* Sun, Hui
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] A very strange bug about using multiple KSPs
>
>  Try moving your print statement in ComputeMatrix2 to be after
> PetscFunctionBeginUser;
>
> You should really use a debugger to see if these functions are being
> called or not - don't use print statements.
>
> On Tuesday, 3 March 2015, Sun, Hui <hus003 at ucsd.edu> wrote:
>
>>  I set up 2 KSPs through KSPSetComputeOperators, with separate compute
>> matrix functions.
>>
>>
>>
>>     ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL
>> );CHKERRQ(ierr);
>>
>>     ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL
>> );CHKERRQ(ierr);
>>
>>  Within each computeMatrix function, I define the matrix, and I also
>> print something to the screen. So
>>
>>  PetscErrorCode ComputeMatrix1(KSP ksp, Mat J,Mat jac, void *ctx)
>>
>> {
>>
>>     ......
>>
>>     ierr    = PetscPrintf(PETSC_COMM_WORLD,"1\n");CHKERRQ(ierr);
>>
>>     ......
>>
>> }
>>
>>
>>  PetscErrorCode ComputeMatrix2(KSP ksp, Mat J,Mat jac, void *ctx)
>>
>> {
>>
>>     ......
>>
>>     ierr    = PetscPrintf(PETSC_COMM_WORLD,"2\n");CHKERRQ(ierr);
>>
>>     ......
>>
>> }
>>
>>
>>  Now if I compile and execute the code, it only print 2, completely
>> ignoring 1, which is not expected, since I have setup ksp1 to use
>> ComputeMatrix1.
>>
>>  Does anyone know what might be wrong? Thank you. I have attached the
>> short code here in this email as well.
>>
>>  Hui
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/deddc724/attachment-0001.html>

From hus003 at ucsd.edu  Tue Mar  3 11:17:05 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Tue, 3 Mar 2015 17:17:05 +0000
Subject: [petsc-users] A very strange bug about using multiple KSPs
In-Reply-To: <CAMYG4Gkv8FzMRHiy7EEY+0C62w6E0jtjJ9miy-Jh1Xyd_oQdOg@mail.gmail.com>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E26@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<CAJ98EDoesDFzxYQuU-GvnSNcn=f_NR6=b8_3kCasRQ4Qe06EQg@mail.gmail.com>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9E56@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<CAMYG4Gkv8FzMRHiy7EEY+0C62w6E0jtjJ9miy-Jh1Xyd_oQdOg@mail.gmail.com>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9E75@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you, Matthew! It does solve the problem.

Best,
Hui


________________________________
From: Matthew Knepley [knepley at gmail.com]
Sent: Tuesday, March 03, 2015 8:26 AM
To: Sun, Hui
Cc: Dave May; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] A very strange bug about using multiple KSPs

On Tue, Mar 3, 2015 at 10:15 AM, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
Dave, thank you for your suggestions. I don't have a debugger, and I think without debugger we can still figure out if a function is called or not.

Putting the print commend before or after PetscFunctionBeginUser doesn't really matter. I just tried to put it after PetscFunctionBeginUser, but something doesn't print just doesn't print.

The ComputeMatrix function is stored in the DM, or more precisely in the DMKSP object. Since both KSPs have the
same DM, they have the same function.

I believe you can DMClone(da, &da2), and then this will work. Please tell me if it does not.

  Thanks,

    Matt

There are other ways to see that computeMatrix1 is not called. For example, I just change b1 to a vector of 1's, and b2 to a vector of 0.5's, and ComputeMatrix1 to give diagonal matrix with entries 1, and  ComputeMatrix2 to give diagonal matrix with entries 2.

Then I output the matrices and the vectors into a file, and visualize them in MATLAB. It does show me that the ksp1 and ksp2 both use ComputeMatrix2; while the ksp1 uses the rhs b1, and the ksp2 uses the rhs b2. This is not expected because I've set ksp1 to use ComputeMatrix1 before:


    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);


Now if I comment out the lines


    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

and


    ierr = KSPSolve(ksp2,b2,x2);CHKERRQ(ierr);


Then ksp1 start to use ComputeMatrix1, as can be visualized from both the output file and the terminal print. So it shows me that there are some crosstalk between ksp1 and ksp2. If I set

    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

ksp1 uses ComputeMatrix1. But if I set


    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

ksp1 uses ComputeMatrix2.



________________________________
From: Dave May [dave.mayhem23 at gmail.com<mailto:dave.mayhem23 at gmail.com>]
Sent: Tuesday, March 03, 2015 1:11 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] A very strange bug about using multiple KSPs

Try moving your print statement in ComputeMatrix2 to be after PetscFunctionBeginUser;

You should really use a debugger to see if these functions are being called or not - don't use print statements.

On Tuesday, 3 March 2015, Sun, Hui <hus003 at ucsd.edu<mailto:hus003 at ucsd.edu>> wrote:
I set up 2 KSPs through KSPSetComputeOperators, with separate compute matrix functions.



    ierr = KSPSetComputeOperators(ksp1,ComputeMatrix1,NULL);CHKERRQ(ierr);

    ierr = KSPSetComputeOperators(ksp2,ComputeMatrix2,NULL);CHKERRQ(ierr);

Within each computeMatrix function, I define the matrix, and I also print something to the screen. So


PetscErrorCode ComputeMatrix1(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"1\n");CHKERRQ(ierr);

    ......

}


PetscErrorCode ComputeMatrix2(KSP ksp, Mat J,Mat jac, void *ctx)

{

    ......

    ierr    = PetscPrintf(PETSC_COMM_WORLD,"2\n");CHKERRQ(ierr);

    ......

}


Now if I compile and execute the code, it only print 2, completely ignoring 1, which is not expected, since I have setup ksp1 to use ComputeMatrix1.

Does anyone know what might be wrong? Thank you. I have attached the short code here in this email as well.

Hui



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/c9641834/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar  3 13:02:20 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Mar 2015 13:02:20 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
	systems for discretized coupled multiphysics problems
In-Reply-To: <1425352768.27237.88.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
	<1425352768.27237.88.camel@gmail.com>
Message-ID: <D298F42E-EE6F-40BD-A083-E178A38EDC10@mcs.anl.gov>


> On Mar 2, 2015, at 9:19 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> 
> On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
>>  Do you really want tolerances:  relative=1e-90, absolute=1.10423, divergence=10000? That is an absolute tolerance of 1.1? Normally that would be huge.
> 
> I started using atol as convergence criterion with -ksp_norm_type
> unpreconditioned. The value of atol gets updated every outer iteration.

   What is the "outer iteration" and why does it exist?

> 
>>  You can provide your matrix with a block size that GAMG will use with MatSetBlockSize().
> 
> I think something went wrong. Setting the block size to 4 and solving
> for (u,v,w,p) the convergence degraded significantly. I attached the
> results for a smaller test case that shows the increase of the number of
> needed inner iterations when setting the block size via
> MatSetBlockSize().

  This is possible. 

  Barry

> 
>> 
>>  I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that it is using SOR for 27 points.
>> 
>> So you must have provided a null space since it printed "has attached null space"
> 
> The system has indeed a one dimensional null space (from the pressure
> equation with Neumann boundary conditions). But now that you mentioned
> it: It seems that the outer GMRES doesn't notice that the matrix has an
> attached nullspace. Replacing
> 
> 	CALL MatSetNullSpace(CMAT,NULLSP,IERR)
> 
> with
> 
> 	CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)
> 
> solves this. What is wrong with using MatSetNullSpace?
> 
> 
> Fabian
> 
> 
>> 
>>  Barry
>> 
>> 
>> 
>>> On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
>>> 
>>> On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
>>>> Fabian Gabel <gabel.fabian at gmail.com> writes:
>>>> 
>>>>> Dear PETSc Team,
>>>>> 
>>>>> I came across the following paragraph in your publication "Composable
>>>>> Linear Solvers for Multiphysics" (2012):
>>>>> 
>>>>> "Rather than splitting the matrix into large blocks and
>>>>> forming a preconditioner from solvers (for example, multi-
>>>>> grid) on each block, one can perform multigrid on the entire
>>>>> system, basing the smoother on solves coming from the tiny
>>>>> blocks coupling the degrees of freedom at a single point (or
>>>>> small number of points). This approach is also handled in
>>>>> PETSc, but we will not elaborate on it here."
>>>>> 
>>>>> How would I use a multigrid preconditioner (GAMG) 
>>>> 
>>>> The heuristics in GAMG are not appropriate for indefinite/saddle-point
>>>> systems such as arise from Navier-Stokes.  You can use geometric
>>>> multigrid and use the fieldsplit techniques described in the paper as a
>>>> smoother, for example.
>>> 
>>> I sadly don't have a solid background on multigrid methods, but as
>>> mentioned in a previous thread
>>> 
>>> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
>>> 
>>> AMG has apparently been used (successfully?) for fully-coupled
>>> finite-volume discretizations of Navier-Stokes:
>>> 
>>> http://dx.doi.org/10.1080/10407790.2014.894448
>>> http://dx.doi.org/10.1016/j.jcp.2008.08.027
>>> 
>>> I was hoping to achieve something similar with the right configuration
>>> of the PETSc preconditioners. So far I have only been using GAMG in a
>>> straightforward manner, without providing any details on the structure
>>> of the linear system. I attached the output of a test run with GAMG.
>>> 
>>>> 
>>>>> from PETSc on linear systems of the form (after reordering the
>>>>> variables):
>>>>> 
>>>>> [A_uu   0     0   A_up  A_uT]
>>>>> [0    A_vv    0   A_vp  A_vT]
>>>>> [0      0   A_ww  A_up  A_wT]
>>>>> [A_pu A_pv  A_pw  A_pp   0  ]
>>>>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
>>>>> 
>>>>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
>>>>> directly from a FVM discretization of the incompressible Navier-Stokes
>>>>> equations and the temperature equation. The fifth row and column are
>>>>> optional, depending on the method I choose to couple the temperature.
>>>>> The Matrix is stored as one AIJ Matrix.
>>>>> 
>>>>> Regards,
>>>>> Fabian Gabel
>>> 
>>> <cpld_0128.out.578677>
>> 
> 
> <log.blocksize4><log.txt>


From jed at jedbrown.org  Tue Mar  3 16:49:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 03 Mar 2015 15:49:12 -0700
Subject: [petsc-users] VTK Output / no file
In-Reply-To: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>
References: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>
Message-ID: <87bnk98zo7.fsf@jedbrown.org>

Grischa Jacobs <grischa.jacobs at email.de> writes:

> Dear,
>  
> I used to save my Vec results to vts files. Unfortunately this is not working
> anymore. My Code:
>
> // Creating 3DDM before
> ierr = DMDACreateNaturalVector(dynamic_Maxwell, &dynamic_fields_global);CHKERRQ
> (ierr);
> ierr = VecSet(dynamic_fields_global,some_number);CHKERRQ(ierr);
>
> PetscViewer viewer;
> ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD, "test.vts", FILE_MODE_WRITE, &
> viewer);CHKERRQ(ierr);
> ierr = VecView(dynamic_fields_global, viewer);CHKERRQ(ierr);
> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

What version of PETSc are you using and what error do you get?

> Saving to ASCII still works:
> ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,"test.vts",&viewer);CHKERRQ(ierr);
>
> I don't get any errors.
>
>  
>
> king regrads,
> Grischa
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150303/a08b5a1e/attachment.pgp>

From M.Imtiaz at victorchang.edu.au  Tue Mar  3 22:41:38 2015
From: M.Imtiaz at victorchang.edu.au (Mohammad Imtiaz)
Date: Wed, 4 Mar 2015 04:41:38 +0000
Subject: [petsc-users] Saving binary output at every tilmestep in ts/ex3.c
Message-ID: <D11CD811.C9C3%m.imtiaz@victorchang.edu.au>

Hi
I am trying to save output at every time step in ts/ex3.c.  But all that gets saved is the final output.  I have changed example ts/ex3.c as follows:


Added a new viewer
Line 77: PetscViewer viewer1,viewer2,viewer3;  /* viewers for the solution and error */

Added
Line 161: PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_WRITE,&appctx.viewer3);


Added
Line 271: ierr = PetscViewerDestroy(&appctx.viewer3);CHKERRQ(ierr);


In Monitor function added:
VecView(u,appctx->viewer3);CHKERRQ(ierr);


I will be thankful for any help understanding what I am missing.

Thanks
Imtiaz




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150304/58e7860e/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar  3 23:00:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Mar 2015 23:00:34 -0600
Subject: [petsc-users] Saving binary output at every tilmestep in
	ts/ex3.c
In-Reply-To: <D11CD811.C9C3%m.imtiaz@victorchang.edu.au>
References: <D11CD811.C9C3%m.imtiaz@victorchang.edu.au>
Message-ID: <02D0E586-B30D-45CC-ADB5-2791BD22F55A@mcs.anl.gov>


  What you have done sounds like it is reasonable.

  You need to run in the debugger and put a break point in VecView() to make sure it is being called multiple times and it is writing each time.

   Barry

> On Mar 3, 2015, at 10:41 PM, Mohammad Imtiaz <M.Imtiaz at victorchang.edu.au> wrote:
> 
> Hi 
> I am trying to save output at every time step in ts/ex3.c.  But all that gets saved is the final output.  I have changed example ts/ex3.c as follows:
> 
> 
> Added a new viewer
> Line 77: PetscViewer viewer1,viewer2,viewer3;  /* viewers for the solution and error */
> 
> Added
> Line 161: PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_WRITE,&appctx.viewer3);
> 
> 
> Added
> Line 271: ierr = PetscViewerDestroy(&appctx.viewer3);CHKERRQ(ierr);
> 
> 
> In Monitor function added:
> VecView(u,appctx->viewer3);CHKERRQ(ierr);
> 
> 
> I will be thankful for any help understanding what I am missing.
> 
> Thanks
> Imtiaz
> 
> 
> 
> 


From hus003 at ucsd.edu  Wed Mar  4 01:20:59 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Wed, 4 Mar 2015 07:20:59 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors. 

Should I somehow use the following functions?
MatCreate
DMDAGetLocalInfo
ISLocalToGlobalMappingCreate
MatSetLocalToGlobalMapping
MatGetLocalSubMatrix

However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time. 

Best,
Hui

________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Sunday, March 01, 2015 9:24 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Thank you Barry. I have yet two more questions:
>
> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>
> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?

  Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.

>
> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>
> Best,
> Hui
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Saturday, February 28, 2015 9:35 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>
>> Is it true that
>> 1) Only DMDA can use mg?
>
>   No this is not true
>
>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>
>   No you do not have to
>
>> We cannot create a matrix and add it to KSP if we want to use mg?
>
>    Yes you can.
>
>   There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>
> More details
>
>> 1) Only DMDA can use mg?
>
>   Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>
>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>
>   Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>
>> We cannot create a matrix and add it to KSP if we want to use mg?
>
>    As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>
>   If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>
>  Barry
>
>>
>> Best,
>> Hui
>>
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Friday, February 27, 2015 5:11 PM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>
>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>
>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>
>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>
>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>
>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>
>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>
>> No it handles dof > 1 fine.
>>
>> Send your code.
>>
>> Barry
>>
>>>
>>> Thank you!

From bsmith at mcs.anl.gov  Wed Mar  4 07:27:04 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Mar 2015 07:27:04 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov>


  If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?

  Barry



> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors. 
> 
> Should I somehow use the following functions?
> MatCreate
> DMDAGetLocalInfo
> ISLocalToGlobalMappingCreate
> MatSetLocalToGlobalMapping
> MatGetLocalSubMatrix
> 
> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time. 
> 
> Best,
> Hui
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Sunday, March 01, 2015 9:24 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>> 
>> Thank you Barry. I have yet two more questions:
>> 
>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>> 
>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
> 
>  Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
> 
>> 
>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>> 
>> Best,
>> Hui
>> 
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Saturday, February 28, 2015 9:35 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>> 
>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>> 
>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>> 
>>> Is it true that
>>> 1) Only DMDA can use mg?
>> 
>>  No this is not true
>> 
>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>> 
>>  No you do not have to
>> 
>>> We cannot create a matrix and add it to KSP if we want to use mg?
>> 
>>   Yes you can.
>> 
>>  There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>> 
>> More details
>> 
>>> 1) Only DMDA can use mg?
>> 
>>  Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>> 
>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>> 
>>  Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>> 
>>> We cannot create a matrix and add it to KSP if we want to use mg?
>> 
>>   As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>> 
>>  If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>> 
>> Barry
>> 
>>> 
>>> Best,
>>> Hui
>>> 
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Friday, February 27, 2015 5:11 PM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>> 
>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>> 
>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>> 
>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>> 
>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>> 
>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>> 
>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>> 
>>> No it handles dof > 1 fine.
>>> 
>>> Send your code.
>>> 
>>> Barry
>>> 
>>>> 
>>>> Thank you!


From hus003 at ucsd.edu  Wed Mar  4 08:53:51 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Wed, 4 Mar 2015 14:53:51 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>

I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i. 

However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i. 

Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Wednesday, March 04, 2015 5:27 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

  If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?

  Barry



> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>
> Should I somehow use the following functions?
> MatCreate
> DMDAGetLocalInfo
> ISLocalToGlobalMappingCreate
> MatSetLocalToGlobalMapping
> MatGetLocalSubMatrix
>
> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>
> Best,
> Hui
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Sunday, March 01, 2015 9:24 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> Thank you Barry. I have yet two more questions:
>>
>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>
>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>
>  Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>
>>
>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>
>> Best,
>> Hui
>>
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Saturday, February 28, 2015 9:35 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>
>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>
>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>
>>> Is it true that
>>> 1) Only DMDA can use mg?
>>
>>  No this is not true
>>
>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>
>>  No you do not have to
>>
>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>
>>   Yes you can.
>>
>>  There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>
>> More details
>>
>>> 1) Only DMDA can use mg?
>>
>>  Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>
>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>
>>  Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>
>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>
>>   As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>
>>  If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>
>> Barry
>>
>>>
>>> Best,
>>> Hui
>>>
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Friday, February 27, 2015 5:11 PM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>
>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>
>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>
>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>
>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>
>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>
>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>
>>> No it handles dof > 1 fine.
>>>
>>> Send your code.
>>>
>>> Barry
>>>
>>>>
>>>> Thank you!

From grischa.jacobs at email.de  Wed Mar  4 11:34:31 2015
From: grischa.jacobs at email.de (Grischa Jacobs)
Date: Wed, 4 Mar 2015 18:34:31 +0100
Subject: [petsc-users] VTK Output / no file
In-Reply-To: <87bnk98zo7.fsf@jedbrown.org>
References: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>,
	<87bnk98zo7.fsf@jedbrown.org>
Message-ID: <trinity-2dc81f84-0046-40c3-a477-372c2e9dbc85-1425490471286@3capp-webde-bs06>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150304/ea442e0c/attachment.html>

From Carol.Brickley at awe.co.uk  Wed Mar  4 12:26:50 2015
From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk)
Date: Wed, 4 Mar 2015 18:26:50 +0000
Subject: [petsc-users] DMDAVecGetArrayF90
Message-ID: <201503041826.t24IQuJN015737@msw1.awe.co.uk>

Hi,

I am using DMDAVecGetArrayF90 with a pointer in petsc 3.4.3. Is there anyway of finding out what the start and finish indices of the pointer coming back from this routine are per processor?

I have declared:


DM                              ::   da
Vec                             ::  X
PetscScalar,pointer      :: xv(:,:)=>null()


DMDAVecGetArrayF90(da,X,xv,ierr)


Do j = 1, m
  Do i = 1, n
    xv(i-1.j-1) = rhs2d(i,j)
  enddo
enddo


DMDAVecRestoreArrayF90(da,X,xv,ierr)

When we have xv(0:n-1,0:m-1) from DMDAVecGetArrayF90 it works fine but other processors may give back eg; xv(0:n-1,1:m-1) or something else and it will fail as the second index
now goes below 0 to 1. Any ideas of what I can do next?


Thanks,

Carol


Dr Carol Brickley
BSc,PhD,ARCS,DIC,MBCS

Senior Software Engineer
Applied Computer Science
DS+T,
AWE
Aldermaston
Reading
Berkshire
RG7 4PR

Direct: 0118 9855035



___________________________________________________
____________________________

The information in this email and in any attachment(s) is 
commercial in confidence. If you are not the named addressee(s) 
or 
if you receive this email in error then any distribution, copying or 
use of this communication or the information in it is strictly 
prohibited. Please notify us immediately by email at 
admin.internet(at)awe.co.uk, and then delete this message from 
your computer. While attachments are virus checked, AWE plc 
does not accept any liability in respect of any virus which is not 
detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR

From knepley at gmail.com  Wed Mar  4 12:35:26 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 4 Mar 2015 12:35:26 -0600
Subject: [petsc-users] DMDAVecGetArrayF90
In-Reply-To: <201503041826.t24IQuJN015737@msw1.awe.co.uk>
References: <201503041826.t24IQuJN015737@msw1.awe.co.uk>
Message-ID: <CAMYG4GmWs6vfRXorofKApzw-mdKG2efzGHm02ZZ=4pMWJV90rg@mail.gmail.com>

On Wed, Mar 4, 2015 at 12:26 PM, <Carol.Brickley at awe.co.uk> wrote:

> Hi,
>
> I am using DMDAVecGetArrayF90 with a pointer in petsc 3.4.3. Is there
> anyway of finding out what the start and finish indices of the pointer
> coming back from this routine are per processor?
>
> I have declared:
>
>
> DM                              ::   da
> Vec                             ::  X
> PetscScalar,pointer      :: xv(:,:)=>null()
>
>
> DMDAVecGetArrayF90(da,X,xv,ierr)
>
>
> Do j = 1, m
>   Do i = 1, n
>     xv(i-1.j-1) = rhs2d(i,j)
>   enddo
> enddo
>
>
> DMDAVecRestoreArrayF90(da,X,xv,ierr)
>
> When we have xv(0:n-1,0:m-1) from DMDAVecGetArrayF90 it works fine but
> other processors may give back eg; xv(0:n-1,1:m-1) or something else and it
> will fail as the second index
> now goes below 0 to 1. Any ideas of what I can do next?
>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGetCorners.html
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGetGhostCorners.html#DMDAGetGhostCorners

   Matt


>
> Thanks,
>
> Carol
>
>
> Dr Carol Brickley
> BSc,PhD,ARCS,DIC,MBCS
>
> Senior Software Engineer
> Applied Computer Science
> DS+T,
> AWE
> Aldermaston
> Reading
> Berkshire
> RG7 4PR
>
> Direct: 0118 9855035
>
>
>
> ___________________________________________________
> ____________________________
>
> The information in this email and in any attachment(s) is
> commercial in confidence. If you are not the named addressee(s)
> or
> if you receive this email in error then any distribution, copying or
> use of this communication or the information in it is strictly
> prohibited. Please notify us immediately by email at
> admin.internet(at)awe.co.uk, and then delete this message from
> your computer. While attachments are virus checked, AWE plc
> does not accept any liability in respect of any virus which is not
> detected.
>
> AWE Plc
> Registered in England and Wales
> Registration No 02763902
> AWE, Aldermaston, Reading, RG7 4PR
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150304/634cfe7c/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar  4 12:37:52 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Mar 2015 12:37:52 -0600
Subject: [petsc-users] DMDAVecGetArrayF90
In-Reply-To: <201503041826.t24IQuJN015737@msw1.awe.co.uk>
References: <201503041826.t24IQuJN015737@msw1.awe.co.uk>
Message-ID: <439EDE4E-39FB-40E2-BFB6-F5F62C198E4E@mcs.anl.gov>


  DMDAGetCorners() for global vectors and DMDAGetGhostCorners() for local (ghosted) vectors.

   Barry

> On Mar 4, 2015, at 12:26 PM, <Carol.Brickley at awe.co.uk> <Carol.Brickley at awe.co.uk> wrote:
> 
> Hi,
> 
> I am using DMDAVecGetArrayF90 with a pointer in petsc 3.4.3. Is there anyway of finding out what the start and finish indices of the pointer coming back from this routine are per processor?
> 
> I have declared:
> 
> 
> DM                              ::   da
> Vec                             ::  X
> PetscScalar,pointer      :: xv(:,:)=>null()
> 
> 
> DMDAVecGetArrayF90(da,X,xv,ierr)
> 
> 
> Do j = 1, m
>  Do i = 1, n
>    xv(i-1.j-1) = rhs2d(i,j)
>  enddo
> enddo
> 
> 
> DMDAVecRestoreArrayF90(da,X,xv,ierr)
> 
> When we have xv(0:n-1,0:m-1) from DMDAVecGetArrayF90 it works fine but other processors may give back eg; xv(0:n-1,1:m-1) or something else and it will fail as the second index
> now goes below 0 to 1. Any ideas of what I can do next?
> 
> 
> Thanks,
> 
> Carol
> 
> 
> Dr Carol Brickley
> BSc,PhD,ARCS,DIC,MBCS
> 
> Senior Software Engineer
> Applied Computer Science
> DS+T,
> AWE
> Aldermaston
> Reading
> Berkshire
> RG7 4PR
> 
> Direct: 0118 9855035
> 
> 
> 
> ___________________________________________________
> ____________________________
> 
> The information in this email and in any attachment(s) is 
> commercial in confidence. If you are not the named addressee(s) 
> or 
> if you receive this email in error then any distribution, copying or 
> use of this communication or the information in it is strictly 
> prohibited. Please notify us immediately by email at 
> admin.internet(at)awe.co.uk, and then delete this message from 
> your computer. While attachments are virus checked, AWE plc 
> does not accept any liability in respect of any virus which is not 
> detected.
> 
> AWE Plc
> Registered in England and Wales
> Registration No 02763902
> AWE, Aldermaston, Reading, RG7 4PR


From Carol.Brickley at awe.co.uk  Wed Mar  4 12:39:43 2015
From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk)
Date: Wed, 4 Mar 2015 18:39:43 +0000
Subject: [petsc-users] EXTERNAL: Re:  DMDAVecGetArrayF90
In-Reply-To: <439EDE4E-39FB-40E2-BFB6-F5F62C198E4E@mcs.anl.gov>
References: <201503041826.t24IQuJN015737@msw1.awe.co.uk>
	<439EDE4E-39FB-40E2-BFB6-F5F62C198E4E@mcs.anl.gov>
Message-ID: <201503041839.t24Idksr014177@msw1.awe.co.uk>

Excellent - just remembered that. I'll see if it works. Had taken it out to do my own thing.

Carol

Dr Carol Brickley 
BSc,PhD,ARCS,DIC,MBCS
 
Senior Software Engineer
Applied Computer Science
DS+T,
AWE
Aldermaston
Reading
Berkshire
RG7 4PR
 
Direct: 0118 9855035
 

-----Original Message-----
From: Barry Smith [mailto:bsmith at mcs.anl.gov] 
Sent: 04 March 2015 18:38
To: Brickley Carol AWE
Cc: petsc-users at mcs.anl.gov
Subject: EXTERNAL: Re: [petsc-users] DMDAVecGetArrayF90


  DMDAGetCorners() for global vectors and DMDAGetGhostCorners() for local (ghosted) vectors.

   Barry

> On Mar 4, 2015, at 12:26 PM, <Carol.Brickley at awe.co.uk> <Carol.Brickley at awe.co.uk> wrote:
> 
> Hi,
> 
> I am using DMDAVecGetArrayF90 with a pointer in petsc 3.4.3. Is there anyway of finding out what the start and finish indices of the pointer coming back from this routine are per processor?
> 
> I have declared:
> 
> 
> DM                              ::   da
> Vec                             ::  X
> PetscScalar,pointer      :: xv(:,:)=>null()
> 
> 
> DMDAVecGetArrayF90(da,X,xv,ierr)
> 
> 
> Do j = 1, m
>  Do i = 1, n
>    xv(i-1.j-1) = rhs2d(i,j)
>  enddo
> enddo
> 
> 
> DMDAVecRestoreArrayF90(da,X,xv,ierr)
> 
> When we have xv(0:n-1,0:m-1) from DMDAVecGetArrayF90 it works fine but other processors may give back eg; xv(0:n-1,1:m-1) or something else and it will fail as the second index
> now goes below 0 to 1. Any ideas of what I can do next?
> 
> 
> Thanks,
> 
> Carol
> 
> 
> Dr Carol Brickley
> BSc,PhD,ARCS,DIC,MBCS
> 
> Senior Software Engineer
> Applied Computer Science
> DS+T,
> AWE
> Aldermaston
> Reading
> Berkshire
> RG7 4PR
> 
> Direct: 0118 9855035
> 
> 
> 
> ___________________________________________________
> ____________________________
> 
> The information in this email and in any attachment(s) is 
> commercial in confidence. If you are not the named addressee(s) 
> or 
> if you receive this email in error then any distribution, copying or 
> use of this communication or the information in it is strictly 
> prohibited. Please notify us immediately by email at 
> admin.internet(at)awe.co.uk, and then delete this message from 
> your computer. While attachments are virus checked, AWE plc 
> does not accept any liability in respect of any virus which is not 
> detected.
> 
> AWE Plc
> Registered in England and Wales
> Registration No 02763902
> AWE, Aldermaston, Reading, RG7 4PR


___________________________________________________
____________________________

The information in this email and in any attachment(s) is 
commercial in confidence. If you are not the named addressee(s) 
or 
if you receive this email in error then any distribution, copying or 
use of this communication or the information in it is strictly 
prohibited. Please notify us immediately by email at 
admin.internet(at)awe.co.uk, and then delete this message from 
your computer. While attachments are virus checked, AWE plc 
does not accept any liability in respect of any virus which is not 
detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR

From bsmith at mcs.anl.gov  Wed Mar  4 12:53:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Mar 2015 12:53:44 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov>


> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i. 

  Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
> 
> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i. 

   PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply. 

   To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);

   Barry


>> DMDAGetLocalInfo



> 
> Hui
> 
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Wednesday, March 04, 2015 5:27 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
>  If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
> 
>  Barry
> 
> 
> 
>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>> 
>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>> 
>> Should I somehow use the following functions?
>> MatCreate
>> DMDAGetLocalInfo
>> ISLocalToGlobalMappingCreate
>> MatSetLocalToGlobalMapping
>> MatGetLocalSubMatrix
>> 
>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>> 
>> Best,
>> Hui
>> 
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Sunday, March 01, 2015 9:24 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>> 
>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>> 
>>> Thank you Barry. I have yet two more questions:
>>> 
>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>> 
>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>> 
>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>> 
>>> 
>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>> 
>>> Best,
>>> Hui
>>> 
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Saturday, February 28, 2015 9:35 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>> 
>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>> 
>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>> 
>>>> Is it true that
>>>> 1) Only DMDA can use mg?
>>> 
>>> No this is not true
>>> 
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>> 
>>> No you do not have to
>>> 
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>> 
>>>  Yes you can.
>>> 
>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>> 
>>> More details
>>> 
>>>> 1) Only DMDA can use mg?
>>> 
>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>> 
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>> 
>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>> 
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>> 
>>>  As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>> 
>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>> 
>>> Barry
>>> 
>>>> 
>>>> Best,
>>>> Hui
>>>> 
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>> 
>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>> 
>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>> 
>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>> 
>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>> 
>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>> 
>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>> 
>>>> No it handles dof > 1 fine.
>>>> 
>>>> Send your code.
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> Thank you!


From hus003 at ucsd.edu  Wed Mar  4 15:09:44 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Wed, 4 Mar 2015 21:09:44 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010E9F79@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you Barry. I think this is what I want. 

Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Wednesday, March 04, 2015 10:53 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i.

  Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
>
> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i.

   PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply.

   To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);

   Barry


>> DMDAGetLocalInfo



>
> Hui
>
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Wednesday, March 04, 2015 5:27 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>  If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
>
>  Barry
>
>
>
>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>>
>> Should I somehow use the following functions?
>> MatCreate
>> DMDAGetLocalInfo
>> ISLocalToGlobalMappingCreate
>> MatSetLocalToGlobalMapping
>> MatGetLocalSubMatrix
>>
>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>>
>> Best,
>> Hui
>>
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Sunday, March 01, 2015 9:24 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>
>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>
>>> Thank you Barry. I have yet two more questions:
>>>
>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>>
>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>>
>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>>
>>>
>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>>
>>> Best,
>>> Hui
>>>
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Saturday, February 28, 2015 9:35 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>
>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>
>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>>
>>>> Is it true that
>>>> 1) Only DMDA can use mg?
>>>
>>> No this is not true
>>>
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>
>>> No you do not have to
>>>
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>
>>>  Yes you can.
>>>
>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>>
>>> More details
>>>
>>>> 1) Only DMDA can use mg?
>>>
>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>>
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>
>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>>
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>
>>>  As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>>
>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>>
>>> Barry
>>>
>>>>
>>>> Best,
>>>> Hui
>>>>
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>
>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>
>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>>
>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>>
>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>>
>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>>
>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>>
>>>> No it handles dof > 1 fine.
>>>>
>>>> Send your code.
>>>>
>>>> Barry
>>>>
>>>>>
>>>>> Thank you!


From rlmackie862 at gmail.com  Wed Mar  4 19:45:53 2015
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Wed, 4 Mar 2015 17:45:53 -0800
Subject: [petsc-users] Question about GAMG and memory use
Message-ID: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>

In my application, I am repeatedly calling KSPSolve with the following options:

-ksp_type gmres \
-pc_type gamg \
-pc_gamg_type agg \
-pc_gamg_agg_nsmooths 1\


each call is after the matrix and right hand side have been updated.

This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.

Is this expected behavior?

I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).

However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.

I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.

I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.


Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?


Any other suggestions for tracking this down? I've run out of ideas.


Thanks in advance,

Randy





From bsmith at mcs.anl.gov  Wed Mar  4 21:30:07 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Mar 2015 21:30:07 -0600
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
Message-ID: <E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>


> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> In my application, I am repeatedly calling KSPSolve with the following options:
> 
> -ksp_type gmres \
> -pc_type gamg \
> -pc_gamg_type agg \
> -pc_gamg_agg_nsmooths 1\
> 
> 
> each call is after the matrix and right hand side have been updated.
> 
> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
> 
> Is this expected behavior?

  No
> 
> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
> 
> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
> 
> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.

   Run your program for a few solves with the command line option -malloc and after each call call to KSPSolve() put in a call to PetscMallocDump(). Take a look at the output and email it to us (this is best done with one process if you can; does the memory problem happen with 1 MPI process?).

   Barry

> 
> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
> 
> 
> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
> 
> 
> Any other suggestions for tracking this down? I've run out of ideas.
> 
> 
> Thanks in advance,
> 
> Randy
> 
> 
> 
> 


From M.Imtiaz at victorchang.edu.au  Thu Mar  5 01:42:27 2015
From: M.Imtiaz at victorchang.edu.au (Mohammad Imtiaz)
Date: Thu, 5 Mar 2015 07:42:27 +0000
Subject: [petsc-users] Saving binary output at every tilmestep in
 ts/ex3.c
In-Reply-To: <02D0E586-B30D-45CC-ADB5-2791BD22F55A@mcs.anl.gov>
References: <D11CD811.C9C3%m.imtiaz@victorchang.edu.au>
	<02D0E586-B30D-45CC-ADB5-2791BD22F55A@mcs.anl.gov>
Message-ID: <D11E5373.C9FD%m.imtiaz@victorchang.edu.au>

Hi Barry,

Thank you for this advice, it solved my problem.

I was able to see that VecVeiw() wrote all the time and narrowed down my
problem to reading back the binary files.

Sorry, I should have done that before posting.

Mohammad





On 4/03/2015 4:00 pm, "Barry Smith" <bsmith at mcs.anl.gov> wrote:

>
>  What you have done sounds like it is reasonable.
>
>  You need to run in the debugger and put a break point in VecView() to
>make sure it is being called multiple times and it is writing each time.
>
>   Barry
>
>> On Mar 3, 2015, at 10:41 PM, Mohammad Imtiaz
>><M.Imtiaz at victorchang.edu.au> wrote:
>> 
>> Hi 
>> I am trying to save output at every time step in ts/ex3.c.  But all
>>that gets saved is the final output.  I have changed example ts/ex3.c as
>>follows:
>> 
>> 
>> Added a new viewer
>> Line 77: PetscViewer viewer1,viewer2,viewer3;  /* viewers for the
>>solution and error */
>> 
>> Added
>> Line 161: 
>>PetscViewerBinaryOpen(PETSC_COMM_WORLD,"out.bin",FILE_MODE_WRITE,&appctx.
>>viewer3);
>> 
>> 
>> Added
>> Line 271: ierr = PetscViewerDestroy(&appctx.viewer3);CHKERRQ(ierr);
>> 
>> 
>> In Monitor function added:
>> VecView(u,appctx->viewer3);CHKERRQ(ierr);
>> 
>> 
>> I will be thankful for any help understanding what I am missing.
>> 
>> Thanks
>> Imtiaz
>> 
>> 
>> 
>> 
>
>Scanned by Messagelabs ***


From popov at uni-mainz.de  Thu Mar  5 05:01:23 2015
From: popov at uni-mainz.de (anton)
Date: Thu, 5 Mar 2015 12:01:23 +0100
Subject: [petsc-users] near null space redundant
Message-ID: <54F83783.2090109@uni-mainz.de>

Hi,

Does the near-null-space get attached to the internal ksp object of 
redundant preconditioner, or a user has to scatter and attach it manually?
If so, what would be a suggested way to do that?

Thanks,

Anton

From knepley at gmail.com  Thu Mar  5 09:18:20 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 5 Mar 2015 09:18:20 -0600
Subject: [petsc-users] near null space redundant
In-Reply-To: <54F83783.2090109@uni-mainz.de>
References: <54F83783.2090109@uni-mainz.de>
Message-ID: <CAMYG4Gnxo_SYAR7h0i9EU3-_P3fy8ygCWZThvOP73zmRkA_WDQ@mail.gmail.com>

On Thu, Mar 5, 2015 at 5:01 AM, anton <popov at uni-mainz.de> wrote:

> Hi,
>
> Does the near-null-space get attached to the internal ksp object of
> redundant preconditioner, or a user has to scatter and attach it manually?
> If so, what would be a suggested way to do that?
>

Its not there, but would be easy to put in. I would either

  a) Open an issue with the Bitbucket tracker, and someone will get to it

or

  b) Code it up in a pull request

I think it would only be a few lines in PCSetUp_Redundant(). You check
whether the pc->mat has
an attached near null space, and then use the VecScatters above to map the
vectors, and carry
over the flag for constant null space.

  Thanks,

     Matt


> Thanks,
>
> Anton
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/c7317d6d/attachment.html>

From rlmackie862 at gmail.com  Thu Mar  5 12:07:56 2015
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 5 Mar 2015 10:07:56 -0800
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
Message-ID: <5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>


> On Mar 4, 2015, at 7:30 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> In my application, I am repeatedly calling KSPSolve with the following options:
>> 
>> -ksp_type gmres \
>> -pc_type gamg \
>> -pc_gamg_type agg \
>> -pc_gamg_agg_nsmooths 1\
>> 
>> 
>> each call is after the matrix and right hand side have been updated.
>> 
>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>> 
>> Is this expected behavior?
> 
>  No
>> 
>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>> 
>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>> 
>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
> 
>   Run your program for a few solves with the command line option -malloc and after each call call to KSPSolve() put in a call to PetscMallocDump(). Take a look at the output and email it to us (this is best done with one process if you can; does the memory problem happen with 1 MPI process?).

Hi Barry,

When I run with 1 MPI process, I get this error message:

0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Arguments are incompatible
[0]PETSC ERROR: MatMatMult requires A, mpiaij, to be compatible with B, seqaij
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: Configure options PETSC_DIR=/home/rmackie/PETSc/petsc-3.5.3 PETSC_ARCH=linux-gfortran-debug --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich
[0]PETSC ERROR: #1 MatMatMult() line 8710 in /home/rmackie/PETSc/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCGAMGOptprol_AGG() line 1328 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/agg.c
[0]PETSC ERROR: #3 PCSetUp_GAMG() line 606 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/gamg.c
[0]PETSC ERROR: #4 PCSetUp() line 902 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #5 KSPSetUp() line 306 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 KSPSolve() line 418 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c


Randy

> 
>   Barry
> 
>> 
>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>> 
>> 
>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>> 
>> 
>> Any other suggestions for tracking this down? I've run out of ideas.
>> 
>> 
>> Thanks in advance,
>> 
>> Randy
>> 
>> 
>> 
>> 
> 


From bsmith at mcs.anl.gov  Thu Mar  5 12:39:01 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Mar 2015 12:39:01 -0600
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
	<5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
Message-ID: <F6A25B86-AD4B-499C-B27C-7AAB24732B3A@mcs.anl.gov>


  This should not happen, probably you are working with a MPIAIJ matrix and GAMG incorrectly generates a SeqAIJ matrix internally and tries to use them together.  I'll try to reproduce 

  Barry

> On Mar 5, 2015, at 12:07 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
>> 
>> On Mar 4, 2015, at 7:30 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>> 
>>> In my application, I am repeatedly calling KSPSolve with the following options:
>>> 
>>> -ksp_type gmres \
>>> -pc_type gamg \
>>> -pc_gamg_type agg \
>>> -pc_gamg_agg_nsmooths 1\
>>> 
>>> 
>>> each call is after the matrix and right hand side have been updated.
>>> 
>>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>>> 
>>> Is this expected behavior?
>> 
>> No
>>> 
>>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>>> 
>>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>>> 
>>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>> 
>>  Run your program for a few solves with the command line option -malloc and after each call call to KSPSolve() put in a call to PetscMallocDump(). Take a look at the output and email it to us (this is best done with one process if you can; does the memory problem happen with 1 MPI process?).
> 
> Hi Barry,
> 
> When I run with 1 MPI process, I get this error message:
> 
> 0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: MatMatMult requires A, mpiaij, to be compatible with B, seqaij
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
> [0]PETSC ERROR: Configure options PETSC_DIR=/home/rmackie/PETSc/petsc-3.5.3 PETSC_ARCH=linux-gfortran-debug --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich
> [0]PETSC ERROR: #1 MatMatMult() line 8710 in /home/rmackie/PETSc/petsc-3.5.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCGAMGOptprol_AGG() line 1328 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/agg.c
> [0]PETSC ERROR: #3 PCSetUp_GAMG() line 606 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/gamg.c
> [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #5 KSPSetUp() line 306 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #6 KSPSolve() line 418 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> 
> 
> Randy
> 
>> 
>>  Barry
>> 
>>> 
>>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>>> 
>>> 
>>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>>> 
>>> 
>>> Any other suggestions for tracking this down? I've run out of ideas.
>>> 
>>> 
>>> Thanks in advance,
>>> 
>>> Randy


From jed at jedbrown.org  Thu Mar  5 13:17:20 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 05 Mar 2015 12:17:20 -0700
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <F6A25B86-AD4B-499C-B27C-7AAB24732B3A@mcs.anl.gov>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
	<5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
	<F6A25B86-AD4B-499C-B27C-7AAB24732B3A@mcs.anl.gov>
Message-ID: <8761afw8xr.fsf@jedbrown.org>

Barry Smith <bsmith at mcs.anl.gov> writes:

>   This should not happen, probably you are working with a MPIAIJ
>   matrix

Randall, it sounds like you are creating an MPIAIJ matrix even in
serial.  Meanwhile, GAMG creates a SeqAIJ prolongation operator (when
running in serial).  Why are you creating an MPIAIJ matrix in serial?
(Maybe you have MATMPIAIJ where it could be MATAIJ?)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/91c93bd6/attachment.pgp>

From jed at jedbrown.org  Thu Mar  5 13:28:58 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 05 Mar 2015 12:28:58 -0700
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <73AD8C9C-7C3C-4D83-808E-D82166D2D4CB@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
	<5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
	<F6A25B86-AD4B-499C-B27C-7AAB24732B3A@mcs.anl.gov>
	<8761afw8xr.fsf@jedbrown.org>
	<73AD8C9C-7C3C-4D83-808E-D82166D2D4CB@gmail.com>
Message-ID: <87385jw8ed.fsf@jedbrown.org>

Please keep the list Cc'd.

Randall Mackie <rlmackie862 at gmail.com> writes:

>> On Mar 5, 2015, at 11:17 AM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>> 
>>>  This should not happen, probably you are working with a MPIAIJ
>>>  matrix
>> 
>> Randall, it sounds like you are creating an MPIAIJ matrix even in
>> serial.  Meanwhile, GAMG creates a SeqAIJ prolongation operator (when
>> running in serial).  Why are you creating an MPIAIJ matrix in serial?
>> (Maybe you have MATMPIAIJ where it could be MATAIJ?)
>
> Well, I am creating an MPIAIJ matrix because that's what I've used for years, and it's never been a problem when I've tested on 1 process before, but then it's only been recently that I've been using GAMG for this particular solve.
>
> I will switch to MATAIJ and try again.

Thanks.  Avoiding over-specification is good.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/fc4e6d79/attachment.pgp>

From popov at uni-mainz.de  Thu Mar  5 14:18:45 2015
From: popov at uni-mainz.de (Anton)
Date: Thu, 5 Mar 2015 21:18:45 +0100
Subject: [petsc-users] near null space redundant
In-Reply-To: <CAMYG4Gnxo_SYAR7h0i9EU3-_P3fy8ygCWZThvOP73zmRkA_WDQ@mail.gmail.com>
References: <54F83783.2090109@uni-mainz.de>
	<CAMYG4Gnxo_SYAR7h0i9EU3-_P3fy8ygCWZThvOP73zmRkA_WDQ@mail.gmail.com>
Message-ID: <54F8BA25.2000800@uni-mainz.de>



On 3/5/15 4:18 PM, Matthew Knepley wrote:
> On Thu, Mar 5, 2015 at 5:01 AM, anton <popov at uni-mainz.de 
> <mailto:popov at uni-mainz.de>> wrote:
>
>     Hi,
>
>     Does the near-null-space get attached to the internal ksp object
>     of redundant preconditioner, or a user has to scatter and attach
>     it manually?
>     If so, what would be a suggested way to do that?
>
>
> Its not there, but would be easy to put in. I would either
>
>   a) Open an issue with the Bitbucket tracker, and someone will get to it
>
> or
>
>   b) Code it up in a pull request
>
> I think it would only be a few lines in PCSetUp_Redundant(). You check 
> whether the pc->mat has
> an attached near null space, and then use the VecScatters above to map 
> the vectors, and carry
> over the flag for constant null space.
>
>   Thanks,
>
>      Matt
Thanks Matt, I'll check this

Anton
>
>     Thanks,
>
>     Anton
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/14d5e218/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar  5 14:44:08 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Mar 2015 14:44:08 -0600
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<E8513B43-004A-4489-A7DA-B3DCFAA40D07@mcs.anl.gov>
	<5F040A96-A34F-4EB5-867B-384FC9EB0A0E@gmail.com>
Message-ID: <6EF17764-3F3A-42D4-BC10-395BD4C59238@mcs.anl.gov>


   I have fixed the bug of PCGAMG crashing on one process when used with MPIAIJ matrix in the branch barry/clean-gamg  and merged to next for testing.

   As Jed notes you can create your matrix as AIJ (which is recommended approach anyways) instead of MPIAIJ to work around the bug.

  Barry

> On Mar 5, 2015, at 12:07 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
>> 
>> On Mar 4, 2015, at 7:30 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>> 
>>> In my application, I am repeatedly calling KSPSolve with the following options:
>>> 
>>> -ksp_type gmres \
>>> -pc_type gamg \
>>> -pc_gamg_type agg \
>>> -pc_gamg_agg_nsmooths 1\
>>> 
>>> 
>>> each call is after the matrix and right hand side have been updated.
>>> 
>>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>>> 
>>> Is this expected behavior?
>> 
>> No
>>> 
>>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>>> 
>>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>>> 
>>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>> 
>>  Run your program for a few solves with the command line option -malloc and after each call call to KSPSolve() put in a call to PetscMallocDump(). Take a look at the output and email it to us (this is best done with one process if you can; does the memory problem happen with 1 MPI process?).
> 
> Hi Barry,
> 
> When I run with 1 MPI process, I get this error message:
> 
> 0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: MatMatMult requires A, mpiaij, to be compatible with B, seqaij
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
> [0]PETSC ERROR: Configure options PETSC_DIR=/home/rmackie/PETSc/petsc-3.5.3 PETSC_ARCH=linux-gfortran-debug --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich
> [0]PETSC ERROR: #1 MatMatMult() line 8710 in /home/rmackie/PETSc/petsc-3.5.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCGAMGOptprol_AGG() line 1328 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/agg.c
> [0]PETSC ERROR: #3 PCSetUp_GAMG() line 606 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/impls/gamg/gamg.c
> [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #5 KSPSetUp() line 306 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #6 KSPSolve() line 418 in /home/rmackie/PETSc/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> 
> 
> Randy
> 
>> 
>>  Barry
>> 
>>> 
>>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>>> 
>>> 
>>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>>> 
>>> 
>>> Any other suggestions for tracking this down? I've run out of ideas.
>>> 
>>> 
>>> Thanks in advance,
>>> 
>>> Randy


From gabel.fabian at gmail.com  Thu Mar  5 15:12:16 2015
From: gabel.fabian at gmail.com (Fabian Gabel)
Date: Thu, 05 Mar 2015 22:12:16 +0100
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <D298F42E-EE6F-40BD-A083-E178A38EDC10@mcs.anl.gov>
References: <1425338778.27237.18.camel@gmail.com>
	<87pp8r9dw2.fsf@jedbrown.org> <1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
	<1425352768.27237.88.camel@gmail.com>
	<D298F42E-EE6F-40BD-A083-E178A38EDC10@mcs.anl.gov>
Message-ID: <1425589936.3710.27.camel@gmail.com>

On Di, 2015-03-03 at 13:02 -0600, Barry Smith wrote:
> > On Mar 2, 2015, at 9:19 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> > 
> > On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
> >>  Do you really want tolerances:  relative=1e-90, absolute=1.10423, divergence=10000? That is an absolute tolerance of 1.1? Normally that would be huge.
> > 
> > I started using atol as convergence criterion with -ksp_norm_type
> > unpreconditioned. The value of atol gets updated every outer iteration.
> 
>    What is the "outer iteration" and why does it exist?

I used the term "outer iteration" as in Ferziger/Peric "Computational
Methods for Fluid Dynamics": 

"... there are two levels of iterations: inner iterations, within which
the linear equations are solved, and outer iterations, that deal with
the non-linearity and coupling of the equations."

I chose the Picard-Iteration approach to linearize the non-linear terms
of the Navier-Stokes equations. The solution process is as follows:

1) Update matrix coefficients for the linear system for (u,v,w,p)
2) Calculate the norm of the initial residual r^(n):=||b - Ax^(n) ||_2
3) if (r^(n)/r(0) <= 1-e8) GOTO 7)
4) Calculate atol = 0.1 * r^(n)
5) Call KSPSetTolerances with rtol=1e-90 and atol as above and solve the
coupled linear system for (u,v,w,p)
6) GOTO 1)
7) END

With an absolute tolerance for the norm of the unpreconditioned residual
I feel like I have control over the convergence of the picard iteration.
Using rtol for the norm of the preconditioned residual did not always
achieve convergence.

> 
> > 
> >>  You can provide your matrix with a block size that GAMG will use with MatSetBlockSize().
> > 
> > I think something went wrong. Setting the block size to 4 and solving
> > for (u,v,w,p) the convergence degraded significantly. I attached the
> > results for a smaller test case that shows the increase of the number of
> > needed inner iterations when setting the block size via
> > MatSetBlockSize().
> 
>   This is possible. 


Ok, so this concludes the experiments with out-of-the-box GAMG for the
fully coupled Navier-Stokes System? Why would the GAMG not perform well
on the linear system resulting from my discretization? What is the
difference between the algebraic multigrid implemented in PETSc and the
method used in the paper [1] I mentioned? 

I am still looking for a scalable solver configuration. So far ILU with
various levels of fill has the best single process performance, but
scalability could be better. I have been experimenting with different
PCFIELDSPLIT configurations but the best configuration so far does not
outperform ILU preconditioning.

Fabian

[1] http://dx.doi.org/10.1016/j.jcp.2008.08.027

> 
>   Barry
> 
> > 
> >> 
> >>  I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that it is using SOR for 27 points.
> >> 
> >> So you must have provided a null space since it printed "has attached null space"
> > 
> > The system has indeed a one dimensional null space (from the pressure
> > equation with Neumann boundary conditions). But now that you mentioned
> > it: It seems that the outer GMRES doesn't notice that the matrix has an
> > attached nullspace. Replacing
> > 
> > 	CALL MatSetNullSpace(CMAT,NULLSP,IERR)
> > 
> > with
> > 
> > 	CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)
> > 
> > solves this. What is wrong with using MatSetNullSpace?
> > 
> > 
> > Fabian
> > 
> > 
> >> 
> >>  Barry
> >> 
> >> 
> >> 
> >>> On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> >>> 
> >>> On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
> >>>> Fabian Gabel <gabel.fabian at gmail.com> writes:
> >>>> 
> >>>>> Dear PETSc Team,
> >>>>> 
> >>>>> I came across the following paragraph in your publication "Composable
> >>>>> Linear Solvers for Multiphysics" (2012):
> >>>>> 
> >>>>> "Rather than splitting the matrix into large blocks and
> >>>>> forming a preconditioner from solvers (for example, multi-
> >>>>> grid) on each block, one can perform multigrid on the entire
> >>>>> system, basing the smoother on solves coming from the tiny
> >>>>> blocks coupling the degrees of freedom at a single point (or
> >>>>> small number of points). This approach is also handled in
> >>>>> PETSc, but we will not elaborate on it here."
> >>>>> 
> >>>>> How would I use a multigrid preconditioner (GAMG) 
> >>>> 
> >>>> The heuristics in GAMG are not appropriate for indefinite/saddle-point
> >>>> systems such as arise from Navier-Stokes.  You can use geometric
> >>>> multigrid and use the fieldsplit techniques described in the paper as a
> >>>> smoother, for example.
> >>> 
> >>> I sadly don't have a solid background on multigrid methods, but as
> >>> mentioned in a previous thread
> >>> 
> >>> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
> >>> 
> >>> AMG has apparently been used (successfully?) for fully-coupled
> >>> finite-volume discretizations of Navier-Stokes:
> >>> 
> >>> http://dx.doi.org/10.1080/10407790.2014.894448
> >>> http://dx.doi.org/10.1016/j.jcp.2008.08.027
> >>> 
> >>> I was hoping to achieve something similar with the right configuration
> >>> of the PETSc preconditioners. So far I have only been using GAMG in a
> >>> straightforward manner, without providing any details on the structure
> >>> of the linear system. I attached the output of a test run with GAMG.
> >>> 
> >>>> 
> >>>>> from PETSc on linear systems of the form (after reordering the
> >>>>> variables):
> >>>>> 
> >>>>> [A_uu   0     0   A_up  A_uT]
> >>>>> [0    A_vv    0   A_vp  A_vT]
> >>>>> [0      0   A_ww  A_up  A_wT]
> >>>>> [A_pu A_pv  A_pw  A_pp   0  ]
> >>>>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> >>>>> 
> >>>>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
> >>>>> directly from a FVM discretization of the incompressible Navier-Stokes
> >>>>> equations and the temperature equation. The fifth row and column are
> >>>>> optional, depending on the method I choose to couple the temperature.
> >>>>> The Matrix is stored as one AIJ Matrix.
> >>>>> 
> >>>>> Regards,
> >>>>> Fabian Gabel
> >>> 
> >>> <cpld_0128.out.578677>
> >> 
> > 
> > <log.blocksize4><log.txt>
> 



From knepley at gmail.com  Thu Mar  5 15:19:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 5 Mar 2015 15:19:18 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <1425589936.3710.27.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
	<1425352768.27237.88.camel@gmail.com>
	<D298F42E-EE6F-40BD-A083-E178A38EDC10@mcs.anl.gov>
	<1425589936.3710.27.camel@gmail.com>
Message-ID: <CAMYG4G=y7mQerUpojMYQumhAf5m63ZOLiZEcrrhXxRZQ3KLHfA@mail.gmail.com>

On Thu, Mar 5, 2015 at 3:12 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:

> On Di, 2015-03-03 at 13:02 -0600, Barry Smith wrote:
> > > On Mar 2, 2015, at 9:19 PM, Fabian Gabel <gabel.fabian at gmail.com>
> wrote:
> > >
> > > On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
> > >>  Do you really want tolerances:  relative=1e-90, absolute=1.10423,
> divergence=10000? That is an absolute tolerance of 1.1? Normally that would
> be huge.
> > >
> > > I started using atol as convergence criterion with -ksp_norm_type
> > > unpreconditioned. The value of atol gets updated every outer iteration.
> >
> >    What is the "outer iteration" and why does it exist?
>
> I used the term "outer iteration" as in Ferziger/Peric "Computational
> Methods for Fluid Dynamics":
>
> "... there are two levels of iterations: inner iterations, within which
> the linear equations are solved, and outer iterations, that deal with
> the non-linearity and coupling of the equations."
>
> I chose the Picard-Iteration approach to linearize the non-linear terms
> of the Navier-Stokes equations. The solution process is as follows:
>
> 1) Update matrix coefficients for the linear system for (u,v,w,p)
> 2) Calculate the norm of the initial residual r^(n):=||b - Ax^(n) ||_2
> 3) if (r^(n)/r(0) <= 1-e8) GOTO 7)
> 4) Calculate atol = 0.1 * r^(n)
> 5) Call KSPSetTolerances with rtol=1e-90 and atol as above and solve the
> coupled linear system for (u,v,w,p)
> 6) GOTO 1)
> 7) END
>
> With an absolute tolerance for the norm of the unpreconditioned residual
> I feel like I have control over the convergence of the picard iteration.
> Using rtol for the norm of the preconditioned residual did not always
> achieve convergence.
>
> >
> > >
> > >>  You can provide your matrix with a block size that GAMG will use
> with MatSetBlockSize().
> > >
> > > I think something went wrong. Setting the block size to 4 and solving
> > > for (u,v,w,p) the convergence degraded significantly. I attached the
> > > results for a smaller test case that shows the increase of the number
> of
> > > needed inner iterations when setting the block size via
> > > MatSetBlockSize().
> >
> >   This is possible.
>
>
> Ok, so this concludes the experiments with out-of-the-box GAMG for the
> fully coupled Navier-Stokes System? Why would the GAMG not perform well
> on the linear system resulting from my discretization? What is the
> difference between the algebraic multigrid implemented in PETSc and the
> method used in the paper [1] I mentioned?
>

>From reading the abstract, these guys are doing SIMPLE, which is an
approximation
of the block systems generated by FieldSplit. This is explained in

  http://dl.acm.org/citation.cfm?id=1327899

You can use FieldSplit to get SIMPLE by using -ksp_type preonly -pc_type
jacobi in
the right places. I try and do this here:

  http://www.cs.uchicago.edu/~knepley/presentations/SIAMCSE13.pdf

Note that here AMG is only applied to the momentum equation.

  Matt


>
> I am still looking for a scalable solver configuration. So far ILU with
> various levels of fill has the best single process performance, but
> scalability could be better. I have been experimenting with different
> PCFIELDSPLIT configurations but the best configuration so far does not
> outperform ILU preconditioning.
>
> Fabian
>
> [1] http://dx.doi.org/10.1016/j.jcp.2008.08.027
>
> >
> >   Barry
> >
> > >
> > >>
> > >>  I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that
> it is using SOR for 27 points.
> > >>
> > >> So you must have provided a null space since it printed "has attached
> null space"
> > >
> > > The system has indeed a one dimensional null space (from the pressure
> > > equation with Neumann boundary conditions). But now that you mentioned
> > > it: It seems that the outer GMRES doesn't notice that the matrix has an
> > > attached nullspace. Replacing
> > >
> > >     CALL MatSetNullSpace(CMAT,NULLSP,IERR)
> > >
> > > with
> > >
> > >     CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)
> > >
> > > solves this. What is wrong with using MatSetNullSpace?
> > >
> > >
> > > Fabian
> > >
> > >
> > >>
> > >>  Barry
> > >>
> > >>
> > >>
> > >>> On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com>
> wrote:
> > >>>
> > >>> On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
> > >>>> Fabian Gabel <gabel.fabian at gmail.com> writes:
> > >>>>
> > >>>>> Dear PETSc Team,
> > >>>>>
> > >>>>> I came across the following paragraph in your publication
> "Composable
> > >>>>> Linear Solvers for Multiphysics" (2012):
> > >>>>>
> > >>>>> "Rather than splitting the matrix into large blocks and
> > >>>>> forming a preconditioner from solvers (for example, multi-
> > >>>>> grid) on each block, one can perform multigrid on the entire
> > >>>>> system, basing the smoother on solves coming from the tiny
> > >>>>> blocks coupling the degrees of freedom at a single point (or
> > >>>>> small number of points). This approach is also handled in
> > >>>>> PETSc, but we will not elaborate on it here."
> > >>>>>
> > >>>>> How would I use a multigrid preconditioner (GAMG)
> > >>>>
> > >>>> The heuristics in GAMG are not appropriate for
> indefinite/saddle-point
> > >>>> systems such as arise from Navier-Stokes.  You can use geometric
> > >>>> multigrid and use the fieldsplit techniques described in the paper
> as a
> > >>>> smoother, for example.
> > >>>
> > >>> I sadly don't have a solid background on multigrid methods, but as
> > >>> mentioned in a previous thread
> > >>>
> > >>>
> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
> > >>>
> > >>> AMG has apparently been used (successfully?) for fully-coupled
> > >>> finite-volume discretizations of Navier-Stokes:
> > >>>
> > >>> http://dx.doi.org/10.1080/10407790.2014.894448
> > >>> http://dx.doi.org/10.1016/j.jcp.2008.08.027
> > >>>
> > >>> I was hoping to achieve something similar with the right
> configuration
> > >>> of the PETSc preconditioners. So far I have only been using GAMG in a
> > >>> straightforward manner, without providing any details on the
> structure
> > >>> of the linear system. I attached the output of a test run with GAMG.
> > >>>
> > >>>>
> > >>>>> from PETSc on linear systems of the form (after reordering the
> > >>>>> variables):
> > >>>>>
> > >>>>> [A_uu   0     0   A_up  A_uT]
> > >>>>> [0    A_vv    0   A_vp  A_vT]
> > >>>>> [0      0   A_ww  A_up  A_wT]
> > >>>>> [A_pu A_pv  A_pw  A_pp   0  ]
> > >>>>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
> > >>>>>
> > >>>>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T},
> results
> > >>>>> directly from a FVM discretization of the incompressible
> Navier-Stokes
> > >>>>> equations and the temperature equation. The fifth row and column
> are
> > >>>>> optional, depending on the method I choose to couple the
> temperature.
> > >>>>> The Matrix is stored as one AIJ Matrix.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Fabian Gabel
> > >>>
> > >>> <cpld_0128.out.578677>
> > >>
> > >
> > > <log.blocksize4><log.txt>
> >
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/06223a76/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar  5 15:26:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Mar 2015 15:26:34 -0600
Subject: [petsc-users] Multigrid preconditioning of entire linear
	systems for discretized coupled multiphysics problems
In-Reply-To: <1425589936.3710.27.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
	<9DE27D9F-3188-4F3D-B18C-A74D7D9024A7@mcs.anl.gov>
	<1425352768.27237.88.camel@gmail.com>
	<D298F42E-EE6F-40BD-A083-E178A38EDC10@mcs.anl.gov>
	<1425589936.3710.27.camel@gmail.com>
Message-ID: <02290635-7655-440F-B7AA-ED95741896F3@mcs.anl.gov>


 From the abstract:  "by using an algebraic multigrid solver."    That's like saying "by using an iterative solver"; there are infinite possibilities of algebraic solvers.

   You should try using hypre's BoomerAMG algebraic solver  ./configure PETSc with --download-hypre and run with -pc_type hypre   to list the various boomeramg options run with -help and grep for boomeramg.

   Barry


  
> On Mar 5, 2015, at 3:12 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
> 
> On Di, 2015-03-03 at 13:02 -0600, Barry Smith wrote:
>>> On Mar 2, 2015, at 9:19 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
>>> 
>>> On Mo, 2015-03-02 at 19:43 -0600, Barry Smith wrote:
>>>> Do you really want tolerances:  relative=1e-90, absolute=1.10423, divergence=10000? That is an absolute tolerance of 1.1? Normally that would be huge.
>>> 
>>> I started using atol as convergence criterion with -ksp_norm_type
>>> unpreconditioned. The value of atol gets updated every outer iteration.
>> 
>>   What is the "outer iteration" and why does it exist?
> 
> I used the term "outer iteration" as in Ferziger/Peric "Computational
> Methods for Fluid Dynamics": 
> 
> "... there are two levels of iterations: inner iterations, within which
> the linear equations are solved, and outer iterations, that deal with
> the non-linearity and coupling of the equations."
> 
> I chose the Picard-Iteration approach to linearize the non-linear terms
> of the Navier-Stokes equations. The solution process is as follows:
> 
> 1) Update matrix coefficients for the linear system for (u,v,w,p)
> 2) Calculate the norm of the initial residual r^(n):=||b - Ax^(n) ||_2
> 3) if (r^(n)/r(0) <= 1-e8) GOTO 7)
> 4) Calculate atol = 0.1 * r^(n)
> 5) Call KSPSetTolerances with rtol=1e-90 and atol as above and solve the
> coupled linear system for (u,v,w,p)
> 6) GOTO 1)
> 7) END
> 
> With an absolute tolerance for the norm of the unpreconditioned residual
> I feel like I have control over the convergence of the picard iteration.
> Using rtol for the norm of the preconditioned residual did not always
> achieve convergence.
> 
>> 
>>> 
>>>> You can provide your matrix with a block size that GAMG will use with MatSetBlockSize().
>>> 
>>> I think something went wrong. Setting the block size to 4 and solving
>>> for (u,v,w,p) the convergence degraded significantly. I attached the
>>> results for a smaller test case that shows the increase of the number of
>>> needed inner iterations when setting the block size via
>>> MatSetBlockSize().
>> 
>>  This is possible. 
> 
> 
> Ok, so this concludes the experiments with out-of-the-box GAMG for the
> fully coupled Navier-Stokes System? Why would the GAMG not perform well
> on the linear system resulting from my discretization? What is the
> difference between the algebraic multigrid implemented in PETSc and the
> method used in the paper [1] I mentioned? 
> 
> I am still looking for a scalable solver configuration. So far ILU with
> various levels of fill has the best single process performance, but
> scalability could be better. I have been experimenting with different
> PCFIELDSPLIT configurations but the best configuration so far does not
> outperform ILU preconditioning.
> 
> Fabian
> 
> [1] http://dx.doi.org/10.1016/j.jcp.2008.08.027
> 
>> 
>>  Barry
>> 
>>> 
>>>> 
>>>> I would use coupledsolve_mg_coarse_sub_pc_type lu   it is weird that it is using SOR for 27 points.
>>>> 
>>>> So you must have provided a null space since it printed "has attached null space"
>>> 
>>> The system has indeed a one dimensional null space (from the pressure
>>> equation with Neumann boundary conditions). But now that you mentioned
>>> it: It seems that the outer GMRES doesn't notice that the matrix has an
>>> attached nullspace. Replacing
>>> 
>>> 	CALL MatSetNullSpace(CMAT,NULLSP,IERR)
>>> 
>>> with
>>> 
>>> 	CALL KSPSetNullSpace(KRYLOV,NULLSP,IERR)
>>> 
>>> solves this. What is wrong with using MatSetNullSpace?
>>> 
>>> 
>>> Fabian
>>> 
>>> 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On Mar 2, 2015, at 6:39 PM, Fabian Gabel <gabel.fabian at gmail.com> wrote:
>>>>> 
>>>>> On Mo, 2015-03-02 at 16:29 -0700, Jed Brown wrote:
>>>>>> Fabian Gabel <gabel.fabian at gmail.com> writes:
>>>>>> 
>>>>>>> Dear PETSc Team,
>>>>>>> 
>>>>>>> I came across the following paragraph in your publication "Composable
>>>>>>> Linear Solvers for Multiphysics" (2012):
>>>>>>> 
>>>>>>> "Rather than splitting the matrix into large blocks and
>>>>>>> forming a preconditioner from solvers (for example, multi-
>>>>>>> grid) on each block, one can perform multigrid on the entire
>>>>>>> system, basing the smoother on solves coming from the tiny
>>>>>>> blocks coupling the degrees of freedom at a single point (or
>>>>>>> small number of points). This approach is also handled in
>>>>>>> PETSc, but we will not elaborate on it here."
>>>>>>> 
>>>>>>> How would I use a multigrid preconditioner (GAMG) 
>>>>>> 
>>>>>> The heuristics in GAMG are not appropriate for indefinite/saddle-point
>>>>>> systems such as arise from Navier-Stokes.  You can use geometric
>>>>>> multigrid and use the fieldsplit techniques described in the paper as a
>>>>>> smoother, for example.
>>>>> 
>>>>> I sadly don't have a solid background on multigrid methods, but as
>>>>> mentioned in a previous thread
>>>>> 
>>>>> http://lists.mcs.anl.gov/pipermail/petsc-users/2015-February/024219.html
>>>>> 
>>>>> AMG has apparently been used (successfully?) for fully-coupled
>>>>> finite-volume discretizations of Navier-Stokes:
>>>>> 
>>>>> http://dx.doi.org/10.1080/10407790.2014.894448
>>>>> http://dx.doi.org/10.1016/j.jcp.2008.08.027
>>>>> 
>>>>> I was hoping to achieve something similar with the right configuration
>>>>> of the PETSc preconditioners. So far I have only been using GAMG in a
>>>>> straightforward manner, without providing any details on the structure
>>>>> of the linear system. I attached the output of a test run with GAMG.
>>>>> 
>>>>>> 
>>>>>>> from PETSc on linear systems of the form (after reordering the
>>>>>>> variables):
>>>>>>> 
>>>>>>> [A_uu   0     0   A_up  A_uT]
>>>>>>> [0    A_vv    0   A_vp  A_vT]
>>>>>>> [0      0   A_ww  A_up  A_wT]
>>>>>>> [A_pu A_pv  A_pw  A_pp   0  ]
>>>>>>> [A_Tu A_Tv  A_Tw  A_Tp  A_TT]
>>>>>>> 
>>>>>>> where each of the block matrices A_ij, with i,j in {u,v,w,p,T}, results
>>>>>>> directly from a FVM discretization of the incompressible Navier-Stokes
>>>>>>> equations and the temperature equation. The fifth row and column are
>>>>>>> optional, depending on the method I choose to couple the temperature.
>>>>>>> The Matrix is stored as one AIJ Matrix.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Fabian Gabel
>>>>> 
>>>>> <cpld_0128.out.578677>
>>>> 
>>> 
>>> <log.blocksize4><log.txt>
>> 
> 
> 


From grischa.jacobs at email.de  Thu Mar  5 15:29:20 2015
From: grischa.jacobs at email.de (Grischa Jacobs)
Date: Thu, 5 Mar 2015 22:29:20 +0100
Subject: [petsc-users] VTK Output / no file
In-Reply-To: <87bnk98zo7.fsf@jedbrown.org>
References: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>,
	<87bnk98zo7.fsf@jedbrown.org>
Message-ID: <trinity-d46b81fa-aeb8-407a-bbc3-b0cf8f352363-1425590960645@3capp-webde-bs01>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/4c2bed8e/attachment-0001.html>

From jed at jedbrown.org  Thu Mar  5 15:34:35 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 05 Mar 2015 14:34:35 -0700
Subject: [petsc-users] VTK Output / no file
In-Reply-To: <trinity-d46b81fa-aeb8-407a-bbc3-b0cf8f352363-1425590960645@3capp-webde-bs01>
References: <trinity-0ff37942-c69e-43dc-87e5-54b60e9b3665-1425397785827@3capp-webde-bs33>
	<87bnk98zo7.fsf@jedbrown.org>
	<trinity-d46b81fa-aeb8-407a-bbc3-b0cf8f352363-1425590960645@3capp-webde-bs01>
Message-ID: <87k2yvuo0k.fsf@jedbrown.org>

Grischa Jacobs <grischa.jacobs at email.de> writes:

> Hi,
>  
> I found the problem. PetscViewerVTKOpen will only work for DMVectors not for
> natural Vectors.
>  
> This doesn't work:
> ierr = DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE,
> DM_BOUNDARY_NONE, DMDA_STENCIL_BOX, 10, 10, 10, PETSC_DECIDE, PETSC_DECIDE,
> PETSC_DECIDE, 1, 1,NULL,NULL,NULL,&test);CHKERRQ(ierr);
>
> Vec a;
> DMDACreateNaturalVector(test,&a);CHKERRQ(ierr);
> ierr = PetscViewerVTKOpen
> (PETSC_COMM_WORLD,"yourfile_natural.vts",FILE_MODE_WRITE, &viewer1);CHKERRQ(
> ierr);
> ierr = VecView(a, viewer1);CHKERRQ(ierr);
> ierr = PetscViewerDestroy(&viewer1);CHKERRQ(ierr);
>
>  
>
> This does work:
> ierr = DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE,
> DM_BOUNDARY_NONE, DMDA_STENCIL_BOX, 10, 10, 10, PETSC_DECIDE, PETSC_DECIDE,
> PETSC_DECIDE, 1, 1,NULL,NULL,NULL,&test);CHKERRQ(ierr);
> Vec a;
> DMCreateGlobalVector(test,&a);
> ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"yourfile_dm.vts",FILE_MODE_WRITE, &
> viewer1);CHKERRQ(ierr);
> ierr = VecView(a, viewer1);CHKERRQ(ierr);
> ierr = PetscViewerDestroy(&viewer1);CHKERRQ(ierr);
>
> Can anyone confirm this?

Sounds accurate since it was never written for natural vectors.  I guess
we should make it error in that case?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/fd81b76f/attachment.pgp>

From david.knezevic at akselos.com  Thu Mar  5 15:51:44 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 5 Mar 2015 16:51:44 -0500
Subject: [petsc-users] Near nullspace for preconditioners other than GAMG?
Message-ID: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>

Hi all,

I found that using MatNullSpaceCreateRigidBody as per example 49
<http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex49.c.html>
works well with GAMG for a standard elasticity problem, so that's great.

I was just wondering if its possible to achieve the same sort of thing with
other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace does
nothing for external solvers like hypre, right?

Thanks,
David

 <http://www.akselos.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/509e27f4/attachment.html>

From jed at jedbrown.org  Thu Mar  5 16:08:49 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 05 Mar 2015 15:08:49 -0700
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
Message-ID: <87h9tzumfi.fsf@jedbrown.org>

David Knezevic <david.knezevic at akselos.com> writes:
> I was just wondering if its possible to achieve the same sort of thing with
> other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace does
> nothing for external solvers like hypre, right?

It is used by ML (smoothed aggregation), but not BoomerAMG (classical
AMG) which uses an algorithm that doesn't have a natural place for such
information.  To my knowledge, classical AMG is not widely used for
elasticity.  It is very robust for M-matrices.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/813f0af0/attachment.pgp>

From david.knezevic at akselos.com  Thu Mar  5 16:27:29 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 5 Mar 2015 17:27:29 -0500
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <87h9tzumfi.fsf@jedbrown.org>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
Message-ID: <CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>

OK, got it, thanks!

David


On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:

> David Knezevic <david.knezevic at akselos.com> writes:
> > I was just wondering if its possible to achieve the same sort of thing
> with
> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
> does
> > nothing for external solvers like hypre, right?
>
> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
> AMG) which uses an algorithm that doesn't have a natural place for such
> information.  To my knowledge, classical AMG is not widely used for
> elasticity.  It is very robust for M-matrices.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/ac2ffc1a/attachment.html>

From jed at jedbrown.org  Thu Mar  5 16:33:34 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 05 Mar 2015 15:33:34 -0700
Subject: [petsc-users] Multigrid preconditioning of entire linear
	systems for discretized coupled multiphysics problems
In-Reply-To: <1425343174.27237.60.camel@gmail.com>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com>
Message-ID: <87d24nula9.fsf@jedbrown.org>

Fabian Gabel <gabel.fabian at gmail.com> writes:

> AMG has apparently been used (successfully?) for fully-coupled
> finite-volume discretizations of Navier-Stokes:
>
> http://dx.doi.org/10.1080/10407790.2014.894448
> http://dx.doi.org/10.1016/j.jcp.2008.08.027

These papers effectively bring a pressure projection or SIMPLE-type
algorithm into a coupled system.  It's not a saddle-point formulation
and is not uniformly valid across parameters (I think; methods of this
sort normally aren't, but I haven't analyzed it).  The multigrid
algorithm is not described and might not be purely algebraic.

Note that the 2014 paper copies liberally from the 2009 paper.

You can try formulating Navier-Stokes this way and test existing solvers
if you like, but I predict it doesn't come without tradeoffs.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/c3d56f37/attachment.pgp>

From bsmith at mcs.anl.gov  Thu Mar  5 16:41:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Mar 2015 16:41:46 -0600
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
Message-ID: <ADC6F827-307A-4C7D-9B5A-D80CBDDF9AAC@mcs.anl.gov>


  Randy,

   I've not been able to reproduce this; let us know if get to the point of having something we can run and debug.

  Barry

> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> In my application, I am repeatedly calling KSPSolve with the following options:
> 
> -ksp_type gmres \
> -pc_type gamg \
> -pc_gamg_type agg \
> -pc_gamg_agg_nsmooths 1\
> 
> 
> each call is after the matrix and right hand side have been updated.
> 
> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
> 
> Is this expected behavior?
> 
> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
> 
> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
> 
> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
> 
> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
> 
> 
> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
> 
> 
> Any other suggestions for tracking this down? I've run out of ideas.
> 
> 
> Thanks in advance,
> 
> Randy
> 
> 
> 
> 


From jchang27 at uh.edu  Thu Mar  5 18:12:52 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Thu, 5 Mar 2015 18:12:52 -0600
Subject: [petsc-users] log_summary MPI
Message-ID: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>

Hi all,

I apologize if these may be simple questions that could be found somewhere.
In the log_summary profiling, I have a few questions about the MPI
messages/MPI message lengths metrics:

1) For MPI messages, does the value under Max correspond to the maximum
number of messages a single processor passes?

2) What are the units of MPI message lengths? Is it in bytes?

3) What does MPI message lengths refer to exactly? Like, does it refer to
the max/min/average size per sent/received message or does it refer to the
cumulative size that a single processor sends/receives?

Thanks

-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150305/3367c9bb/attachment.html>

From knepley at gmail.com  Fri Mar  6 09:10:25 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 6 Mar 2015 09:10:25 -0600
Subject: [petsc-users] log_summary MPI
In-Reply-To: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>
References: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>
Message-ID: <CAMYG4G=JtwPKxL2FZEGtydTqwVktThCOStd5OYZHY1w1dfPidA@mail.gmail.com>

On Thu, Mar 5, 2015 at 6:12 PM, Justin Chang <jchang27 at uh.edu> wrote:

> Hi all,
>
> I apologize if these may be simple questions that could be found
> somewhere. In the log_summary profiling, I have a few questions about the
> MPI messages/MPI message lengths metrics:
>
> 1) For MPI messages, does the value under Max correspond to the maximum
> number of messages a single processor passes?
>

Yes.


> 2) What are the units of MPI message lengths? Is it in bytes?
>

Yes, returned by PetscMPITypeSize().


> 3) What does MPI message lengths refer to exactly? Like, does it refer to
> the max/min/average size per sent/received message or does it refer to the
> cumulative size that a single processor sends/receives?
>

Cumulative, see


https://bitbucket.org/petsc/petsc/src/d8151eeaff97562eb317e17b7b0cecab16831f69/include/petsclog.h?at=master#cl-343

  Thanks,

     Matt


> Thanks
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/714ee89c/attachment-0001.html>

From knepley at gmail.com  Fri Mar  6 10:48:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 6 Mar 2015 10:48:39 -0600
Subject: [petsc-users] Local/global numbering to natural numbering?
In-Reply-To: <CAP2=TMio_K=WatVNvs_stO_QMU=VkDTtZP1JAy6XRpmfjVqqsg@mail.gmail.com>
References: <CAP2=TMiB_=pv1cnxRzAhuSgGT1f12gbvnnCzWstimQTG1ryjrQ@mail.gmail.com>
	<CAP2=TMio_K=WatVNvs_stO_QMU=VkDTtZP1JAy6XRpmfjVqqsg@mail.gmail.com>
Message-ID: <CAMYG4GnERRsf43GH+ZXVfcQA173puJbWvin2LmUHPmpJXgx6pg@mail.gmail.com>

You will finally get this soon. There is a postdoc at LSU writing the code
to output in original, or "natural", numbering the same way that DMDA does
now.

  Thanks,

     Matt

On Sun, Nov 9, 2014 at 4:33 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Or would PetscSF sort of give you that information? Say if I had auxiliary
> data associated for each vertex point, I would first distribute the mesh
> via DMPlexDistribute, obtain the PetscSF from that routine, and then call
> DMPlexDistributeField for the vector containing said auxiliary data?
>
> Thanks,
> Justin
>
> On Sun, Nov 9, 2014 at 4:10 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hi all,
>>
>> Given the local/global numbering of vector/array entries generated after
>> DMPlexDistribute(), is it possible to obtain the "natural numbering"? That
>> is, the numbering of the vertex points before the mesh was distributed. I
>> see that this has been discussed thoroughly for DMDA's but is it possible
>> to obtain some sort of natural numbering for DMPlex?
>>
>> For instance, when I call DMPlexCreateFromDAG() or
>> DMPlexCreateFromCellList() I would consider vertexCoords[] to be of
>> "natural numbering" since it's the original ordering specified by the mesh
>> generator.
>>
>> Thanks,
>> Justin
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/e819d793/attachment.html>

From rlmackie862 at gmail.com  Fri Mar  6 12:10:01 2015
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Fri, 6 Mar 2015 10:10:01 -0800
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <ADC6F827-307A-4C7D-9B5A-D80CBDDF9AAC@mcs.anl.gov>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<ADC6F827-307A-4C7D-9B5A-D80CBDDF9AAC@mcs.anl.gov>
Message-ID: <A6F7D4CA-78E4-4265-8132-4B1147BF2112@gmail.com>

Hi Barry,

I've followed your suggestion to use PetscMallocDump after the call to KSPSolve and I've looked at the output.

What I find is that for a smaller sized model that I can run on my laptop, I don't see any increase in the memory after it is allocated the first pass through.

However, with this other much larger model, I do clearly see an increase in memory after each call to that particular KSPSolve:

[0]Total space allocated 1885177504 bytes
[0]Total space allocated 2034781200 bytes
[0]Total space allocated 2184383136 bytes
[0]Total space allocated 2333985072 bytes
[0]Total space allocated 2483587008 bytes


That model is too large to run on just 1 process, but I've got it down to 8 processes, and each process shows a similar increase in memory.

Just to confirm that it was happening in that KSPSolve call, I put an additional PetscMallocDump before and after, and I can confirm that it is clearly increasing the memory after the call to KSPSolve.

I've tried sorting through the output of that malloc dump, but the file is 4 Mbytes in size for process 0, so I've tried grepping for various things, like GAMG, etc, and I don't yet see what is causing the increase in memory.

Do you guys have any sophisticated scripts to parse the output with?

If you'd like the output file, I can put it somewhere, I just don't want to drop a 4 Mbyte file in everyone's inbox.

As another datapoint, I did save the matrix and vector to a binary file using Vec/Mat View, then wrote a little test program that read them in (Vec/Mat Load) and did the KSPSolve. I did not see any memory issues with that.

So, I am unsure what to do at this point, or how to proceed to track this down.

Any suggestions appreciated.


Randy


> On Mar 5, 2015, at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Randy,
> 
>   I've not been able to reproduce this; let us know if get to the point of having something we can run and debug.
> 
>  Barry
> 
>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> In my application, I am repeatedly calling KSPSolve with the following options:
>> 
>> -ksp_type gmres \
>> -pc_type gamg \
>> -pc_gamg_type agg \
>> -pc_gamg_agg_nsmooths 1\
>> 
>> 
>> each call is after the matrix and right hand side have been updated.
>> 
>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>> 
>> Is this expected behavior?
>> 
>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>> 
>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>> 
>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>> 
>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>> 
>> 
>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>> 
>> 
>> Any other suggestions for tracking this down? I've run out of ideas.
>> 
>> 
>> Thanks in advance,
>> 
>> Randy
>> 
>> 
>> 
>> 
> 


From bsmith at mcs.anl.gov  Fri Mar  6 12:35:47 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 6 Mar 2015 12:35:47 -0600
Subject: [petsc-users] Question about GAMG and memory use
In-Reply-To: <A6F7D4CA-78E4-4265-8132-4B1147BF2112@gmail.com>
References: <288CF918-350E-4265-9B6C-6A20D5FDB9D3@gmail.com>
	<ADC6F827-307A-4C7D-9B5A-D80CBDDF9AAC@mcs.anl.gov>
	<A6F7D4CA-78E4-4265-8132-4B1147BF2112@gmail.com>
Message-ID: <27C7F72B-FCF1-41E3-9B59-AB3AA8495C03@mcs.anl.gov>


  Randy,

  Just email me the PetscMallocDump output file directly.

  Barry

> On Mar 6, 2015, at 12:10 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> Hi Barry,
> 
> I've followed your suggestion to use PetscMallocDump after the call to KSPSolve and I've looked at the output.
> 
> What I find is that for a smaller sized model that I can run on my laptop, I don't see any increase in the memory after it is allocated the first pass through.
> 
> However, with this other much larger model, I do clearly see an increase in memory after each call to that particular KSPSolve:
> 
> [0]Total space allocated 1885177504 bytes
> [0]Total space allocated 2034781200 bytes
> [0]Total space allocated 2184383136 bytes
> [0]Total space allocated 2333985072 bytes
> [0]Total space allocated 2483587008 bytes
> 
> 
> That model is too large to run on just 1 process, but I've got it down to 8 processes, and each process shows a similar increase in memory.
> 
> Just to confirm that it was happening in that KSPSolve call, I put an additional PetscMallocDump before and after, and I can confirm that it is clearly increasing the memory after the call to KSPSolve.
> 
> I've tried sorting through the output of that malloc dump, but the file is 4 Mbytes in size for process 0, so I've tried grepping for various things, like GAMG, etc, and I don't yet see what is causing the increase in memory.
> 
> Do you guys have any sophisticated scripts to parse the output with?
> 
> If you'd like the output file, I can put it somewhere, I just don't want to drop a 4 Mbyte file in everyone's inbox.
> 
> As another datapoint, I did save the matrix and vector to a binary file using Vec/Mat View, then wrote a little test program that read them in (Vec/Mat Load) and did the KSPSolve. I did not see any memory issues with that.
> 
> So, I am unsure what to do at this point, or how to proceed to track this down.
> 
> Any suggestions appreciated.
> 
> 
> Randy
> 
> 
>> On Mar 5, 2015, at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Randy,
>> 
>>  I've not been able to reproduce this; let us know if get to the point of having something we can run and debug.
>> 
>> Barry
>> 
>>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>> 
>>> In my application, I am repeatedly calling KSPSolve with the following options:
>>> 
>>> -ksp_type gmres \
>>> -pc_type gamg \
>>> -pc_gamg_type agg \
>>> -pc_gamg_agg_nsmooths 1\
>>> 
>>> 
>>> each call is after the matrix and right hand side have been updated.
>>> 
>>> This works well in the sense that it solves the system in a reasonable number of steps, however, I have noticed that the memory footprint of the application increases by about 500 Mbytes after each call to KSPSolve (this is a big problem), and after several calls, I've maxed out the memory.
>>> 
>>> Is this expected behavior?
>>> 
>>> I've combed through my code looking to make sure I don't have any memory leaks, and so far I haven't found any (doesn't mean there are not there).
>>> 
>>> However, when I use another PC, like jacobi, just to compare, I don't see this memory issue, or if I comment out that call to KSPSolve (there is a lot of other stuff going on in the code besides this call), I don't see this issue.
>>> 
>>> I've tried to destroy the KSP after each solve and recreate it each time, but there still seems to be some memory getting added.
>>> 
>>> I've tried to distill this down to a smaller problem and test program, but so far I have been unsuccessful.
>>> 
>>> 
>>> Is there a way to completely release the memory associated with the GAMG preconditioner after a call to KSPSolve?
>>> 
>>> 
>>> Any other suggestions for tracking this down? I've run out of ideas.
>>> 
>>> 
>>> Thanks in advance,
>>> 
>>> Randy
>>> 
>>> 
>>> 
>>> 
>> 
> 


From jchang27 at uh.edu  Fri Mar  6 12:56:52 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Fri, 6 Mar 2015 12:56:52 -0600
Subject: [petsc-users] log_summary MPI
In-Reply-To: <CAMYG4G=JtwPKxL2FZEGtydTqwVktThCOStd5OYZHY1w1dfPidA@mail.gmail.com>
References: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>
	<CAMYG4G=JtwPKxL2FZEGtydTqwVktThCOStd5OYZHY1w1dfPidA@mail.gmail.com>
Message-ID: <CAP2=TMibsQSNjMFmWhFzQTuE3sYC4ncSCA4s_Oxp4xCJyhb2yw@mail.gmail.com>

Matt, thank you for the response. One more question, is there a way to
output only the MPI messages and MPI message lengths? I don't want to print
everything when I do -log_summary, so are there any command line options
that do this? I can't seem to find anything in the manual that discusses
this.

Thanks,

On Fri, Mar 6, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Mar 5, 2015 at 6:12 PM, Justin Chang <jchang27 at uh.edu> wrote:
>
>> Hi all,
>>
>> I apologize if these may be simple questions that could be found
>> somewhere. In the log_summary profiling, I have a few questions about the
>> MPI messages/MPI message lengths metrics:
>>
>> 1) For MPI messages, does the value under Max correspond to the maximum
>> number of messages a single processor passes?
>>
>
> Yes.
>
>
>> 2) What are the units of MPI message lengths? Is it in bytes?
>>
>
> Yes, returned by PetscMPITypeSize().
>
>
>> 3) What does MPI message lengths refer to exactly? Like, does it refer to
>> the max/min/average size per sent/received message or does it refer to the
>> cumulative size that a single processor sends/receives?
>>
>
> Cumulative, see
>
>
> https://bitbucket.org/petsc/petsc/src/d8151eeaff97562eb317e17b7b0cecab16831f69/include/petsclog.h?at=master#cl-343
>
>   Thanks,
>
>      Matt
>
>
>> Thanks
>>
>> --
>> Justin Chang
>> PhD Candidate, Civil Engineering - Computational Sciences
>> University of Houston, Department of Civil and Environmental Engineering
>> Houston, TX 77004
>> (512) 963-3262
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/29134dc3/attachment.html>

From knepley at gmail.com  Fri Mar  6 13:06:41 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 6 Mar 2015 13:06:41 -0600
Subject: [petsc-users] log_summary MPI
In-Reply-To: <CAP2=TMibsQSNjMFmWhFzQTuE3sYC4ncSCA4s_Oxp4xCJyhb2yw@mail.gmail.com>
References: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>
	<CAMYG4G=JtwPKxL2FZEGtydTqwVktThCOStd5OYZHY1w1dfPidA@mail.gmail.com>
	<CAP2=TMibsQSNjMFmWhFzQTuE3sYC4ncSCA4s_Oxp4xCJyhb2yw@mail.gmail.com>
Message-ID: <CAMYG4Gny091qwQp7SkCctPMsfg0A6zWSDCm2ZYK2Ww6cdjk2rw@mail.gmail.com>

On Fri, Mar 6, 2015 at 12:56 PM, Justin Chang <jchang27 at uh.edu> wrote:

> Matt, thank you for the response. One more question, is there a way to
> output only the MPI messages and MPI message lengths? I don't want to print
> everything when I do -log_summary, so are there any command line options
> that do this? I can't seem to find anything in the manual that discusses
> this.
>

These are just global counters:


https://bitbucket.org/petsc/petsc/src/1ddf9febf6a13a173c13b6b0dde02ed9ccfd590e/src/sys/logging/plog.c?at=master#cl-51

You can output them whenever you want. If you want global numbers, you will
need to call reductions yourself.

  Thanks,

    Matt


> Thanks,
>
> On Fri, Mar 6, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Mar 5, 2015 at 6:12 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>
>>> Hi all,
>>>
>>> I apologize if these may be simple questions that could be found
>>> somewhere. In the log_summary profiling, I have a few questions about the
>>> MPI messages/MPI message lengths metrics:
>>>
>>> 1) For MPI messages, does the value under Max correspond to the maximum
>>> number of messages a single processor passes?
>>>
>>
>> Yes.
>>
>>
>>> 2) What are the units of MPI message lengths? Is it in bytes?
>>>
>>
>> Yes, returned by PetscMPITypeSize().
>>
>>
>>> 3) What does MPI message lengths refer to exactly? Like, does it refer
>>> to the max/min/average size per sent/received message or does it refer to
>>> the cumulative size that a single processor sends/receives?
>>>
>>
>> Cumulative, see
>>
>>
>> https://bitbucket.org/petsc/petsc/src/d8151eeaff97562eb317e17b7b0cecab16831f69/include/petsclog.h?at=master#cl-343
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks
>>>
>>> --
>>> Justin Chang
>>> PhD Candidate, Civil Engineering - Computational Sciences
>>> University of Houston, Department of Civil and Environmental Engineering
>>> Houston, TX 77004
>>> (512) 963-3262
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/8fd884f0/attachment.html>

From fdkong.jd at gmail.com  Fri Mar  6 19:13:05 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Fri, 6 Mar 2015 18:13:05 -0700
Subject: [petsc-users] is it possible to gather a distributed matrix to a
	subset of processors?
Message-ID: <CAN5Wd-LyhiaX_QFk=NsQXAvTzNXxYpjZHqOsi0vobQtnyf1k3g@mail.gmail.com>

Hi all,

I want to solve a coarse problem on a subset of processors, but the matrix
already has been distributed across all processors. The coarse problem is
not small enough to solve using a single core. I want to solve this coarse
problem using superLU on a small number of processors. Is it possible to
gather a distributed matrix to a subset of processors?

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/9c9d1a5c/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri Mar  6 19:30:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 6 Mar 2015 19:30:46 -0600
Subject: [petsc-users] is it possible to gather a distributed matrix to
	a subset of processors?
In-Reply-To: <CAN5Wd-LyhiaX_QFk=NsQXAvTzNXxYpjZHqOsi0vobQtnyf1k3g@mail.gmail.com>
References: <CAN5Wd-LyhiaX_QFk=NsQXAvTzNXxYpjZHqOsi0vobQtnyf1k3g@mail.gmail.com>
Message-ID: <843B8A5B-8467-4622-8711-DCD183F9DC36@mcs.anl.gov>


  MatCreateRedundantMatrix()

  Or you can use PCREDUNDANT -pc_type redundant see the manual page that automatically handles the gather of the matrix onto the smaller number of processes and setting up and solving the system.

  For example see the rules in src/ksp/ksp/examples/tutorials/makefile (there is nothing special about MUMPS it can be done with SuperLU_Dist and even iterative solvers).

NSUBCOMM = 8 7 6 5 4 3 2 1
runex10_mumps_redundant:
	- at touch ex10_mumps_redundant.tmp
	- at for nsubcomm in ${NSUBCOMM}; do \
	${MPIEXEC} -n 8 ./ex10 -f0 ${DATAFILESPATH}/matrices/medium -ksp_type preonly -pc_type redundant -pc_redundant_number $$nsubcomm -redundant_pc_factor_mat_solver_package mumps -num_numfac 2 -num_rhs 2 >> ex10_mumps_redundant.tmp 2>&1; \
	done;
	- at if (${DIFF} output/ex10_mumps_redundant.out ex10_mumps_redundant.tmp) then true; \
	   else printf "${PWD}\nPossible problem with ex10_mumps_redundant, diffs above\n=========================================\n"; fi; \
	${RM} -f ex10_mumps_redundant.tmp;



  Barry
 


> On Mar 6, 2015, at 7:13 PM, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> Hi all,
> 
> I want to solve a coarse problem on a subset of processors, but the matrix already has been distributed across all processors. The coarse problem is not small enough to solve using a single core. I want to solve this coarse problem using superLU on a small number of processors. Is it possible to gather a distributed matrix to a subset of processors?
> 
> Thanks,
> 
> Fande,


From jchang27 at uh.edu  Fri Mar  6 20:48:17 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Fri, 6 Mar 2015 20:48:17 -0600
Subject: [petsc-users] log_summary MPI
In-Reply-To: <CAMYG4Gny091qwQp7SkCctPMsfg0A6zWSDCm2ZYK2Ww6cdjk2rw@mail.gmail.com>
References: <CAP2=TMgyWOHYhyo4UVXJaKOpkizxH5dGkCRKSY5E0dbD9CCxTA@mail.gmail.com>
	<CAMYG4G=JtwPKxL2FZEGtydTqwVktThCOStd5OYZHY1w1dfPidA@mail.gmail.com>
	<CAP2=TMibsQSNjMFmWhFzQTuE3sYC4ncSCA4s_Oxp4xCJyhb2yw@mail.gmail.com>
	<CAMYG4Gny091qwQp7SkCctPMsfg0A6zWSDCm2ZYK2Ww6cdjk2rw@mail.gmail.com>
Message-ID: <CAP2=TMhC+J+5g9e3_eEnixoUOLhPqU5wzvfeu9m8SQhu1MdYmg@mail.gmail.com>

Okay got it, thank you very much

On Fri, Mar 6, 2015 at 1:06 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Mar 6, 2015 at 12:56 PM, Justin Chang <jchang27 at uh.edu> wrote:
>
>> Matt, thank you for the response. One more question, is there a way to
>> output only the MPI messages and MPI message lengths? I don't want to print
>> everything when I do -log_summary, so are there any command line options
>> that do this? I can't seem to find anything in the manual that discusses
>> this.
>>
>
> These are just global counters:
>
>
> https://bitbucket.org/petsc/petsc/src/1ddf9febf6a13a173c13b6b0dde02ed9ccfd590e/src/sys/logging/plog.c?at=master#cl-51
>
> You can output them whenever you want. If you want global numbers, you
> will need to call reductions yourself.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>>
>> On Fri, Mar 6, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Mar 5, 2015 at 6:12 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I apologize if these may be simple questions that could be found
>>>> somewhere. In the log_summary profiling, I have a few questions about the
>>>> MPI messages/MPI message lengths metrics:
>>>>
>>>> 1) For MPI messages, does the value under Max correspond to the maximum
>>>> number of messages a single processor passes?
>>>>
>>>
>>> Yes.
>>>
>>>
>>>> 2) What are the units of MPI message lengths? Is it in bytes?
>>>>
>>>
>>> Yes, returned by PetscMPITypeSize().
>>>
>>>
>>>> 3) What does MPI message lengths refer to exactly? Like, does it refer
>>>> to the max/min/average size per sent/received message or does it refer to
>>>> the cumulative size that a single processor sends/receives?
>>>>
>>>
>>> Cumulative, see
>>>
>>>
>>> https://bitbucket.org/petsc/petsc/src/d8151eeaff97562eb317e17b7b0cecab16831f69/include/petsclog.h?at=master#cl-343
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks
>>>>
>>>> --
>>>> Justin Chang
>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>> University of Houston, Department of Civil and Environmental Engineering
>>>> Houston, TX 77004
>>>> (512) 963-3262
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Justin Chang
>> PhD Candidate, Civil Engineering - Computational Sciences
>> University of Houston, Department of Civil and Environmental Engineering
>> Houston, TX 77004
>> (512) 963-3262
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150306/2537f6f8/attachment.html>

From Lukasz.Kaczmarczyk at glasgow.ac.uk  Sat Mar  7 07:28:23 2015
From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk)
Date: Sat, 7 Mar 2015 13:28:23 +0000
Subject: [petsc-users] petsc 3.5.3, Time Stepping, printing -help, bug
Message-ID: <24E529E3-24B3-4F9F-B3FA-4BB94618D9B2@glasgow.ac.uk>

Hello, 

Printing -help I get following error. I believe that bug is here,

[0]PETSC ERROR: #4 PetscOptionsRealArray_Private() line 1306 in /opt/petsc/src/sys/objects/aoptions.c
ierr = (*PetscHelpPrintf)(PetscOptionsObject->comm,"  -%s%s <%G",PetscOptionsObject->prefix?PetscOptionsObject->prefix:"",opt+1,value[0]);CHKERRQ(ierr); 

My git commit:
git rev-parse HEAD
ee900cc92187a7889ca4f268c65f36fb91826ff2

Pleas see following error trial:

[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: %G format is no longer supported, use %g and cast the argument to double
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-1524-gee900cc  GIT Date: 2015-01-31 17:44:15 -0600
[0]PETSC ERROR: /mofem_build/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-linux2-c-debug named likask by root Sat Mar  7 13:07:58 2015
[0]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-zoltan=1 --download-blacs=1 --download-moab = 1 --download-ptscotch=1 --with-hdf5=1 --with-hdf5-dir=/usr --download-netcdf=1 --with-shared-libraries=1
[0]PETSC ERROR: #1 PetscFormatConvert() line 67 in /opt/petsc/src/sys/fileio/mprint.c
[0]PETSC ERROR: #2 PetscVFPrintfDefault() line 192 in /opt/petsc/src/sys/fileio/mprint.c
[0]PETSC ERROR: #3 PetscHelpPrintfDefault() line 601 in /opt/petsc/src/sys/fileio/mprint.c
[0]PETSC ERROR: #4 PetscOptionsRealArray_Private() line 1306 in /opt/petsc/src/sys/objects/aoptions.c
[0]PETSC ERROR: #5 TSAdaptSetFromOptions_Basic() line 83 in /opt/petsc/src/ts/adapt/impls/basic/adaptbasic.c
[0]PETSC ERROR: #6 TSAdaptSetFromOptions() line 392 in /opt/petsc/src/ts/adapt/interface/tsadapt.c
[0]PETSC ERROR: #7 TSSetFromOptions() line 332 in /opt/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #8 main() line 579 in /mnt/home/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp

[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -help
[0]PETSC ERROR: -ksp_atol 1e-10
[0]PETSC ERROR: -ksp_rtol 1e-10
[0]PETSC ERROR: -ksp_type fgmres
[0]PETSC ERROR: -my_disp_order 3
[0]PETSC ERROR: -my_file LShape.cub
[0]PETSC ERROR: -my_max_post_proc_ref_level 0
[0]PETSC ERROR: -my_output_prt 1
[0]PETSC ERROR: -my_time_data_file data.txt
[0]PETSC ERROR: -my_vel_order 3
[0]PETSC ERROR: -pc_factor_mat_solver_package superlu_dist
[0]PETSC ERROR: -pc_type lu
[0]PETSC ERROR: -snes_atol 1e-7
[0]PETSC ERROR: -snes_linesearch_type basic
[0]PETSC ERROR: -snes_max_it 100
[0]PETSC ERROR: -snes_monitor
[0]PETSC ERROR: -snes_rtol 1e-7
[0]PETSC ERROR: -snes_type newtonls
[0]PETSC ERROR: -ts_dt 0.001
[0]PETSC ERROR: -ts_final_time 0.002
[0]PETSC ERROR: -ts_monitor
[0]PETSC ERROR: -ts_type beuler


Kind regards,
Lukasz


Lukasz Kaczmarczyk

"Prediction is very difficult, especially about the future." -- Niels Bohr
"... there are dark corners in the Bourne shell, and people use all of them." --Chet Ramey

HomePage:
http://www.gla.ac.uk/schools/engineering/staff/lukaszkaczmarczyk/

MoFEM JosePH (Finite Element System) 
* https://en.wikipedia.org/wiki/MoFEM_JosePH
* https://bitbucket.org/likask/mofem-joseph/wiki/Home




From bsmith at mcs.anl.gov  Sat Mar  7 11:37:11 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Mar 2015 11:37:11 -0600
Subject: [petsc-users] petsc 3.5.3, Time Stepping, printing -help, bug
In-Reply-To: <24E529E3-24B3-4F9F-B3FA-4BB94618D9B2@glasgow.ac.uk>
References: <24E529E3-24B3-4F9F-B3FA-4BB94618D9B2@glasgow.ac.uk>
Message-ID: <C8CF5196-45E8-4FC7-8646-D4505C9B1845@mcs.anl.gov>


  Thanks for reporting this. I just fixed it a couple of days ago in a branch that will soon get into master

  Barry

> On Mar 7, 2015, at 7:28 AM, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> 
> Hello, 
> 
> Printing -help I get following error. I believe that bug is here,
> 
> [0]PETSC ERROR: #4 PetscOptionsRealArray_Private() line 1306 in /opt/petsc/src/sys/objects/aoptions.c
> ierr = (*PetscHelpPrintf)(PetscOptionsObject->comm,"  -%s%s <%G",PetscOptionsObject->prefix?PetscOptionsObject->prefix:"",opt+1,value[0]);CHKERRQ(ierr); 
> 
> My git commit:
> git rev-parse HEAD
> ee900cc92187a7889ca4f268c65f36fb91826ff2
> 
> Pleas see following error trial:
> 
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: %G format is no longer supported, use %g and cast the argument to double
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-1524-gee900cc  GIT Date: 2015-01-31 17:44:15 -0600
> [0]PETSC ERROR: /mofem_build/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-linux2-c-debug named likask by root Sat Mar  7 13:07:58 2015
> [0]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-zoltan=1 --download-blacs=1 --download-moab = 1 --download-ptscotch=1 --with-hdf5=1 --with-hdf5-dir=/usr --download-netcdf=1 --with-shared-libraries=1
> [0]PETSC ERROR: #1 PetscFormatConvert() line 67 in /opt/petsc/src/sys/fileio/mprint.c
> [0]PETSC ERROR: #2 PetscVFPrintfDefault() line 192 in /opt/petsc/src/sys/fileio/mprint.c
> [0]PETSC ERROR: #3 PetscHelpPrintfDefault() line 601 in /opt/petsc/src/sys/fileio/mprint.c
> [0]PETSC ERROR: #4 PetscOptionsRealArray_Private() line 1306 in /opt/petsc/src/sys/objects/aoptions.c
> [0]PETSC ERROR: #5 TSAdaptSetFromOptions_Basic() line 83 in /opt/petsc/src/ts/adapt/impls/basic/adaptbasic.c
> [0]PETSC ERROR: #6 TSAdaptSetFromOptions() line 392 in /opt/petsc/src/ts/adapt/interface/tsadapt.c
> [0]PETSC ERROR: #7 TSSetFromOptions() line 332 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #8 main() line 579 in /mnt/home/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
> 
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -help
> [0]PETSC ERROR: -ksp_atol 1e-10
> [0]PETSC ERROR: -ksp_rtol 1e-10
> [0]PETSC ERROR: -ksp_type fgmres
> [0]PETSC ERROR: -my_disp_order 3
> [0]PETSC ERROR: -my_file LShape.cub
> [0]PETSC ERROR: -my_max_post_proc_ref_level 0
> [0]PETSC ERROR: -my_output_prt 1
> [0]PETSC ERROR: -my_time_data_file data.txt
> [0]PETSC ERROR: -my_vel_order 3
> [0]PETSC ERROR: -pc_factor_mat_solver_package superlu_dist
> [0]PETSC ERROR: -pc_type lu
> [0]PETSC ERROR: -snes_atol 1e-7
> [0]PETSC ERROR: -snes_linesearch_type basic
> [0]PETSC ERROR: -snes_max_it 100
> [0]PETSC ERROR: -snes_monitor
> [0]PETSC ERROR: -snes_rtol 1e-7
> [0]PETSC ERROR: -snes_type newtonls
> [0]PETSC ERROR: -ts_dt 0.001
> [0]PETSC ERROR: -ts_final_time 0.002
> [0]PETSC ERROR: -ts_monitor
> [0]PETSC ERROR: -ts_type beuler
> 
> 
> Kind regards,
> Lukasz
> 
> 
> Lukasz Kaczmarczyk
> 
> "Prediction is very difficult, especially about the future." -- Niels Bohr
> "... there are dark corners in the Bourne shell, and people use all of them." --Chet Ramey
> 
> HomePage:
> http://www.gla.ac.uk/schools/engineering/staff/lukaszkaczmarczyk/
> 
> MoFEM JosePH (Finite Element System) 
> * https://en.wikipedia.org/wiki/MoFEM_JosePH
> * https://bitbucket.org/likask/mofem-joseph/wiki/Home
> 
> 
> 


From fdkong.jd at gmail.com  Sat Mar  7 11:53:49 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Sat, 7 Mar 2015 10:53:49 -0700
Subject: [petsc-users] a line of code is weird
Message-ID: <CAN5Wd-J0SM4+W_PYVHLp5BX2Pc8ezG0Zwm9K36Ayt_aasJgVOA@mail.gmail.com>

Hi all,

I am trying to understand how to compute Jacobian using finite difference
methods. There is a function called SNESComputeJacobianDefault, where a
line of code at 120th
line, MatSetValues(B,1,&j,1,&i,y+j-start,INSERT_VALUES), is difficult to
understand for me. Because I think inserting  'y' is enough, but there is
two extra values 'j-start'. Any explanation is welcome.

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150307/c3f3e6d7/attachment.html>

From bsmith at mcs.anl.gov  Sat Mar  7 12:11:32 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Mar 2015 12:11:32 -0600
Subject: [petsc-users] a line of code is weird
In-Reply-To: <CAN5Wd-J0SM4+W_PYVHLp5BX2Pc8ezG0Zwm9K36Ayt_aasJgVOA@mail.gmail.com>
References: <CAN5Wd-J0SM4+W_PYVHLp5BX2Pc8ezG0Zwm9K36Ayt_aasJgVOA@mail.gmail.com>
Message-ID: <C0C7DF27-B062-4C82-8210-BF174E8B9D5F@mcs.anl.gov>


 Note the lines immediately above it.

 ierr = VecGetOwnershipRange(x1,&start,&end);CHKERRQ(ierr);
.....
for (j=start; j<end; j++) {
       if (PetscAbsScalar(y[j-start]) > amax || j == i) {
        ierr = MatSetValues(B,1,&j,1,&i,y+j-start,INSERT_VALUES);CHKERRQ(ierr);

 j is looping over the global rows of the vector. The -start is to convert to the local row inside the array y[] since y only contains the local part of the vector. 

Note that y+j-start is equivalent to &y[j-start] so this is inserting into the jth global row the jth global row value in the vector which is the j-start local entry in the vector.

Barry


> On Mar 7, 2015, at 11:53 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> Hi all,
> 
> I am trying to understand how to compute Jacobian using finite difference methods. There is a function called SNESComputeJacobianDefault, where a line of code at 120th line, MatSetValues(B,1,&j,1,&i,y+j-start,INSERT_VALUES), is difficult to understand for me. Because I think inserting  'y' is enough, but there is two extra values 'j-start'. Any explanation is welcome.
> 
> Thanks,
> 
> Fande,


From hus003 at ucsd.edu  Sat Mar  7 13:17:49 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sat, 7 Mar 2015 19:17:49 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010EA0FC@XMAIL-MBX-BH1.AD.UCSD.EDU>

Here is one more problem along this line. The matrix A created in this way may not have a natural global numbering. Let's say, if we are on grid point (i,j,k), the element in A with row (i,j,k) and column 1 is not necessarily at row position (i+(j+k*my)*mx globally. I mean, how do I set the correspondence between the MatStencils and int for the index? Should I use something like ISLocalToGlobalMapping? 

Best,
Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Wednesday, March 04, 2015 10:53 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i.

  Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
>
> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i.

   PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply.

   To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);

   Barry


>> DMDAGetLocalInfo



>
> Hui
>
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Wednesday, March 04, 2015 5:27 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>  If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
>
>  Barry
>
>
>
>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>>
>> Should I somehow use the following functions?
>> MatCreate
>> DMDAGetLocalInfo
>> ISLocalToGlobalMappingCreate
>> MatSetLocalToGlobalMapping
>> MatGetLocalSubMatrix
>>
>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>>
>> Best,
>> Hui
>>
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Sunday, March 01, 2015 9:24 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>
>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>
>>> Thank you Barry. I have yet two more questions:
>>>
>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>>
>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>>
>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>>
>>>
>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>>
>>> Best,
>>> Hui
>>>
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Saturday, February 28, 2015 9:35 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>
>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>
>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>>
>>>> Is it true that
>>>> 1) Only DMDA can use mg?
>>>
>>> No this is not true
>>>
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>
>>> No you do not have to
>>>
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>
>>>  Yes you can.
>>>
>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>>
>>> More details
>>>
>>>> 1) Only DMDA can use mg?
>>>
>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>>
>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>
>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>>
>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>
>>>  As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>>
>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>>
>>> Barry
>>>
>>>>
>>>> Best,
>>>> Hui
>>>>
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>
>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>
>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>>
>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>>
>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>>
>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>>
>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>>
>>>> No it handles dof > 1 fine.
>>>>
>>>> Send your code.
>>>>
>>>> Barry
>>>>
>>>>>
>>>>> Thank you!

From bsmith at mcs.anl.gov  Sat Mar  7 14:17:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Mar 2015 14:17:00 -0600
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010EA0FC@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010EA0FC@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <CE4A7535-BF1D-4E21-8066-51FA1BFE4708@mcs.anl.gov>


> On Mar 7, 2015, at 1:17 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Here is one more problem along this line. The matrix A created in this way may not have a natural global numbering. Let's say, if we are on grid point (i,j,k), the element in A with row (i,j,k) and column 1 is not necessarily at row position (i+(j+k*my)*mx globally. I mean, how do I set the correspondence between the MatStencils and int for the index? Should I use something like ISLocalToGlobalMapping? 

  You can.

   Look at the routines in PETSc that form the matrices like DMCreateMatrix_DA_3d_MPIAIJ() and modify it for the matrix you need.

   BTW: what is the matrix you a creating for. PETSc also creates matrices that interpolate between levels of DMDA like DMCreateInterpolation_DA_3D_Q1() it's structure might be useful for you.

  Barry


> 
> Best,
> Hui
> 
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Wednesday, March 04, 2015 10:53 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
>> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>> 
>> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i.
> 
>  Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
>> 
>> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i.
> 
>   PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply.
> 
>   To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);
> 
>   Barry
> 
> 
>>> DMDAGetLocalInfo
> 
> 
> 
>> 
>> Hui
>> 
>> 
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Wednesday, March 04, 2015 5:27 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>> 
>> If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
>> 
>> Barry
>> 
>> 
>> 
>>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>> 
>>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>>> 
>>> Should I somehow use the following functions?
>>> MatCreate
>>> DMDAGetLocalInfo
>>> ISLocalToGlobalMappingCreate
>>> MatSetLocalToGlobalMapping
>>> MatGetLocalSubMatrix
>>> 
>>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>>> 
>>> Best,
>>> Hui
>>> 
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Sunday, March 01, 2015 9:24 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>> 
>>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>> 
>>>> Thank you Barry. I have yet two more questions:
>>>> 
>>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>>> 
>>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>>> 
>>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>>> 
>>>> 
>>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>>> 
>>>> Best,
>>>> Hui
>>>> 
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Saturday, February 28, 2015 9:35 AM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>> 
>>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>> 
>>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>>> 
>>>>> Is it true that
>>>>> 1) Only DMDA can use mg?
>>>> 
>>>> No this is not true
>>>> 
>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>> 
>>>> No you do not have to
>>>> 
>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>> 
>>>> Yes you can.
>>>> 
>>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>>> 
>>>> More details
>>>> 
>>>>> 1) Only DMDA can use mg?
>>>> 
>>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>>> 
>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>> 
>>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>>> 
>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>> 
>>>> As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>>> 
>>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> Best,
>>>>> Hui
>>>>> 
>>>>> ________________________________________
>>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>>> To: Sun, Hui
>>>>> Cc: petsc-users at mcs.anl.gov
>>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>> 
>>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>> 
>>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>>> 
>>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>>> 
>>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>>> 
>>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>>> 
>>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>>> 
>>>>> No it handles dof > 1 fine.
>>>>> 
>>>>> Send your code.
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> Thank you!


From gtheler at cites-gss.com  Sat Mar  7 15:38:06 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Sat, 7 Mar 2015 21:38:06 +0000
Subject: [petsc-users] saving views as images
Message-ID: <C638F8EEDA1EC642B0B13905A2ADAC5C3A1354@SS10175.sancorseguros.net>

Hi

Last week I asked how to dump a matrix structure into an image and Barry kindly pointed me out the  PetscDrawSetSave function. But I cannot figure out how to use it. So far, I can show my matrix structure into an X window with something like

    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
    MatView(milonga.R, viewer);

I tried with this snippet of code but had no luck:

    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
    MatView(milonga.R, viewer);
    PetscViewerDrawGetDraw(viewer, 0, &draw);
    PetscDrawFlush(draw);
    PetscDrawSetSave(draw, "test.gif", PETSC_FALSE);
    PetscDrawSetSaveFinalImage(draw, "test2.gif");

The result is that test.gif and test2.gif are created but they are empty directories.
What am I missing?

--
jeremy

________________________________
 Imprima este mensaje s?lo si es absolutamente necesario.
Para imprimir, en lo posible utilice el papel de ambos lados.
El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.



************AVISO DE CONFIDENCIALIDAD************

El Grupo Sancor Seguros comunica que:

Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.

Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.

From bsmith at mcs.anl.gov  Sat Mar  7 15:51:19 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Mar 2015 15:51:19 -0600
Subject: [petsc-users] saving views as images
In-Reply-To: <C638F8EEDA1EC642B0B13905A2ADAC5C3A1354@SS10175.sancorseguros.net>
References: <C638F8EEDA1EC642B0B13905A2ADAC5C3A1354@SS10175.sancorseguros.net>
Message-ID: <66F34E87-B7B5-4B31-BD98-664C3CB8A37E@mcs.anl.gov>


 Did you follow the directions in the manual page for PetscDrawSetSave()? 

 "Requires that PETSc be configured with the option --with-afterimage to save the images and ffmpeg must be in your path to make the movie"

 So you have to install afterimage on your machine and then ./configure PETSc so it knows about it. If you have done all that and PETSc is being linked with the afterimage libraries then run your code with -info and send all the output; it may indicate why the images are not being saved.

  Barry


> On Mar 7, 2015, at 3:38 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> Hi
> 
> Last week I asked how to dump a matrix structure into an image and Barry kindly pointed me out the  PetscDrawSetSave function. But I cannot figure out how to use it. So far, I can show my matrix structure into an X window with something like
> 
>    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
>    MatView(milonga.R, viewer);
> 
> I tried with this snippet of code but had no luck:
> 
>    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
>    MatView(milonga.R, viewer);
>    PetscViewerDrawGetDraw(viewer, 0, &draw);
>    PetscDrawFlush(draw);
>    PetscDrawSetSave(draw, "test.gif", PETSC_FALSE);
>    PetscDrawSetSaveFinalImage(draw, "test2.gif");
> 
> The result is that test.gif and test2.gif are created but they are empty directories.
> What am I missing?
> 
> --
> jeremy
> 
> ________________________________
> Imprima este mensaje s?lo si es absolutamente necesario.
> Para imprimir, en lo posible utilice el papel de ambos lados.
> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> 
> 
> 
> ************AVISO DE CONFIDENCIALIDAD************
> 
> El Grupo Sancor Seguros comunica que:
> 
> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> 
> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.


From mfadams at lbl.gov  Sat Mar  7 16:40:25 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 7 Mar 2015 17:40:25 -0500
Subject: [petsc-users] Multigrid preconditioning of entire linear
 systems for discretized coupled multiphysics problems
In-Reply-To: <87d24nula9.fsf@jedbrown.org>
References: <1425338778.27237.18.camel@gmail.com> <87pp8r9dw2.fsf@jedbrown.org>
	<1425343174.27237.60.camel@gmail.com> <87d24nula9.fsf@jedbrown.org>
Message-ID: <CADOhEh7AY1Q-dKERirNRqmrr_=8d862Y-ZgNuOjNpACn0VVoDg@mail.gmail.com>

It sounds like AMG is used on the whole system. I've done this for a
different saddle point system but it required enough algorithm extension
that I wrote a paper on it.  These methods are not in GAMG.

I would at least not use smoothed aggregation, use plane aggregation
(-pc_gamg_agg_nsmooths 0).  It might work if it has a stable MG smoother.

Mark

@Article{,
  author =   {Adams, M.~F.},
  title =   {Algebraic multrigrid methods for constrained linear systems
with applications to contact problems in solid mechanics},
  journal =   {Numerical Linear Algebra with Applications},
  year =   {2004},
  volume =   {11},
  number =   {2-3},
  pages =   {141-153}
}

On Thu, Mar 5, 2015 at 5:33 PM, Jed Brown <jed at jedbrown.org> wrote:

> Fabian Gabel <gabel.fabian at gmail.com> writes:
>
> > AMG has apparently been used (successfully?) for fully-coupled
> > finite-volume discretizations of Navier-Stokes:
> >
> > http://dx.doi.org/10.1080/10407790.2014.894448
> > http://dx.doi.org/10.1016/j.jcp.2008.08.027
>
> These papers effectively bring a pressure projection or SIMPLE-type
> algorithm into a coupled system.  It's not a saddle-point formulation
> and is not uniformly valid across parameters (I think; methods of this
> sort normally aren't, but I haven't analyzed it).  The multigrid
> algorithm is not described and might not be purely algebraic.
>
> Note that the 2014 paper copies liberally from the 2009 paper.
>
> You can try formulating Navier-Stokes this way and test existing solvers
> if you like, but I predict it doesn't come without tradeoffs.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150307/1070e34f/attachment.html>

From mfadams at lbl.gov  Sat Mar  7 16:57:34 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 7 Mar 2015 17:57:34 -0500
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
Message-ID: <CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>

FYI, st?ben used classical AMG for elasticity but he has articulated his
code for elasticity more than Hypre as I understand it.  Hypre can work OK
for elasticity in my experience.  Its worth a try.

Mark

On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> OK, got it, thanks!
>
> David
>
>
> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> David Knezevic <david.knezevic at akselos.com> writes:
>> > I was just wondering if its possible to achieve the same sort of thing
>> with
>> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
>> does
>> > nothing for external solvers like hypre, right?
>>
>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>> AMG) which uses an algorithm that doesn't have a natural place for such
>> information.  To my knowledge, classical AMG is not widely used for
>> elasticity.  It is very robust for M-matrices.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150307/0b5db74d/attachment.html>

From david.knezevic at akselos.com  Sat Mar  7 17:27:07 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Sat, 7 Mar 2015 18:27:07 -0500
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
Message-ID: <CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>

OK, thanks for letting me know.

I've tried GAMG and ML with MatNullSpaceCreateRigidBody and those both work
well for me. (I tried ML after Jed pointed out that ML also uses the near
nullspace.)

I have also tried hypre but the convergence hasn't been as good for the
elasticity models I've been using, though I may not have been setting the
hypre options in an optimal way for elasticity.

Another situation I've been meaning to ask about: If the elasticity model
includes some "near rigid" regions (very high Young's modulus) all
iterative solvers I've tried fare poorly. I guess it's because the highly
contrasting stiffnesses give a large condition number. Is there a standard
way to get the preconditioner compensate for this?

Thanks,
David



On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:

> FYI, st?ben used classical AMG for elasticity but he has articulated his
> code for elasticity more than Hypre as I understand it.  Hypre can work OK
> for elasticity in my experience.  Its worth a try.
>
> Mark
>
> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <david.knezevic at akselos.com
> > wrote:
>
>> OK, got it, thanks!
>>
>> David
>>
>>
>> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> David Knezevic <david.knezevic at akselos.com> writes:
>>> > I was just wondering if its possible to achieve the same sort of thing
>>> with
>>> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
>>> does
>>> > nothing for external solvers like hypre, right?
>>>
>>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>>> AMG) which uses an algorithm that doesn't have a natural place for such
>>> information.  To my knowledge, classical AMG is not widely used for
>>> elasticity.  It is very robust for M-matrices.
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150307/90297390/attachment.html>

From bsmith at mcs.anl.gov  Sat Mar  7 17:52:20 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Mar 2015 17:52:20 -0600
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
Message-ID: <FFF16029-02AB-480E-B4AF-06AB11A4D086@mcs.anl.gov>


> On Mar 7, 2015, at 5:27 PM, David Knezevic <david.knezevic at akselos.com> wrote:
> 
> OK, thanks for letting me know.
> 
> I've tried GAMG and ML with MatNullSpaceCreateRigidBody and those both work well for me. (I tried ML after Jed pointed out that ML also uses the near nullspace.)
> 
> I have also tried hypre but the convergence hasn't been as good for the elasticity models I've been using, though I may not have been setting the hypre options in an optimal way for elasticity.
> 
> Another situation I've been meaning to ask about: If the elasticity model includes some "near rigid" regions (very high Young's modulus) all iterative solvers I've tried fare poorly. I guess it's because the highly contrasting stiffnesses give a large condition number. Is there a standard way to get the preconditioner compensate for this?

  Off the top of my head:   a "domain decomposition" answer might be to use a sparse direct solver on that region; of course how to tie that into the rest of the regions and the iterative solver is a broad question. 

   Maybe in the context of AMG all the nodes in that region could be retained in the coarser grids (assuming it is a relatively small part of the domain) and so all those points end up in the coarse grid which is solved with a sparse direct solver?  We don't currently have an interface for you to indicate nodes you want to keep in the coarse grid; Mark would have know if this makes any sense at all.

   Barry

> 
> Thanks,
> David
> 
> 
> 
> On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
> FYI, st?ben used classical AMG for elasticity but he has articulated his code for elasticity more than Hypre as I understand it.  Hypre can work OK for elasticity in my experience.  Its worth a try.
> 
> Mark
> 
> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <david.knezevic at akselos.com> wrote:
> OK, got it, thanks!
> 
> David
> 
> 
> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
> David Knezevic <david.knezevic at akselos.com> writes:
> > I was just wondering if its possible to achieve the same sort of thing with
> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace does
> > nothing for external solvers like hypre, right?
> 
> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
> AMG) which uses an algorithm that doesn't have a natural place for such
> information.  To my knowledge, classical AMG is not widely used for
> elasticity.  It is very robust for M-matrices.
> 
> 
> 


From fdkong.jd at gmail.com  Sat Mar  7 18:35:47 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Sat, 7 Mar 2015 17:35:47 -0700
Subject: [petsc-users] a line of code is weird
In-Reply-To: <C0C7DF27-B062-4C82-8210-BF174E8B9D5F@mcs.anl.gov>
References: <CAN5Wd-J0SM4+W_PYVHLp5BX2Pc8ezG0Zwm9K36Ayt_aasJgVOA@mail.gmail.com>
	<C0C7DF27-B062-4C82-8210-BF174E8B9D5F@mcs.anl.gov>
Message-ID: <CAN5Wd-K26-Kd8Lazja+0_XkboKxmX8BburyN4xaXYrWo4gG7hA@mail.gmail.com>

Thanks, Barry.

I understood now.



On Sat, Mar 7, 2015 at 11:11 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  Note the lines immediately above it.
>
>  ierr = VecGetOwnershipRange(x1,&start,&end);CHKERRQ(ierr);
> .....
> for (j=start; j<end; j++) {
>        if (PetscAbsScalar(y[j-start]) > amax || j == i) {
>         ierr =
> MatSetValues(B,1,&j,1,&i,y+j-start,INSERT_VALUES);CHKERRQ(ierr);
>
>  j is looping over the global rows of the vector. The -start is to convert
> to the local row inside the array y[] since y only contains the local part
> of the vector.
>
> Note that y+j-start is equivalent to &y[j-start] so this is inserting into
> the jth global row the jth global row value in the vector which is the
> j-start local entry in the vector.
>
> Barry
>
>
> > On Mar 7, 2015, at 11:53 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
> >
> > Hi all,
> >
> > I am trying to understand how to compute Jacobian using finite
> difference methods. There is a function called SNESComputeJacobianDefault,
> where a line of code at 120th line,
> MatSetValues(B,1,&j,1,&i,y+j-start,INSERT_VALUES), is difficult to
> understand for me. Because I think inserting  'y' is enough, but there is
> two extra values 'j-start'. Any explanation is welcome.
> >
> > Thanks,
> >
> > Fande,
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150307/82a6d7c0/attachment.html>

From hus003 at ucsd.edu  Sat Mar  7 19:20:15 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sun, 8 Mar 2015 01:20:15 +0000
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <CE4A7535-BF1D-4E21-8066-51FA1BFE4708@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<,<12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010EA0FC@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<CE4A7535-BF1D-4E21-8066-51FA1BFE4708@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010EA11B@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you, Barry. 

In DMCreateMatrix_DA_3d_MPIAIJ, there are two lines which I think are crucial: 
ierr = MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
ierr = MatSetLocalToGlobalMapping(J,ltog,ltog);CHKERRQ(ierr);

For my case, I just need the ltog for the rows, and my columns are sequential, what shall I do here? I want to get a rltog for rows and a cltog for columns. There is no question that I get rltog from da, but where do I get the cltog? 

It is not for interpolation between levels of DMDA. The matrix is for some quantity (of dimension M) that doesn't live on the grid (of size n^3), but depends on some other quantity (of dimension N=n^3, with 1<<M<<N) which lives on the grid. 

Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Saturday, March 07, 2015 12:17 PM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver

> On Mar 7, 2015, at 1:17 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Here is one more problem along this line. The matrix A created in this way may not have a natural global numbering. Let's say, if we are on grid point (i,j,k), the element in A with row (i,j,k) and column 1 is not necessarily at row position (i+(j+k*my)*mx globally. I mean, how do I set the correspondence between the MatStencils and int for the index? Should I use something like ISLocalToGlobalMapping?

  You can.

   Look at the routines in PETSc that form the matrices like DMCreateMatrix_DA_3d_MPIAIJ() and modify it for the matrix you need.

   BTW: what is the matrix you a creating for. PETSc also creates matrices that interpolate between levels of DMDA like DMCreateInterpolation_DA_3D_Q1() it's structure might be useful for you.

  Barry


>
> Best,
> Hui
>
>
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Wednesday, March 04, 2015 10:53 AM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>
>> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>
>> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i.
>
>  Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
>>
>> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i.
>
>   PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply.
>
>   To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);
>
>   Barry
>
>
>>> DMDAGetLocalInfo
>
>
>
>>
>> Hui
>>
>>
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Wednesday, March 04, 2015 5:27 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>
>> If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
>>
>> Barry
>>
>>
>>
>>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>
>>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>>>
>>> Should I somehow use the following functions?
>>> MatCreate
>>> DMDAGetLocalInfo
>>> ISLocalToGlobalMappingCreate
>>> MatSetLocalToGlobalMapping
>>> MatGetLocalSubMatrix
>>>
>>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>>>
>>> Best,
>>> Hui
>>>
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Sunday, March 01, 2015 9:24 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>
>>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>
>>>> Thank you Barry. I have yet two more questions:
>>>>
>>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>>>
>>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>>>
>>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>>>
>>>>
>>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>>>
>>>> Best,
>>>> Hui
>>>>
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Saturday, February 28, 2015 9:35 AM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>
>>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>
>>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>>>
>>>>> Is it true that
>>>>> 1) Only DMDA can use mg?
>>>>
>>>> No this is not true
>>>>
>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>>
>>>> No you do not have to
>>>>
>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>>
>>>> Yes you can.
>>>>
>>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>>>
>>>> More details
>>>>
>>>>> 1) Only DMDA can use mg?
>>>>
>>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>>>
>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>>
>>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>>>
>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>>
>>>> As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>>>
>>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>>>
>>>> Barry
>>>>
>>>>>
>>>>> Best,
>>>>> Hui
>>>>>
>>>>> ________________________________________
>>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>>> To: Sun, Hui
>>>>> Cc: petsc-users at mcs.anl.gov
>>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>>
>>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>>
>>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>>>
>>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>>>
>>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>>>
>>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>>>
>>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>>>
>>>>> No it handles dof > 1 fine.
>>>>>
>>>>> Send your code.
>>>>>
>>>>> Barry
>>>>>
>>>>>>
>>>>>> Thank you!

From knepley at gmail.com  Sun Mar  8 07:45:55 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 8 Mar 2015 07:45:55 -0500
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <FFF16029-02AB-480E-B4AF-06AB11A4D086@mcs.anl.gov>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
	<FFF16029-02AB-480E-B4AF-06AB11A4D086@mcs.anl.gov>
Message-ID: <CAMYG4G=mY4Dv3OMLi6UHd4-cW=Rd-+xVJ9_=uCbAFPAvvaG-fw@mail.gmail.com>

On Sat, Mar 7, 2015 at 5:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 7, 2015, at 5:27 PM, David Knezevic <david.knezevic at akselos.com>
> wrote:
> >
> > OK, thanks for letting me know.
> >
> > I've tried GAMG and ML with MatNullSpaceCreateRigidBody and those both
> work well for me. (I tried ML after Jed pointed out that ML also uses the
> near nullspace.)
> >
> > I have also tried hypre but the convergence hasn't been as good for the
> elasticity models I've been using, though I may not have been setting the
> hypre options in an optimal way for elasticity.
> >
> > Another situation I've been meaning to ask about: If the elasticity
> model includes some "near rigid" regions (very high Young's modulus) all
> iterative solvers I've tried fare poorly. I guess it's because the highly
> contrasting stiffnesses give a large condition number. Is there a standard
> way to get the preconditioner compensate for this?
>
>   Off the top of my head:   a "domain decomposition" answer might be to
> use a sparse direct solver on that region; of course how to tie that into
> the rest of the regions and the iterative solver is a broad question.
>
>    Maybe in the context of AMG all the nodes in that region could be
> retained in the coarser grids (assuming it is a relatively small part of
> the domain) and so all those points end up in the coarse grid which is
> solved with a sparse direct solver?  We don't currently have an interface
> for you to indicate nodes you want to keep in the coarse grid; Mark would
> have know if this makes any sense at all.


Isn't this just another near null vector? If the thing is very stiff, the
Jacobian is approximately singular there since you really
only have the 3 rigid body modes as degrees of freedom. You are right that
it should be a coarse basis function, but the way
to get it in there is to put it in as a near null mode, just as he is doing
now.

   Matt


>
>    Barry
>
> >
> > Thanks,
> > David
> >
> >
> >
> > On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > FYI, st?ben used classical AMG for elasticity but he has articulated his
> code for elasticity more than Hypre as I understand it.  Hypre can work OK
> for elasticity in my experience.  Its worth a try.
> >
> > Mark
> >
> > On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> > OK, got it, thanks!
> >
> > David
> >
> >
> > On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
> > David Knezevic <david.knezevic at akselos.com> writes:
> > > I was just wondering if its possible to achieve the same sort of thing
> with
> > > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
> does
> > > nothing for external solvers like hypre, right?
> >
> > It is used by ML (smoothed aggregation), but not BoomerAMG (classical
> > AMG) which uses an algorithm that doesn't have a natural place for such
> > information.  To my knowledge, classical AMG is not widely used for
> > elasticity.  It is very robust for M-matrices.
> >
> >
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/a8ae23b3/attachment.html>

From mfadams at lbl.gov  Sun Mar  8 09:31:55 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Sun, 8 Mar 2015 10:31:55 -0400
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
Message-ID: <CADOhEh4hP=oEnJrnbdBt-cLLVDb+Czy0y0COEncUdQdkodFCJQ@mail.gmail.com>

On Sat, Mar 7, 2015 at 6:27 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> OK, thanks for letting me know.
>
> I've tried GAMG and ML with MatNullSpaceCreateRigidBody and those both
> work well for me. (I tried ML after Jed pointed out that ML also uses the
> near nullspace.)
>
> I have also tried hypre but the convergence hasn't been as good for the
> elasticity models I've been using, though I may not have been setting the
> hypre options in an optimal way for elasticity.
>

As Jed said, SA should be better for elasticity but it is worth testing.


> Another situation I've been meaning to ask about: If the elasticity model
> includes some "near rigid" regions (very high Young's modulus)
>

Do you mean Poisson ratio?  Nearly incompressible is hard.  bigger
smoothers, like ASM, can help.


> all iterative solvers I've tried fare poorly. I guess it's because the
> highly contrasting stiffnesses give a large condition number. Is there a
> standard way to get the preconditioner compensate for this?
>



>
> Thanks,
> David
>
>
>
> On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> FYI, st?ben used classical AMG for elasticity but he has articulated his
>> code for elasticity more than Hypre as I understand it.  Hypre can work OK
>> for elasticity in my experience.  Its worth a try.
>>
>> Mark
>>
>> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>>
>>> OK, got it, thanks!
>>>
>>> David
>>>
>>>
>>> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> David Knezevic <david.knezevic at akselos.com> writes:
>>>> > I was just wondering if its possible to achieve the same sort of
>>>> thing with
>>>> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
>>>> does
>>>> > nothing for external solvers like hypre, right?
>>>>
>>>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>>>> AMG) which uses an algorithm that doesn't have a natural place for such
>>>> information.  To my knowledge, classical AMG is not widely used for
>>>> elasticity.  It is very robust for M-matrices.
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/85d819ba/attachment.html>

From david.knezevic at akselos.com  Sun Mar  8 10:38:35 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Sun, 8 Mar 2015 11:38:35 -0400
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CADOhEh4hP=oEnJrnbdBt-cLLVDb+Czy0y0COEncUdQdkodFCJQ@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
	<CADOhEh4hP=oEnJrnbdBt-cLLVDb+Czy0y0COEncUdQdkodFCJQ@mail.gmail.com>
Message-ID: <CAJCWK9B9YO-w7+Xxf1NyZ_V9cNtuKS8TAsyTjzmPFzJXqTXECg@mail.gmail.com>

On Sun, Mar 8, 2015 at 10:31 AM, Mark Adams <mfadams at lbl.gov> wrote:

>
>
>
>> Another situation I've been meaning to ask about: If the elasticity model
>> includes some "near rigid" regions (very high Young's modulus)
>>
>
> Do you mean Poisson ratio?
>


No, just high Young's modulus in subregions. Engineers sometimes implement
a rotation boundary condition about a node by including rigid (or, in
practice, very high Young's modulus) parts in a model. For example, imagine
including a rigid pyramid in a model, in which the tip of the pyramid is is
clamped so that the base of the pyramid (which is connected to a surface in
the model) can rotate about the tip.

This works fine with a direct solver, so I was curious about getting a good
iterative solver for this case too.



> Nearly incompressible is hard.  bigger smoothers, like ASM, can help.
>

Agreed. But the Poisson ratio is still just 0.3 or so in this context, so
locking and near incompressibility aren't an issue here.

Thanks,
David





> On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> FYI, st?ben used classical AMG for elasticity but he has articulated his
>> code for elasticity more than Hypre as I understand it.  Hypre can work OK
>> for elasticity in my experience.  Its worth a try.
>>
>> Mark
>>
>> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>>
>>> OK, got it, thanks!
>>>
>>> David
>>>
>>>
>>> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> David Knezevic <david.knezevic at akselos.com> writes:
>>>> > I was just wondering if its possible to achieve the same sort of
>>>> thing with
>>>> > other AMG solvers (e.g. BoomerAMG)? I assume that MatSetNearNullSpace
>>>> does
>>>> > nothing for external solvers like hypre, right?
>>>>
>>>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>>>> AMG) which uses an algorithm that doesn't have a natural place for such
>>>> information.  To my knowledge, classical AMG is not widely used for
>>>> elasticity.  It is very robust for M-matrices.
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/7bf6e97c/attachment.html>

From dang.hvu at gmail.com  Sun Mar  8 18:08:45 2015
From: dang.hvu at gmail.com (Hoang-Vu Dang)
Date: Sun, 8 Mar 2015 18:08:45 -0500
Subject: [petsc-users] Sparse triangular solver
Message-ID: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>

Hi,

I would like to use petcs to perform parallel backward/forward substitution
for sparse triangular matrices in a distributed memory cluster (with MPI).

Could someone provide me some pointers on how to do this or whether petsc
is good for this task ?

I think there is MatSolve method, but unsure whether it supports good
algorithm for sparse triangular matrices and how to provide an input in a
MartrixMarket format / CSR format.

Thank you
Vu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/93d33ca4/attachment.html>

From bsmith at mcs.anl.gov  Sun Mar  8 18:26:01 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 8 Mar 2015 18:26:01 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
Message-ID: <8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>


  PETSc provides sparse parallel LU (and Cholesky) factorizations and solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need to first configure PETSc to use one or more of those packages for example ./configure --download-superlu_dist --download-metis --download-parmetis.

  It is generally best to use the linear solvers via the PETSc KSP interface (even for direct solvers such as LU). So you create a KSP object, provide the matrix object and call KSPSolve(). You can control the solver used via the options database; to use the installed SuperLU_Dist you would use -pc_type lu -pc_factor_mat_solver_package superlu_dist

  The MatrixMarket format is no good for parallel computing so you must first convert the file from MatrixMarket format to the PETSc binary format (see http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format ) and then  you can use MatLoad() to load the matrix in parallel and then pass it to the KSP solver. For example src/ksp/ksp/examples/tutorials/ex10.c does this. 


  Barry

> On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> 
> Hi,
> 
> I would like to use petcs to perform parallel backward/forward substitution for sparse triangular matrices in a distributed memory cluster (with MPI). 
> 
> Could someone provide me some pointers on how to do this or whether petsc is good for this task ?
> 
> I think there is MatSolve method, but unsure whether it supports good algorithm for sparse triangular matrices and how to provide an input in a MartrixMarket format / CSR format.
> 
> Thank you
> Vu


From dang.hvu at gmail.com  Sun Mar  8 18:47:19 2015
From: dang.hvu at gmail.com (Hoang-Vu Dang)
Date: Sun, 8 Mar 2015 18:47:19 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
Message-ID: <CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>

Thank you for the prompt reply. That is very helpful.

If I do not need the full solver/factorization but just the backward subs,
do i need any special treatment ? Is there a way to hint the solver to
apply only the last step to reduce overhead ?

Vu
On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:

>
>   PETSc provides sparse parallel LU (and Cholesky) factorizations and
> solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need
> to first configure PETSc to use one or more of those packages for example
> ./configure --download-superlu_dist --download-metis --download-parmetis.
>
>   It is generally best to use the linear solvers via the PETSc KSP
> interface (even for direct solvers such as LU). So you create a KSP object,
> provide the matrix object and call KSPSolve(). You can control the solver
> used via the options database; to use the installed SuperLU_Dist you would
> use -pc_type lu -pc_factor_mat_solver_package superlu_dist
>
>   The MatrixMarket format is no good for parallel computing so you must
> first convert the file from MatrixMarket format to the PETSc binary format
> (see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format
> ) and then  you can use MatLoad() to load the matrix in parallel and then
> pass it to the KSP solver. For example
> src/ksp/ksp/examples/tutorials/ex10.c does this.
>
>
>   Barry
>
> > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> >
> > Hi,
> >
> > I would like to use petcs to perform parallel backward/forward
> substitution for sparse triangular matrices in a distributed memory cluster
> (with MPI).
> >
> > Could someone provide me some pointers on how to do this or whether
> petsc is good for this task ?
> >
> > I think there is MatSolve method, but unsure whether it supports good
> algorithm for sparse triangular matrices and how to provide an input in a
> MartrixMarket format / CSR format.
> >
> > Thank you
> > Vu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/fc56bf7d/attachment.html>

From bsmith at mcs.anl.gov  Sun Mar  8 21:25:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 8 Mar 2015 21:25:23 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
Message-ID: <0C05DE06-1577-4578-8B91-0818964257E5@mcs.anl.gov>


> On Mar 8, 2015, at 6:47 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> 
> Thank you for the prompt reply. That is very helpful.
> 
> If I do not need the full solver/factorization but just the backward subs, do i need any special treatment ? Is there a way to hint the solver to apply only the last step to reduce overhead ?

  These packages provide the factorizations and solves; there is no way to do the triangular solvers without the factorizations so I don't know what you are asking for. Please be more specific about what you need and what you are providing; are you doing the factorizations?

  Barry

> 
> Vu
> 
> On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
> 
>   PETSc provides sparse parallel LU (and Cholesky) factorizations and solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need to first configure PETSc to use one or more of those packages for example ./configure --download-superlu_dist --download-metis --download-parmetis.
> 
>   It is generally best to use the linear solvers via the PETSc KSP interface (even for direct solvers such as LU). So you create a KSP object, provide the matrix object and call KSPSolve(). You can control the solver used via the options database; to use the installed SuperLU_Dist you would use -pc_type lu -pc_factor_mat_solver_package superlu_dist
> 
>   The MatrixMarket format is no good for parallel computing so you must first convert the file from MatrixMarket format to the PETSc binary format (see http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format ) and then  you can use MatLoad() to load the matrix in parallel and then pass it to the KSP solver. For example src/ksp/ksp/examples/tutorials/ex10.c does this.
> 
> 
>   Barry
> 
> > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> >
> > Hi,
> >
> > I would like to use petcs to perform parallel backward/forward substitution for sparse triangular matrices in a distributed memory cluster (with MPI).
> >
> > Could someone provide me some pointers on how to do this or whether petsc is good for this task ?
> >
> > I think there is MatSolve method, but unsure whether it supports good algorithm for sparse triangular matrices and how to provide an input in a MartrixMarket format / CSR format.
> >
> > Thank you
> > Vu
> 


From hzhang at mcs.anl.gov  Sun Mar  8 21:28:35 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Sun, 8 Mar 2015 21:28:35 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
Message-ID: <CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>

Hoang-Vu :
>
> If I do not need the full solver/factorization but just the backward subs,
> do i need any special treatment ? Is there a way to hint the solver to
> apply only the last step to reduce overhead ?
>
What do you mean " do not need the full solver/factorization"?
Do you need incomplete matrix factorization, e.g., ILU, instead of full
factorization?
The backward subs are steps AFTER matrix factorization.

Hong

On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
>
>>
>>   PETSc provides sparse parallel LU (and Cholesky) factorizations and
>> solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need
>> to first configure PETSc to use one or more of those packages for example
>> ./configure --download-superlu_dist --download-metis --download-parmetis.
>>
>>   It is generally best to use the linear solvers via the PETSc KSP
>> interface (even for direct solvers such as LU). So you create a KSP object,
>> provide the matrix object and call KSPSolve(). You can control the solver
>> used via the options database; to use the installed SuperLU_Dist you would
>> use -pc_type lu -pc_factor_mat_solver_package superlu_dist
>>
>>   The MatrixMarket format is no good for parallel computing so you must
>> first convert the file from MatrixMarket format to the PETSc binary format
>> (see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format
>> ) and then  you can use MatLoad() to load the matrix in parallel and then
>> pass it to the KSP solver. For example
>> src/ksp/ksp/examples/tutorials/ex10.c does this.
>>
>>
>>   Barry
>>
>> > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I would like to use petcs to perform parallel backward/forward
>> substitution for sparse triangular matrices in a distributed memory cluster
>> (with MPI).
>> >
>> > Could someone provide me some pointers on how to do this or whether
>> petsc is good for this task ?
>> >
>> > I think there is MatSolve method, but unsure whether it supports good
>> algorithm for sparse triangular matrices and how to provide an input in a
>> MartrixMarket format / CSR format.
>> >
>> > Thank you
>> > Vu
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/d70000fc/attachment.html>

From dang.hvu at gmail.com  Sun Mar  8 22:40:37 2015
From: dang.hvu at gmail.com (Hoang-Vu Dang)
Date: Sun, 8 Mar 2015 22:40:37 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
	<CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>
Message-ID: <CAGrabN6EjbjaxYTjXE3Hx4=uYnB9h-4ZimwT6bXEfxYJbOxqmA@mail.gmail.com>

Sorry for causing the confusion, I should have clarify the term "triangular
solver".

What I mean is that I do not need a factorization from a general matrix to
LU form.

 I already have the matrix in lower/upper triangular form. So the solver is
really just backward/forward substitution.

See here, I got the terminology from http://www.mcs.anl.gov/papers/P1658.pdf

The algorithm "sparse triangular solve", unless I misunderstood the paper.
The paper specifically mentioned PETSC so that's where I'm starting from.

I hope that makes thing clearer,

Cheers,
Vu

On Sun, Mar 8, 2015 at 9:28 PM, Hong <hzhang at mcs.anl.gov> wrote:

> Hoang-Vu :
>>
>> If I do not need the full solver/factorization but just the backward
>> subs, do i need any special treatment ? Is there a way to hint the solver
>> to apply only the last step to reduce overhead ?
>>
> What do you mean " do not need the full solver/factorization"?
> Do you need incomplete matrix factorization, e.g., ILU, instead of full
> factorization?
> The backward subs are steps AFTER matrix factorization.
>
> Hong
>
> On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   PETSc provides sparse parallel LU (and Cholesky) factorizations and
>>> solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need
>>> to first configure PETSc to use one or more of those packages for example
>>> ./configure --download-superlu_dist --download-metis --download-parmetis.
>>>
>>>   It is generally best to use the linear solvers via the PETSc KSP
>>> interface (even for direct solvers such as LU). So you create a KSP object,
>>> provide the matrix object and call KSPSolve(). You can control the solver
>>> used via the options database; to use the installed SuperLU_Dist you would
>>> use -pc_type lu -pc_factor_mat_solver_package superlu_dist
>>>
>>>   The MatrixMarket format is no good for parallel computing so you must
>>> first convert the file from MatrixMarket format to the PETSc binary format
>>> (see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format
>>> ) and then  you can use MatLoad() to load the matrix in parallel and then
>>> pass it to the KSP solver. For example
>>> src/ksp/ksp/examples/tutorials/ex10.c does this.
>>>
>>>
>>>   Barry
>>>
>>> > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I would like to use petcs to perform parallel backward/forward
>>> substitution for sparse triangular matrices in a distributed memory cluster
>>> (with MPI).
>>> >
>>> > Could someone provide me some pointers on how to do this or whether
>>> petsc is good for this task ?
>>> >
>>> > I think there is MatSolve method, but unsure whether it supports good
>>> algorithm for sparse triangular matrices and how to provide an input in a
>>> MartrixMarket format / CSR format.
>>> >
>>> > Thank you
>>> > Vu
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150308/88064c38/attachment-0001.html>

From mirok at math.uio.no  Mon Mar  9 10:16:02 2015
From: mirok at math.uio.no (Miroslav Kuchta)
Date: Mon, 9 Mar 2015 15:16:02 +0000
Subject: [petsc-users] pesc4py setValue of dense matrix segfaults
Message-ID: <F66A163C-2C80-4C2F-99F7-102368D04325@math.uio.no>

Hi everyone,

I have a problem when setting values of a dense matrix with petsc4py. The minimal example is

import petsc4py 
petsc4py.init() 
from petsc4py import PETSc 
 
mat = PETSc.Mat() 
mat.createAIJ(size=(2, 2)) 
mat.setValue(0, 0, 1)  # Crash here 
mat.assemble() 
print mat.getValues([0, 1], [0, 1]) 

This works nicely with version 3.4 but using version 3.5 I get 

[mirok at nefele src]$ python petsc4py_bug.py 
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

Can you point out what is going wrong? Thanks for your help.

Regards, Miro



From jed at jedbrown.org  Mon Mar  9 10:37:05 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 09 Mar 2015 09:37:05 -0600
Subject: [petsc-users] pesc4py setValue of dense matrix segfaults
In-Reply-To: <F66A163C-2C80-4C2F-99F7-102368D04325@math.uio.no>
References: <F66A163C-2C80-4C2F-99F7-102368D04325@math.uio.no>
Message-ID: <87y4n6npwe.fsf@jedbrown.org>

Miroslav Kuchta <mirok at math.uio.no> writes:

> Hi everyone,
>
> I have a problem when setting values of a dense matrix with petsc4py. The minimal example is
>
> import petsc4py 
> petsc4py.init() 
> from petsc4py import PETSc 
>  
> mat = PETSc.Mat() 
> mat.createAIJ(size=(2, 2)) 

Add mat.setUp() here if you are not going to preallocate.  Of course we
should be able to produce a better error message in this case.

> mat.setValue(0, 0, 1)  # Crash here 
> mat.assemble() 
> print mat.getValues([0, 1], [0, 1]) 
>
> This works nicely with version 3.4 but using version 3.5 I get 
>
> [mirok at nefele src]$ python petsc4py_bug.py 
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> Can you point out what is going wrong? Thanks for your help.
>
> Regards, Miro
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/0789ee36/attachment.pgp>

From bsmith at mcs.anl.gov  Mon Mar  9 10:38:12 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Mar 2015 10:38:12 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGrabN6EjbjaxYTjXE3Hx4=uYnB9h-4ZimwT6bXEfxYJbOxqmA@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
	<CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>
	<CAGrabN6EjbjaxYTjXE3Hx4=uYnB9h-4ZimwT6bXEfxYJbOxqmA@mail.gmail.com>
Message-ID: <30B8E3E3-BEA8-4E0A-971F-5BC41542CD36@mcs.anl.gov>


> On Mar 8, 2015, at 10:40 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> 
> Sorry for causing the confusion, I should have clarify the term "triangular solver".
> 
> What I mean is that I do not need a factorization from a general matrix to LU form. 
> 
>  I already have the matrix in lower/upper triangular form. So the solver is really just backward/forward substitution. 

  The details of the backward/forward substitution depend in great deal on the (parallel) data structure used to store the lower and upper triangular part. This is why the lower/upper triangular solvers are tied to the same code that does the factorization; for example I cannot factor with MUMPS and then do the triangular solves with SuperLU_Dist since they assume different and complicated (parallel) data structures. 

   Are you free to pick whatever (parallel) data structure for the lower and upper triangular parts you want? Or are they given to you? Are you doing many triangular solves at the same time with different right hand sides, or are you doing them one at a time?

  Barry

> 
> See here, I got the terminology from http://www.mcs.anl.gov/papers/P1658.pdf
> 
> The algorithm "sparse triangular solve", unless I misunderstood the paper. The paper specifically mentioned PETSC so that's where I'm starting from.
> 
> I hope that makes thing clearer,
> 
> Cheers,
> Vu
> 
> On Sun, Mar 8, 2015 at 9:28 PM, Hong <hzhang at mcs.anl.gov> wrote:
> Hoang-Vu :
> If I do not need the full solver/factorization but just the backward subs, do i need any special treatment ? Is there a way to hint the solver to apply only the last step to reduce overhead ?
> 
> What do you mean " do not need the full solver/factorization"?
> Do you need incomplete matrix factorization, e.g., ILU, instead of full factorization?
> The backward subs are steps AFTER matrix factorization. 
> 
> Hong
> 
> On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
> 
>   PETSc provides sparse parallel LU (and Cholesky) factorizations and solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need to first configure PETSc to use one or more of those packages for example ./configure --download-superlu_dist --download-metis --download-parmetis.
> 
>   It is generally best to use the linear solvers via the PETSc KSP interface (even for direct solvers such as LU). So you create a KSP object, provide the matrix object and call KSPSolve(). You can control the solver used via the options database; to use the installed SuperLU_Dist you would use -pc_type lu -pc_factor_mat_solver_package superlu_dist
> 
>   The MatrixMarket format is no good for parallel computing so you must first convert the file from MatrixMarket format to the PETSc binary format (see http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format ) and then  you can use MatLoad() to load the matrix in parallel and then pass it to the KSP solver. For example src/ksp/ksp/examples/tutorials/ex10.c does this.
> 
> 
>   Barry
> 
> > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> >
> > Hi,
> >
> > I would like to use petcs to perform parallel backward/forward substitution for sparse triangular matrices in a distributed memory cluster (with MPI).
> >
> > Could someone provide me some pointers on how to do this or whether petsc is good for this task ?
> >
> > I think there is MatSolve method, but unsure whether it supports good algorithm for sparse triangular matrices and how to provide an input in a MartrixMarket format / CSR format.
> >
> > Thank you
> > Vu
> 
> 
> 


From saurabh.chawdhary at gmail.com  Mon Mar  9 10:53:28 2015
From: saurabh.chawdhary at gmail.com (Saurabh Chawdhary)
Date: Mon, 9 Mar 2015 10:53:28 -0500
Subject: [petsc-users] Error with debug
Message-ID: <CAEu6SOn=th47=AFsFyMKJNvWnrdEx9PkKnmMN0tyJnHXdw9HUg@mail.gmail.com>

Hi,
I used to work with petsc and running everything fine thus far. But then I
decided to run it with a debugger. To this end I re-installed petsc
--with-debugging=1. I can compile the code with this new petsc installation
but when I run it I get Segmentation fault error. Same code compiled with
the old without debug option runs perfectly fine.
Is something wrong with my new installation? I can compile the code without
any error so I guess all the libraries are being read fine. Not sure what
the problem is. Has anyone else faced similar problem?

Thanks
-- 
???? ?????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/49dc5d5a/attachment.html>

From jed at jedbrown.org  Mon Mar  9 10:55:13 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 09 Mar 2015 09:55:13 -0600
Subject: [petsc-users] Error with debug
In-Reply-To: <CAEu6SOn=th47=AFsFyMKJNvWnrdEx9PkKnmMN0tyJnHXdw9HUg@mail.gmail.com>
References: <CAEu6SOn=th47=AFsFyMKJNvWnrdEx9PkKnmMN0tyJnHXdw9HUg@mail.gmail.com>
Message-ID: <87sidenp26.fsf@jedbrown.org>

Saurabh Chawdhary <saurabh.chawdhary at gmail.com> writes:

> Hi,
> I used to work with petsc and running everything fine thus far. But then I
> decided to run it with a debugger. To this end I re-installed petsc
> --with-debugging=1. I can compile the code with this new petsc installation
> but when I run it I get Segmentation fault error. Same code compiled with
> the old without debug option runs perfectly fine.
> Is something wrong with my new installation? 

Run in a debugger and send the stack trace.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/c51f52e6/attachment.pgp>

From mirok at math.uio.no  Mon Mar  9 10:59:20 2015
From: mirok at math.uio.no (Miroslav Kuchta)
Date: Mon, 09 Mar 2015 16:59:20 +0100
Subject: [petsc-users] pesc4py setValue of dense matrix segfaults
In-Reply-To: <87y4n6npwe.fsf@jedbrown.org>
References: <F66A163C-2C80-4C2F-99F7-102368D04325@math.uio.no>
	<87y4n6npwe.fsf@jedbrown.org>
Message-ID: <54FDC358.4020507@math.uio.no>

Thanks, that helped.

MK
On 03/09/2015 04:37 PM, Jed Brown wrote:
> Miroslav Kuchta <mirok at math.uio.no> writes:
>
>> Hi everyone,
>>
>> I have a problem when setting values of a dense matrix with petsc4py. The minimal example is
>>
>> import petsc4py
>> petsc4py.init()
>> from petsc4py import PETSc
>>   
>> mat = PETSc.Mat()
>> mat.createAIJ(size=(2, 2))
> Add mat.setUp() here if you are not going to preallocate.  Of course we
> should be able to produce a better error message in this case.
>
>> mat.setValue(0, 0, 1)  # Crash here
>> mat.assemble()
>> print mat.getValues([0, 1], [0, 1])
>>
>> This works nicely with version 3.4 but using version 3.5 I get
>>
>> [mirok at nefele src]$ python petsc4py_bug.py
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
>> [0]PETSC ERROR: to get more information on the crash.
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>
>> Can you point out what is going wrong? Thanks for your help.
>>
>> Regards, Miro


From dang.hvu at gmail.com  Mon Mar  9 11:12:49 2015
From: dang.hvu at gmail.com (Hoang-Vu Dang)
Date: Mon, 9 Mar 2015 11:12:49 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <30B8E3E3-BEA8-4E0A-971F-5BC41542CD36@mcs.anl.gov>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
	<CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>
	<CAGrabN6EjbjaxYTjXE3Hx4=uYnB9h-4ZimwT6bXEfxYJbOxqmA@mail.gmail.com>
	<30B8E3E3-BEA8-4E0A-971F-5BC41542CD36@mcs.anl.gov>
Message-ID: <CAGrabN5Hmv+KryHkbdXe1P2yPo8PVQGyGSR++0_ppUE75OPNVw@mail.gmail.com>

Hi Barry,

>  Are you free to pick whatever (parallel) data structure for the lower
and upper triangular parts you want? Or are they given to you?

Yes I am free to pick any data structure, of course it must be possible for
me to work with or be able to convert from a different format (say CSR /
COO). I do not care yet about how efficient is the conversion part.

>  Are you doing many triangular solves at the same time with different
right hand sides, or are you doing them one at a time?

My current intention is to work with one vector at a time.


On Mon, Mar 9, 2015 at 10:38 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 8, 2015, at 10:40 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> >
> > Sorry for causing the confusion, I should have clarify the term
> "triangular solver".
> >
> > What I mean is that I do not need a factorization from a general matrix
> to LU form.
> >
> >  I already have the matrix in lower/upper triangular form. So the solver
> is really just backward/forward substitution.
>
>   The details of the backward/forward substitution depend in great deal on
> the (parallel) data structure used to store the lower and upper triangular
> part. This is why the lower/upper triangular solvers are tied to the same
> code that does the factorization; for example I cannot factor with MUMPS
> and then do the triangular solves with SuperLU_Dist since they assume
> different and complicated (parallel) data structures.
>
>    Are you free to pick whatever (parallel) data structure for the lower
> and upper triangular parts you want? Or are they given to you? Are you
> doing many triangular solves at the same time with different right hand
> sides, or are you doing them one at a time?
>
>   Barry
>
> >
> > See here, I got the terminology from
> http://www.mcs.anl.gov/papers/P1658.pdf
> >
> > The algorithm "sparse triangular solve", unless I misunderstood the
> paper. The paper specifically mentioned PETSC so that's where I'm starting
> from.
> >
> > I hope that makes thing clearer,
> >
> > Cheers,
> > Vu
> >
> > On Sun, Mar 8, 2015 at 9:28 PM, Hong <hzhang at mcs.anl.gov> wrote:
> > Hoang-Vu :
> > If I do not need the full solver/factorization but just the backward
> subs, do i need any special treatment ? Is there a way to hint the solver
> to apply only the last step to reduce overhead ?
> >
> > What do you mean " do not need the full solver/factorization"?
> > Do you need incomplete matrix factorization, e.g., ILU, instead of full
> factorization?
> > The backward subs are steps AFTER matrix factorization.
> >
> > Hong
> >
> > On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
> >
> >   PETSc provides sparse parallel LU (and Cholesky) factorizations and
> solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need
> to first configure PETSc to use one or more of those packages for example
> ./configure --download-superlu_dist --download-metis --download-parmetis.
> >
> >   It is generally best to use the linear solvers via the PETSc KSP
> interface (even for direct solvers such as LU). So you create a KSP object,
> provide the matrix object and call KSPSolve(). You can control the solver
> used via the options database; to use the installed SuperLU_Dist you would
> use -pc_type lu -pc_factor_mat_solver_package superlu_dist
> >
> >   The MatrixMarket format is no good for parallel computing so you must
> first convert the file from MatrixMarket format to the PETSc binary format
> (see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format
> ) and then  you can use MatLoad() to load the matrix in parallel and then
> pass it to the KSP solver. For example
> src/ksp/ksp/examples/tutorials/ex10.c does this.
> >
> >
> >   Barry
> >
> > > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I would like to use petcs to perform parallel backward/forward
> substitution for sparse triangular matrices in a distributed memory cluster
> (with MPI).
> > >
> > > Could someone provide me some pointers on how to do this or whether
> petsc is good for this task ?
> > >
> > > I think there is MatSolve method, but unsure whether it supports good
> algorithm for sparse triangular matrices and how to provide an input in a
> MartrixMarket format / CSR format.
> > >
> > > Thank you
> > > Vu
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/5e62f1c6/attachment-0001.html>

From saurabh.chawdhary at gmail.com  Mon Mar  9 11:25:46 2015
From: saurabh.chawdhary at gmail.com (Saurabh Chawdhary)
Date: Mon, 9 Mar 2015 11:25:46 -0500
Subject: [petsc-users] Error with debug
In-Reply-To: <87sidenp26.fsf@jedbrown.org>
References: <CAEu6SOn=th47=AFsFyMKJNvWnrdEx9PkKnmMN0tyJnHXdw9HUg@mail.gmail.com>
	<87sidenp26.fsf@jedbrown.org>
Message-ID: <CAEu6SOnzPedAozUuga4hhmBTG7jjfprMLFOROF4LJ5EbTwo2uQ@mail.gmail.com>

Sorry here is the Error trace I get on running on 1 processor:
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: PetscMallocValidate: error detected at
 PetscSignalHandlerDefault() line 149 in src/sys/error/signal.c
[0]PETSC ERROR: Memory at address 0x7fe4fdc6f010 is corrupted
[0]PETSC ERROR: Probably write past beginning or end of array
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Memory corruption!
[0]PETSC ERROR:  !
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ../IB-AMRS/test1 on a arch-linux2-c-debug named
aegean.safl.umn.edu by chaw0023 Mon Mar  9 11:24:50 2015
[0]PETSC ERROR: Libraries linked from
/home/chaw0023/software/aegean/debug/petsc-3.4.5/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Thu Feb 19 12:22:29 2015
[0]PETSC ERROR: Configure options
--with-mpi-dir=/safl/software/aegean/openmpi/1.5.5/gcc/4.7.0-fix
--with-blas-lapack-dir=/safl/software/x86_64/acml/4.4.0/gfortran64
--download-hypre=yes --with-shared-libraries=0 --with-debugging=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscMallocValidate() line 137 in src/sys/memory/mtr.c
[0]PETSC ERROR: PetscSignalHandlerDefault() line 149 in
src/sys/error/signal.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 0.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------



On Mon, Mar 9, 2015 at 10:55 AM, Jed Brown <jed at jedbrown.org> wrote:

> Saurabh Chawdhary <saurabh.chawdhary at gmail.com> writes:
>
> > Hi,
> > I used to work with petsc and running everything fine thus far. But then
> I
> > decided to run it with a debugger. To this end I re-installed petsc
> > --with-debugging=1. I can compile the code with this new petsc
> installation
> > but when I run it I get Segmentation fault error. Same code compiled with
> > the old without debug option runs perfectly fine.
> > Is something wrong with my new installation?
>
> Run in a debugger and send the stack trace.
>



-- 
???? ?????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/8552b140/attachment.html>

From bsmith at mcs.anl.gov  Mon Mar  9 11:26:18 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Mar 2015 11:26:18 -0500
Subject: [petsc-users] Sparse triangular solver
In-Reply-To: <CAGrabN5Hmv+KryHkbdXe1P2yPo8PVQGyGSR++0_ppUE75OPNVw@mail.gmail.com>
References: <CAGrabN7Y=Apvpa3Zpvx+HN-D+n6QHv1ro-8EX8NHRY+C8A9hnw@mail.gmail.com>
	<8A2C5901-E34B-4162-A7A9-5BE33B2E57A4@mcs.anl.gov>
	<CAGrabN5+AoV7vZOgBeVUeHbk16Syy2h6BjrD6b0NrZM_GvaEmA@mail.gmail.com>
	<CAGCphBs=x=pYq2TpfN1b1nKczj985nXs2i5mD2U3cuO5Shbc9g@mail.gmail.com>
	<CAGrabN6EjbjaxYTjXE3Hx4=uYnB9h-4ZimwT6bXEfxYJbOxqmA@mail.gmail.com>
	<30B8E3E3-BEA8-4E0A-971F-5BC41542CD36@mcs.anl.gov>
	<CAGrabN5Hmv+KryHkbdXe1P2yPo8PVQGyGSR++0_ppUE75OPNVw@mail.gmail.com>
Message-ID: <0B3C2163-1898-41FF-A000-B02E7C492505@mcs.anl.gov>


> On Mar 9, 2015, at 11:12 AM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> 
> Hi Barry,
> 
> >  Are you free to pick whatever (parallel) data structure for the lower and upper triangular parts you want? Or are they given to you?
> 
> Yes I am free to pick any data structure, of course it must be possible for me to work with or be able to convert from a different format (say CSR / COO). I do not care yet about how efficient is the conversion part.

   Then you can either design your own or try to use one used by SuperLU_Dist, MUMPS, or Pastix. PETSc does't directly provide such data structures or code.

  Barry

> 
> >  Are you doing many triangular solves at the same time with different right hand sides, or are you doing them one at a time?
> 
> My current intention is to work with one vector at a time.
> 
> 
> On Mon, Mar 9, 2015 at 10:38 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 8, 2015, at 10:40 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> >
> > Sorry for causing the confusion, I should have clarify the term "triangular solver".
> >
> > What I mean is that I do not need a factorization from a general matrix to LU form.
> >
> >  I already have the matrix in lower/upper triangular form. So the solver is really just backward/forward substitution.
> 
>   The details of the backward/forward substitution depend in great deal on the (parallel) data structure used to store the lower and upper triangular part. This is why the lower/upper triangular solvers are tied to the same code that does the factorization; for example I cannot factor with MUMPS and then do the triangular solves with SuperLU_Dist since they assume different and complicated (parallel) data structures.
> 
>    Are you free to pick whatever (parallel) data structure for the lower and upper triangular parts you want? Or are they given to you? Are you doing many triangular solves at the same time with different right hand sides, or are you doing them one at a time?
> 
>   Barry
> 
> >
> > See here, I got the terminology from http://www.mcs.anl.gov/papers/P1658.pdf
> >
> > The algorithm "sparse triangular solve", unless I misunderstood the paper. The paper specifically mentioned PETSC so that's where I'm starting from.
> >
> > I hope that makes thing clearer,
> >
> > Cheers,
> > Vu
> >
> > On Sun, Mar 8, 2015 at 9:28 PM, Hong <hzhang at mcs.anl.gov> wrote:
> > Hoang-Vu :
> > If I do not need the full solver/factorization but just the backward subs, do i need any special treatment ? Is there a way to hint the solver to apply only the last step to reduce overhead ?
> >
> > What do you mean " do not need the full solver/factorization"?
> > Do you need incomplete matrix factorization, e.g., ILU, instead of full factorization?
> > The backward subs are steps AFTER matrix factorization.
> >
> > Hong
> >
> > On Mar 8, 2015 6:26 PM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
> >
> >   PETSc provides sparse parallel LU (and Cholesky) factorizations and solves via the external packages SuperLU_Dist, MUMPS, and Pastix. You need to first configure PETSc to use one or more of those packages for example ./configure --download-superlu_dist --download-metis --download-parmetis.
> >
> >   It is generally best to use the linear solvers via the PETSc KSP interface (even for direct solvers such as LU). So you create a KSP object, provide the matrix object and call KSPSolve(). You can control the solver used via the options database; to use the installed SuperLU_Dist you would use -pc_type lu -pc_factor_mat_solver_package superlu_dist
> >
> >   The MatrixMarket format is no good for parallel computing so you must first convert the file from MatrixMarket format to the PETSc binary format (see http://www.mcs.anl.gov/petsc/documentation/faq.html#sparse-matrix-ascii-format ) and then  you can use MatLoad() to load the matrix in parallel and then pass it to the KSP solver. For example src/ksp/ksp/examples/tutorials/ex10.c does this.
> >
> >
> >   Barry
> >
> > > On Mar 8, 2015, at 6:08 PM, Hoang-Vu Dang <dang.hvu at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I would like to use petcs to perform parallel backward/forward substitution for sparse triangular matrices in a distributed memory cluster (with MPI).
> > >
> > > Could someone provide me some pointers on how to do this or whether petsc is good for this task ?
> > >
> > > I think there is MatSolve method, but unsure whether it supports good algorithm for sparse triangular matrices and how to provide an input in a MartrixMarket format / CSR format.
> > >
> > > Thank you
> > > Vu
> >
> >
> >
> 
> 


From amesga1 at tigers.lsu.edu  Mon Mar  9 11:28:51 2015
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Mon, 9 Mar 2015 11:28:51 -0500
Subject: [petsc-users] DMView and DMLoad
Message-ID: <CAC+VmGeK8+4NOXKj+9CerkkJxVNCorz7D_fM-fPjmSuJv6p63g@mail.gmail.com>

Dear all,

I'm trying to save and load distributed DMPlex in HDF5 format with DMView
and DMLoad. The problem is even though there are no errors while loading
the DM when I look at the DM loaded from file is not equal to the DM that
was saved. Also, when loading the DM if DMSetType is called before the
DMLoad, DMLoad produces errors (like line 316
<https://bitbucket.org/petsc/petsc/src/236f5a4d227fe9d0affddb8701edb9509ad39525/src/snes/examples/tutorials/ex12.c?at=master#cl-316>
in src/snes/examples/tutorials/ex12.c ).

I'm including a small code to replicate the problem here plus an input mesh
file.

Best,
Ata

/**

 * @file

 *

 * Created by Ataollah Mesgarnejad on 3/6/15.

 *

 * This is a test to read and write a distributed DM.

 *

 */



static char help[] = "An example of the usage of PetscSF to scatter data
back and forth between a \

a serial DM and its parallel counterpart.\n";


#include <petscdmplex.h>

#include <petscsnes.h>

#include <petscsf.h>

#include <exodusII.h>

#include <petsc-private/dmimpl.h>     /*I      "petscdm.h"     I*/

#include <petscsf.h>

#include <petscviewerhdf5.h>


#undef __FUNCT__

#define __FUNCT__ "main"

int main(int argc,char **argv)

{

    PetscErrorCode      ierr;

    DM                  dm, distDM,distDMold;

    char                ifilename[PETSC_MAX_PATH_LEN];

    PetscBool           flg;

    int                 CPU_word_size = 0,IO_word_size = 0,exoid;

    float               version;

    PetscInt            numproc,rank;

    PetscViewer         stdoutViewer,DMPlexHDF5View;

    PetscReal           *VArray;

    PetscInt            dim;

    PetscInt            numFields = 2;

    PetscInt            numComp[2] = {1,1};

    PetscInt            numDof[6] = {1, 0, 0,

        0, 0, 1}; /*{Vertex, Edge, Cell} */

    PetscInt            bcFields[1] = {0}, numBC=0;

    //PetscInt          *remoteOffsets;

    char**              namelist;

    PetscInt            off;

    IS                  bcPoints[1] = {NULL};

    IS                  *ISlist;

    PetscSection        seqSection, distSection;

    Vec                 distV, V;

    PetscSF             pointSF;//,pointSFold;

    PetscInt            pStart, pEnd, dof;

    PetscBool           load_files = PETSC_FALSE, save_files = PETSC_FALSE;

    MPI_Comm            comm;



    ierr = PetscInitialize(&argc, &argv, (char*)0, help);CHKERRQ(ierr);

    ierr = PetscViewerASCIIGetStdout(PETSC_COMM_SELF,&stdoutViewer);CHKERRQ
(ierr);

    comm = PETSC_COMM_WORLD;

    // PetscErrorCode DMCreateSubDM(DM dm, PetscInt numFields, PetscInt
fields[], IS *is, DM *subdm)







    ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","PetscSF Test Options",
"none");CHKERRQ(ierr);

    ierr = PetscOptionsBool("-save_files","save the distributed vector in
HDF5 format","",PETSC_FALSE,&save_files,NULL);CHKERRQ(ierr);

    ierr = PetscOptionsBool("-load_files","Load the distributed vector in
HDF5 format","",PETSC_FALSE,&load_files,NULL);CHKERRQ(ierr);

    ierr = PetscOptionsEnd();

    ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof
ifilename,&flg);CHKERRQ(ierr);



    ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc);

    ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);

    if (!rank)

    {

        exoid = ex_open(ifilename,EX_READ
,&CPU_word_size,&IO_word_size,&version);

        if (exoid <= 0)
SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"ex_open(\"%s\",...)
did not return a valid file ID",ifilename);

    }

    else

    {

        exoid = -1;                 /* Not used */

    }

    ierr = DMPlexCreateExodus(PETSC_COMM_WORLD,exoid,PETSC_FALSE,&dm);
                                            CHKERRQ(ierr);

    ierr = PetscObjectSetName((PetscObject) dm,"sequential-DM");

    ierr = DMGetDimension(dm, &dim);
                                            CHKERRQ(ierr);



    if (numproc < 2 ) distDM = dm;

    else ierr = DMPlexDistribute(dm, 0, &pointSF, &distDM);
                                            CHKERRQ(ierr);

    ierr = PetscObjectSetName((PetscObject) distDM,"Distributed-DM");


    ierr = DMPlexCreateSection(dm,      dim, numFields, numComp, numDof,

                               numBC, bcFields, bcPoints,PETSC_NULL,
&seqSection);                                      CHKERRQ(ierr);

    ierr = DMPlexCreateSection(distDM,  dim, numFields, numComp, numDof,

                               numBC, bcFields, bcPoints,PETSC_NULL,
&distSection);                                     CHKERRQ(ierr);



    ierr = DMSetDefaultSection(dm, seqSection);
                                            CHKERRQ(ierr);

    ierr = DMSetDefaultSection(distDM, distSection);
                                            CHKERRQ(ierr);



    ierr = DMCreateGlobalVector(dm, &V);
                                            CHKERRQ(ierr);

    ierr = VecCreate(comm, &distV);
                                            CHKERRQ(ierr);

    ierr = PetscObjectSetName((PetscObject) distV,"Distributed-V");


    ierr = VecGetArray(V, &VArray);
                                            CHKERRQ(ierr);



    // fill vertex data

    ierr = DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
                                            CHKERRQ(ierr);

    for (int p = pStart; p < pEnd; p++)

    {

        ierr = PetscSectionGetDof(seqSection, p, &dof);
                                            CHKERRQ(ierr);

        ierr = PetscSectionGetOffset(seqSection, p, &off);
                                            CHKERRQ(ierr);

        for (int d = 0; d < dof; d++)

        {

            VArray[off + d] = p;

        }

    }



    // fill cell data

    ierr = DMPlexGetDepthStratum(dm, 1, &pStart, &pEnd);
                                            CHKERRQ(ierr);

    for (int p = pStart; p < pEnd; p++)

    {

        ierr = PetscSectionGetDof(seqSection, p, &dof);
                                            CHKERRQ(ierr);

        ierr = PetscSectionGetOffset(seqSection, p, &off);
                                            CHKERRQ(ierr);

        for (int d = 0; d < dof; d++)

        {

            VArray[off + d] = -p-1;

        }

    }



    ierr = VecRestoreArray(V, &VArray); CHKERRQ(ierr);

    ierr = DMCreateFieldIS(dm, &numFields, &namelist, &ISlist);
                                            CHKERRQ(ierr);

    ierr = DMPlexDistributeField(dm, pointSF, seqSection, V, distSection,
distV);                                       CHKERRQ(ierr);

#if defined(PETSC_HAVE_HDF5)

    if (save_files)

    {

        // distribute fields



        ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Distrubuted DM\n");
                                            CHKERRQ(ierr);

        ierr = DMView(distDM, PETSC_VIEWER_STDOUT_WORLD);
                                            CHKERRQ(ierr);

        PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);

        // Save distributed data



        ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject) distDM),
"distDM.h5", FILE_MODE_WRITE, &DMPlexHDF5View);CHKERRQ(ierr);

        ierr = PetscPrintf(PETSC_COMM_WORLD,"Writing dist DM ...\n");

        PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);

        ierr = DMView(distDM, DMPlexHDF5View);
                                            CHKERRQ(ierr);



        PetscViewerDestroy(&DMPlexHDF5View);



    }

    else if (load_files)

    {

        ierr = PetscPrintf(PETSC_COMM_WORLD,"Loading dist vectors ...\n");
                                            CHKERRQ(ierr);

        ierr = PetscViewerHDF5Open(PETSC_COMM_SELF,"distDM.h5",
FILE_MODE_READ, &DMPlexHDF5View);                       CHKERRQ(ierr);

        ierr = DMCreate(comm,&distDMold);
                                            CHKERRQ(ierr);



        ierr = DMLoad(distDMold,DMPlexHDF5View);
                                            CHKERRQ(ierr);

        ierr = DMSetType(distDMold, DMPLEX);
                                            CHKERRQ(ierr);

        /*ierr = DMPlexCreateSection(distDMold,  dim, numFields, numComp,
numDof,

                                   numBC, bcFields, bcPoints,PETSC_NULL,
&distSectionold);                              CHKERRQ(ierr);

        ierr = DMSetDefaultSection(distDMold, distSectionold);CHKERRQ(ierr);

        ierr = DMGetPointSF(distDMold,&pointSFold);
                                            CHKERRQ(ierr);*/

        PetscViewerDestroy(&DMPlexHDF5View);

        ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Read DM\n");
                                            CHKERRQ(ierr);

        ierr = DMView(distDMold, PETSC_VIEWER_STDOUT_WORLD);
                                            CHKERRQ(ierr);

        PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
                                            CHKERRQ(ierr);

        DMPlexEqual(distDM, distDMold, &flg);

        if (flg)

        {

            PetscPrintf(PETSC_COMM_WORLD,"\n DMs equal\n");

        }

        else

        {

            PetscPrintf(PETSC_COMM_WORLD,"\n DMs are not equal\n\n");

        }

    }

#endif



    ierr = VecDestroy(&V); CHKERRQ(ierr);

    ierr = VecDestroy(&distV); CHKERRQ(ierr);

    //ierr = VecDestroy(&seqV); CHKERRQ(ierr);



    ierr = DMDestroy(&dm);CHKERRQ(ierr);

    ierr = DMDestroy(&distDM);CHKERRQ(ierr);



    ierr = PetscFinalize();

    return 0;

}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/c499a5dc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testDistributeSaveLoad.cpp
Type: text/x-c++src
Size: 9270 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/c499a5dc/attachment-0001.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Rect-tri3.gen
Type: application/octet-stream
Size: 4972 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/c499a5dc/attachment-0001.obj>

From knepley at gmail.com  Mon Mar  9 11:49:00 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 9 Mar 2015 11:49:00 -0500
Subject: [petsc-users] Error with debug
In-Reply-To: <CAEu6SOnzPedAozUuga4hhmBTG7jjfprMLFOROF4LJ5EbTwo2uQ@mail.gmail.com>
References: <CAEu6SOn=th47=AFsFyMKJNvWnrdEx9PkKnmMN0tyJnHXdw9HUg@mail.gmail.com>
	<87sidenp26.fsf@jedbrown.org>
	<CAEu6SOnzPedAozUuga4hhmBTG7jjfprMLFOROF4LJ5EbTwo2uQ@mail.gmail.com>
Message-ID: <CAMYG4GkT1zhOW-owy+ni9KNhQ3Ax=s2ntRaHGSF-OPYnEfUuLA@mail.gmail.com>

Use valgrind like it says and find out where memory is being overwritten.

  Thanks,

     Matt

On Mon, Mar 9, 2015 at 11:25 AM, Saurabh Chawdhary <
saurabh.chawdhary at gmail.com> wrote:

> Sorry here is the Error trace I get on running on 1 processor:
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: PetscMallocValidate: error detected at
>  PetscSignalHandlerDefault() line 149 in src/sys/error/signal.c
> [0]PETSC ERROR: Memory at address 0x7fe4fdc6f010 is corrupted
> [0]PETSC ERROR: Probably write past beginning or end of array
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Memory corruption!
> [0]PETSC ERROR:  !
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ../IB-AMRS/test1 on a arch-linux2-c-debug named
> aegean.safl.umn.edu by chaw0023 Mon Mar  9 11:24:50 2015
> [0]PETSC ERROR: Libraries linked from
> /home/chaw0023/software/aegean/debug/petsc-3.4.5/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Thu Feb 19 12:22:29 2015
> [0]PETSC ERROR: Configure options
> --with-mpi-dir=/safl/software/aegean/openmpi/1.5.5/gcc/4.7.0-fix
> --with-blas-lapack-dir=/safl/software/x86_64/acml/4.4.0/gfortran64
> --download-hypre=yes --with-shared-libraries=0 --with-debugging=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocValidate() line 137 in src/sys/memory/mtr.c
> [0]PETSC ERROR: PetscSignalHandlerDefault() line 149 in
> src/sys/error/signal.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 0.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
>
>
>
> On Mon, Mar 9, 2015 at 10:55 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Saurabh Chawdhary <saurabh.chawdhary at gmail.com> writes:
>>
>> > Hi,
>> > I used to work with petsc and running everything fine thus far. But
>> then I
>> > decided to run it with a debugger. To this end I re-installed petsc
>> > --with-debugging=1. I can compile the code with this new petsc
>> installation
>> > but when I run it I get Segmentation fault error. Same code compiled
>> with
>> > the old without debug option runs perfectly fine.
>> > Is something wrong with my new installation?
>>
>> Run in a debugger and send the stack trace.
>>
>
>
>
> --
> ???? ?????
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/ac314def/attachment.html>

From bsmith at mcs.anl.gov  Mon Mar  9 17:08:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Mar 2015 17:08:57 -0500
Subject: [petsc-users] DMDA with dof=4, multigrid solver
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010EA11B@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010E9C4F@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <DB7F9E86-5D6F-4C79-91E6-1ACD80A96F40@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9C62@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <B4FED060-3D21-416B-9CFE-3209338D66B5@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9CBC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <9B0A08A2-EFF5-4180-8784-4E487FED8A06@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9ED9@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <AD80DDBA-92CC-47A7-A85F-C703DDDDDBE0@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010E9F04@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <12254963-6959-4A74-BFAF-260CE439E9DD@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010EA0FC@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<, <CE4A7535-BF1D-4E21-8066-51FA1BFE4708@mcs.anl.gov> <>>
	<7501CC2B7BBCC44A92ECEEC316170ECB010EA11B@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <892BDAF4-2E11-4B16-B831-07DCBF879925@mcs.anl.gov>


  The local to global mapping depends on how you choose to put the values in parallel; take the product  y = A*x  where your matrix is A (and has very few columns). You can define x to have all its values on the first process. If you want a ghosted x which has all the values on each process in the ghosted version of x then on each process your local to global mapping would simply be  0,1,...M. That is the global number of each local entry is simply the same as the local number. 

  Barry

> On Mar 7, 2015, at 7:20 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Thank you, Barry. 
> 
> In DMCreateMatrix_DA_3d_MPIAIJ, there are two lines which I think are crucial: 
> ierr = MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
> ierr = MatSetLocalToGlobalMapping(J,ltog,ltog);CHKERRQ(ierr);
> 
> For my case, I just need the ltog for the rows, and my columns are sequential, what shall I do here? I want to get a rltog for rows and a cltog for columns. There is no question that I get rltog from da, but where do I get the cltog? 
> 
> It is not for interpolation between levels of DMDA. The matrix is for some quantity (of dimension M) that doesn't live on the grid (of size n^3), but depends on some other quantity (of dimension N=n^3, with 1<<M<<N) which lives on the grid. 
> 
> Hui
> 
> 
> ________________________________________
> From: Barry Smith [bsmith at mcs.anl.gov]
> Sent: Saturday, March 07, 2015 12:17 PM
> To: Sun, Hui
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
> 
>> On Mar 7, 2015, at 1:17 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>> 
>> Here is one more problem along this line. The matrix A created in this way may not have a natural global numbering. Let's say, if we are on grid point (i,j,k), the element in A with row (i,j,k) and column 1 is not necessarily at row position (i+(j+k*my)*mx globally. I mean, how do I set the correspondence between the MatStencils and int for the index? Should I use something like ISLocalToGlobalMapping?
> 
>  You can.
> 
>   Look at the routines in PETSc that form the matrices like DMCreateMatrix_DA_3d_MPIAIJ() and modify it for the matrix you need.
> 
>   BTW: what is the matrix you a creating for. PETSc also creates matrices that interpolate between levels of DMDA like DMCreateInterpolation_DA_3D_Q1() it's structure might be useful for you.
> 
>  Barry
> 
> 
>> 
>> Best,
>> Hui
>> 
>> 
>> ________________________________________
>> From: Barry Smith [bsmith at mcs.anl.gov]
>> Sent: Wednesday, March 04, 2015 10:53 AM
>> To: Sun, Hui
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>> 
>>> On Mar 4, 2015, at 8:53 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>> 
>>> I thought DMDAcreateMatrix would already specify the size of the matrix to be N by N, where N is the number of unknowns in the grid. And each processor takes care of a submatrix of size N_i by N_i.
>> 
>> Not really. Each process "owns" certain entire rows of the matrix. Different processors cannot contain parts of the same rows.
>>> 
>>> However, I wish to form a matrix of size M by N, where N is the number of unknowns in the grid, and  1<<M<<N. Furthermore, I wish to divide the M by N matrix A into the processors, in such a way, that each processor has a submatrix of size M by N_i.
>> 
>>  PETSc doesn't provide a matrix format where sets of columns are on different processes. It only provides formats where certain rows are on each process. Depending on what you want to do with your matrix; for example if you only need to do Matrix-Vector products with your matrix, you could store by row the transpose of the matrix you want and use MatMultTranspose() to do the multiply.
>> 
>>  To create such a "transposed matrix" you can call DMGetGlobalVector(), VecGetOwnershipRange(v,&start,&end); then call MatCreateMPIAIJ(comm, end-start,PETSC_DECIDE,PETSC_DETERMINE,M,0,NULL,0,NULL,&A);  With this matrix you can then call MatMultTranspose(A,v,w);
>> 
>>  Barry
>> 
>> 
>>>> DMDAGetLocalInfo
>> 
>> 
>> 
>>> 
>>> Hui
>>> 
>>> 
>>> ________________________________________
>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Wednesday, March 04, 2015 5:27 AM
>>> To: Sun, Hui
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>> 
>>> If at all possible you should use DMCreateMatrix() it manages everything for you. Why can't you use it?
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> On Mar 4, 2015, at 1:20 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>> 
>>>> Thank you Barry for giving me some hint of using DMDAGetLocalInfo() to determine the local size. However, I'm still confused about the process of creating such a matrix, which is composed of serial rows of DMDA parallel vectors.
>>>> 
>>>> Should I somehow use the following functions?
>>>> MatCreate
>>>> DMDAGetLocalInfo
>>>> ISLocalToGlobalMappingCreate
>>>> MatSetLocalToGlobalMapping
>>>> MatGetLocalSubMatrix
>>>> 
>>>> However, I still need some more help on putting everything together to create such a matrix. I'd really appreciate your time.
>>>> 
>>>> Best,
>>>> Hui
>>>> 
>>>> ________________________________________
>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>> Sent: Sunday, March 01, 2015 9:24 AM
>>>> To: Sun, Hui
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>> 
>>>>> On Mar 1, 2015, at 12:30 AM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>> 
>>>>> Thank you Barry. I have yet two more questions:
>>>>> 
>>>>> 1) If I have a DMDA and I use KSPSetComputeOperators and KSPSetComputeRHS to set up matrices and rhs, and I use geometric mg, what if I want to change my rhs many times? Should I write many KSPSetComputeRHS, and register them with ksp? Or is there a simple way to just register the rhs with ksp as a vector?
>>>>> 
>>>>> 2) How do I create a Mat, whose cols follow the DMDA parallelization, and whose rows are serial?
>>>> 
>>>> Normally one uses DMCreateMatrix() to get the matrices; it has the correct parallel layout and the correct nonzero pattern.   If you create the matrices yourself you need to first call DMDAGetLocalInfo() and from that information determine how many local rows you have.
>>>> 
>>>>> 
>>>>> By the way, I've figured out and fixed the bugs in my code concerning using mg with DMDA having 4 dof. It has to do with the interpolations. Now I can see mg works well with 4 dof DMDA.
>>>>> 
>>>>> Best,
>>>>> Hui
>>>>> 
>>>>> ________________________________________
>>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>>> Sent: Saturday, February 28, 2015 9:35 AM
>>>>> To: Sun, Hui
>>>>> Cc: petsc-users at mcs.anl.gov
>>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>> 
>>>>>> On Feb 27, 2015, at 7:25 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>> 
>>>>>> Thank you Barry. Another question: I observe that in those ksp examples, whenever multigrid is used, DMDA is also used, besides, KSPSetComputeOperators and KSPSetComputeRHS are also used.
>>>>>> 
>>>>>> Is it true that
>>>>>> 1) Only DMDA can use mg?
>>>>> 
>>>>> No this is not true
>>>>> 
>>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>>> 
>>>>> No you do not have to
>>>>> 
>>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>>> 
>>>>> Yes you can.
>>>>> 
>>>>> There are many many variants of multigrid one can do with PETSc; we don't have the time to have examples of all the possibilities.
>>>>> 
>>>>> More details
>>>>> 
>>>>>> 1) Only DMDA can use mg?
>>>>> 
>>>>> Because DMDA provides structured grids with easy interpolation between levels and it is easy for users to write Jacobians we have many examples that use the DMDA. However, so long as YOU (or something) can provide interpolation between the multigrid levels you can use multigrid. For example PCGAMG uses algebraic multigrid to generate the interpolations. If you have your own interpolations you can provide them with PCMGSetInterpolation() (when you use PCMG with DMDA PETSc essentially handles those details automatically for you).
>>>>> 
>>>>>> 2) We have to set up matrices and rhs using KSPSetComputeOperators  and KSPSetComputeRHS?
>>>>> 
>>>>> Normally with geometric multigrid one discretizes the operator on each level of the grid. Thus the user has to provide several matrices (one for each level). KSPSetComputeOperators() is ONE way that the user can provide them. You can also provide them by call PCMGetSmoother(pc,level,&ksp) and then call KSPSetOperators(ksp,...) for each of the levels (KSPSetComputeOperators() essentially does the book keeping for you).
>>>>> 
>>>>>> We cannot create a matrix and add it to KSP if we want to use mg?
>>>>> 
>>>>> As I said in 2 normally multigrid requires you to provide a discretized operator at each level. But with Galerkin coarse grids (which is what algebraic multigrid users and can also be used by geometric multigrid) the user does not provide coarser grid operators instead the code computes them automatically from the formula R*A*P where R is the restriction operator used in multigrid and P is the interpolation operator (usually the transpose of P).
>>>>> 
>>>>> If you are looking for a simple automatic multigrid then you want to use PCGAMG in PETSc, it does algebraic multigrid and doesn't require you provide interpolations or coarser operators. However algebraic multigrid doesn't work for all problems; though it does work for many. Try it with -pc_type gamg
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> Hui
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: Barry Smith [bsmith at mcs.anl.gov]
>>>>>> Sent: Friday, February 27, 2015 5:11 PM
>>>>>> To: Sun, Hui
>>>>>> Cc: petsc-users at mcs.anl.gov
>>>>>> Subject: Re: [petsc-users] DMDA with dof=4, multigrid solver
>>>>>> 
>>>>>>> On Feb 27, 2015, at 6:36 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>>>>>>> 
>>>>>>> I'm trying to work on 4 Poisson's equations defined on a DMDA grid, Hence the parameter dof in DMDACreate3d should be 4, and I've set stencil width to be 4, and stencil type to be star.
>>>>>> 
>>>>>> Use a stencil width of 1, not 4. The stencil width is defined in terms of dof.
>>>>>>> 
>>>>>>> If I run the code with -pc_type ilu and -ksp_type gmres, it works alright.
>>>>>>> 
>>>>>>> However, if I run with pc_type mg, it gives me an error saying that when it is doing MatSetValues, the argument is out of range, and there is a new nonzero at (60,64) in the matrix. However, that new nonzero is expected to be there, the row number 60 corresponds to i=15 and c=0 in x direction, and the column number 64 corresponds to i=16 and c=0 in x direction. So they are next to each other, and the star stencil with width 1 should include that. I have also checked with the memory allocations, and I'm found no problem.
>>>>>>> 
>>>>>>> So I'm wondering if there is any problem of using multigrid on a DMDA with dof greater than 1?
>>>>>> 
>>>>>> No it handles dof > 1 fine.
>>>>>> 
>>>>>> Send your code.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> 
>>>>>>> Thank you!


From orxan.shibli at gmail.com  Mon Mar  9 17:23:10 2015
From: orxan.shibli at gmail.com (Orxan Shibliyev)
Date: Mon, 9 Mar 2015 16:23:10 -0600
Subject: [petsc-users] GMRES stability
In-Reply-To: <041B1984-4BFB-4A6C-92F1-8CC8DBBF0B07@mcs.anl.gov>
References: <CAOYX9gzxrd7-tHWLS6RZZK2AaO_O7tVbDw1s1b6vF=YewwCSuQ@mail.gmail.com>
	<0A826E90-DD1A-450B-BDE2-D5A4867B682B@mcs.anl.gov>
	<CAOYX9gx5e=TigrB3A0hE8G-qXJX9h83mRB+GSYwD3YE9oVATtg@mail.gmail.com>
	<239AD9F8-8DBE-42D6-B185-7BF246FC4900@mcs.anl.gov>
	<CAOYX9gzSfgswMAPnxPMNHab8As9MdkOj3g-9ZJGqPc6Rr8M4pQ@mail.gmail.com>
	<041B1984-4BFB-4A6C-92F1-8CC8DBBF0B07@mcs.anl.gov>
Message-ID: <CAOYX9gyG6Pu9S3JTVTtYxTWn8NcS6K5S1pMrvXdYer0-nV8aPw@mail.gmail.com>

I use proactive *point* GS which updates the RHS of neighbours instead of
updating the variable itself in subsequent iterations. I use natural
ordering. I am sure that the matrices are the same for both solvers.

On Sun, Mar 1, 2015 at 4:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Ok, we need to understand what is algorithmically different between what
> your code is doing and PETSc is doing.
>
>   From below PETSc is running a symmetric point block SOR with a block
> size of 5. Are you using a point or point block GS? Are you using any
> particular ordering of the unknowns in your GS, or just natural ordering?
> Are you sure the matrices are the same between the two codes? Print them
> out for a tiny grid and compare.
>
>   Barry
>
> > On Mar 1, 2015, at 4:33 PM, Orxan Shibliyev <orxan.shibli at gmail.com>
> wrote:
> >
> > I don't run in parallel. I tried -ksp_richardson -pc_type but still
> > cannot get over CFL=5 (with my own GS code CFL=40 was converging).
> > Also, lowering damping factor does not help. Default number of
> > iterations (10000) does not help as well. -ksp_view outputs:
> >
> > KSP Object: 1 MPI processes
> >  type: richardson
> >    Richardson: damping factor=0.5
> >  maximum iterations=10, initial guess is zero
> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >  left preconditioning
> >  using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >  type: sor
> >    SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
> >  linear system matrix = precond matrix:
> >  Mat Object:   1 MPI processes
> >    type: seqbaij
> >    rows=13250, cols=13250, bs=5
> >    total: nonzeros=261150, allocated nonzeros=1.325e+06
> >    total number of mallocs used during MatSetValues calls =0
> >        block size is 5
> >
> > On Fri, Feb 27, 2015 at 4:30 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>
> >>  Ok, please provide the rest of the information I asked for.
> >>
> >>  Barry
> >>
> >>> On Feb 27, 2015, at 2:33 AM, Orxan Shibliyev <orxan.shibli at gmail.com>
> wrote:
> >>>
> >>> No. It does not converge at all or iow it diverges.
> >>>
> >>> On Thu, Feb 26, 2015 at 9:36 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>>>
> >>>> By stability I assume you mean the the GMRES does not converge (or
> converges too slowly)?
> >>>>
> >>>> The way to improve GMRES convergence is with a preconditioner better
> suited to your problem. By default PETSc uses GMRES with a block Jacobi
> preconditioner with one block per process and ILU(0) on each block. For
> some problems this is fine, but for many problems it will give bad
> convergence.
> >>>>
> >>>>  What do you get for -ksp_view  (are you using the default?) Are you
> running yet in parallel?
> >>>>
> >>>> As a test on one process you can use GS in PETSc as the
> preconditioner and make sure you get similar convergence to your code. For
> example -ksp_richardson -pc_type sor on one processor will give you a GS
> solver.
> >>>>
> >>>>  Once we know a bit more about your problem we can suggest better
> preconditioners.
> >>>>
> >>>> Barry
> >>>>
> >>>>
> >>>>> On Feb 26, 2015, at 10:25 PM, Orxan Shibliyev <
> orxan.shibli at gmail.com> wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I tried to solve Ax=b with my own Gauss-Seidel code and Petsc's
> GMRES.
> >>>>> With my GS, for a steady state problem I can set CFL=40 and for
> >>>>> unsteady case can set dt=0.1. However, for GMRES I can't set CFL more
> >>>>> than 5 and for unsteady case dt more than 0.00001. I need GMRES for
> >>>>> parallel computations so I cannot use GS for this purpose. Is there a
> >>>>> way to improve the stability of GMRES?
> >>>>
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/fe453a77/attachment.html>

From bsmith at mcs.anl.gov  Mon Mar  9 18:21:05 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Mar 2015 18:21:05 -0500
Subject: [petsc-users] GMRES stability
In-Reply-To: <CAOYX9gyG6Pu9S3JTVTtYxTWn8NcS6K5S1pMrvXdYer0-nV8aPw@mail.gmail.com>
References: <CAOYX9gzxrd7-tHWLS6RZZK2AaO_O7tVbDw1s1b6vF=YewwCSuQ@mail.gmail.com>
	<0A826E90-DD1A-450B-BDE2-D5A4867B682B@mcs.anl.gov>
	<CAOYX9gx5e=TigrB3A0hE8G-qXJX9h83mRB+GSYwD3YE9oVATtg@mail.gmail.com>
	<239AD9F8-8DBE-42D6-B185-7BF246FC4900@mcs.anl.gov>
	<CAOYX9gzSfgswMAPnxPMNHab8As9MdkOj3g-9ZJGqPc6Rr8M4pQ@mail.gmail.com>
	<041B1984-4BFB-4A6C-92F1-8CC8DBBF0B07@mcs.anl.gov>
	<CAOYX9gyG6Pu9S3JTVTtYxTWn8NcS6K5S1pMrvXdYer0-nV8aPw@mail.gmail.com>
Message-ID: <B5979305-38D7-4399-973D-45EF95296C36@mcs.anl.gov>


> On Mar 9, 2015, at 5:23 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> 
> I use proactive point GS which updates the RHS of neighbours instead of updating the variable itself in subsequent iterations. I use natural ordering. I am sure that the matrices are the same for both solvers.

   Ok, so you are doing some very strange custom GS in your code that I have never heard of and yet you are surprised that the PETSc GS doesn't converge exactly the same? Unless that exact same "proactive point GS" was implemented in PETSc of course you are going to get different convergence behavior.

  Barry

> 
> On Sun, Mar 1, 2015 at 4:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Ok, we need to understand what is algorithmically different between what your code is doing and PETSc is doing.
> 
>   From below PETSc is running a symmetric point block SOR with a block size of 5. Are you using a point or point block GS? Are you using any particular ordering of the unknowns in your GS, or just natural ordering? Are you sure the matrices are the same between the two codes? Print them out for a tiny grid and compare.
> 
>   Barry
> 
> > On Mar 1, 2015, at 4:33 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> >
> > I don't run in parallel. I tried -ksp_richardson -pc_type but still
> > cannot get over CFL=5 (with my own GS code CFL=40 was converging).
> > Also, lowering damping factor does not help. Default number of
> > iterations (10000) does not help as well. -ksp_view outputs:
> >
> > KSP Object: 1 MPI processes
> >  type: richardson
> >    Richardson: damping factor=0.5
> >  maximum iterations=10, initial guess is zero
> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >  left preconditioning
> >  using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >  type: sor
> >    SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
> >  linear system matrix = precond matrix:
> >  Mat Object:   1 MPI processes
> >    type: seqbaij
> >    rows=13250, cols=13250, bs=5
> >    total: nonzeros=261150, allocated nonzeros=1.325e+06
> >    total number of mallocs used during MatSetValues calls =0
> >        block size is 5
> >
> > On Fri, Feb 27, 2015 at 4:30 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>
> >>  Ok, please provide the rest of the information I asked for.
> >>
> >>  Barry
> >>
> >>> On Feb 27, 2015, at 2:33 AM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> >>>
> >>> No. It does not converge at all or iow it diverges.
> >>>
> >>> On Thu, Feb 26, 2015 at 9:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>>>
> >>>> By stability I assume you mean the the GMRES does not converge (or converges too slowly)?
> >>>>
> >>>> The way to improve GMRES convergence is with a preconditioner better suited to your problem. By default PETSc uses GMRES with a block Jacobi preconditioner with one block per process and ILU(0) on each block. For some problems this is fine, but for many problems it will give bad convergence.
> >>>>
> >>>>  What do you get for -ksp_view  (are you using the default?) Are you running yet in parallel?
> >>>>
> >>>> As a test on one process you can use GS in PETSc as the preconditioner and make sure you get similar convergence to your code. For example -ksp_richardson -pc_type sor on one processor will give you a GS solver.
> >>>>
> >>>>  Once we know a bit more about your problem we can suggest better preconditioners.
> >>>>
> >>>> Barry
> >>>>
> >>>>
> >>>>> On Feb 26, 2015, at 10:25 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I tried to solve Ax=b with my own Gauss-Seidel code and Petsc's GMRES.
> >>>>> With my GS, for a steady state problem I can set CFL=40 and for
> >>>>> unsteady case can set dt=0.1. However, for GMRES I can't set CFL more
> >>>>> than 5 and for unsteady case dt more than 0.00001. I need GMRES for
> >>>>> parallel computations so I cannot use GS for this purpose. Is there a
> >>>>> way to improve the stability of GMRES?
> >>>>
> >>
> 
> 


From mrosso at uci.edu  Tue Mar 10 00:09:53 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Mon, 09 Mar 2015 22:09:53 -0700
Subject: [petsc-users] Superlu_dist installation problem
Message-ID: <1425964193.7018.2.camel@enterprise-A>

Hi,

I tried to install petsc-3.5.3 + superlu_dist using the attached script
that used to work.
Now the configure fails because of superlu_dist (see attached
configure.log).
What am I doing wrong?

Thanks,
Michele

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/4287634b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 1854380 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/4287634b/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gnu-dbg-32idx.py
Type: text/x-python
Size: 734 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150309/4287634b/attachment-0001.py>

From balay at mcs.anl.gov  Tue Mar 10 00:37:13 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 10 Mar 2015 00:37:13 -0500
Subject: [petsc-users] Superlu_dist installation problem
In-Reply-To: <1425964193.7018.2.camel@enterprise-A>
References: <1425964193.7018.2.camel@enterprise-A>
Message-ID: <alpine.LFD.2.11.1503100036350.27750@asterix>

Superlu_dist requires the following patch to work with '-Werror=format-security' compiler option..

Satish

------------

$ git diff SRC/memory.c
diff --git a/SRC/memory.c b/SRC/memory.c
index a9dcee5..902cbd1 100755
--- a/SRC/memory.c
+++ b/SRC/memory.c
@@ -36,7 +36,7 @@ superlu_abort_and_exit_dist(char *msg)
 {
     /*fprintf(stderr, msg);
     fflush(stderr);*/
-    printf(msg);
+    printf("%s",msg);
     exit (-1);
 }

On Tue, 10 Mar 2015, Michele Rosso wrote:

> Hi,
> 
> I tried to install petsc-3.5.3 + superlu_dist using the attached script
> that used to work.
> Now the configure fails because of superlu_dist (see attached
> configure.log).
> What am I doing wrong?
> 
> Thanks,
> Michele
> 
> 


From gideon.simpson at gmail.com  Tue Mar 10 03:07:42 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 10 Mar 2015 04:07:42 -0400
Subject: [petsc-users] time stepping questions
Message-ID: <08CD7EE0-72A3-433F-B554-13BBFC868F0E@gmail.com>

I had two questions on time stepping routines:

1.  What?s the proper usage of TSMonitorSolutionBinary?  As near as I can tell from source, I need to call:

PetscViewerBinaryOpen(PETSC_COMM_WORLD, "sim.bin",FILE_MODE_WRITE, &viewer);

TSMonitorSet(ts, TSMonitorSolutionBinary, viewer,(PetscErrorCode (*)(void**))PetscViewerDestroy);

I?m guessing it?s necessary to include the PetscViewerDestroy command?  Also, is there a reason why I?m not passing a pointer to the viewer, but rather just the view itself?

2.  Is there a built in way to track the time steps that the TS solver takes?  Assuming it?s using adaptivity, this could be variable.  I imagine I could use a TSMonitor routine to print it to the screen, but is there some way to get it into a file?

-gideon


From jed at jedbrown.org  Tue Mar 10 09:03:26 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 10 Mar 2015 08:03:26 -0600
Subject: [petsc-users] time stepping questions
In-Reply-To: <08CD7EE0-72A3-433F-B554-13BBFC868F0E@gmail.com>
References: <08CD7EE0-72A3-433F-B554-13BBFC868F0E@gmail.com>
Message-ID: <877fuplzkh.fsf@jedbrown.org>

Gideon Simpson <gideon.simpson at gmail.com> writes:

> I had two questions on time stepping routines:
>
> 1.  What?s the proper usage of TSMonitorSolutionBinary?  As near as I can tell from source, I need to call:
>
> PetscViewerBinaryOpen(PETSC_COMM_WORLD, "sim.bin",FILE_MODE_WRITE, &viewer);
>
> TSMonitorSet(ts, TSMonitorSolutionBinary, viewer,(PetscErrorCode (*)(void**))PetscViewerDestroy);
>
> I?m guessing it?s necessary to include the PetscViewerDestroy command?  

If you created a viewer, you need to destroy it.

> Also, is there a reason why I?m not passing a pointer to the viewer,
> but rather just the view itself?

All PETSc objects are pointers.

  ierr = PetscOptionsString("-ts_monitor_solution_binary","Save each solution to a binary file","TSMonitorSolutionBinary",0,monfilename,PETSC_MAX_PATH_LEN,&flg);CHKERRQ(ierr);
  if (flg) {
    PetscViewer ctx;
    if (monfilename[0]) {
      ierr = PetscViewerBinaryOpen(PetscObjectComm((PetscObject)ts),monfilename,FILE_MODE_WRITE,&ctx);CHKERRQ(ierr);
      ierr = TSMonitorSet(ts,TSMonitorSolutionBinary,ctx,(PetscErrorCode (*)(void**))PetscViewerDestroy);CHKERRQ(ierr);
    } else {
      ctx = PETSC_VIEWER_BINARY_(PetscObjectComm((PetscObject)ts));
      ierr = TSMonitorSet(ts,TSMonitorSolutionBinary,ctx,(PetscErrorCode (*)(void**))NULL);CHKERRQ(ierr);
    }
  }

> 2.  Is there a built in way to track the time steps that the TS solver takes?  Assuming it?s using adaptivity, this could be variable.  I imagine I could use a TSMonitor routine to print it to the screen, but is there some way to get it into a file?

Do you want output from -ts_monitor or -ts_adapt_monitor?  The former
takes a filename argument and the latter probably should (instead of
just beeing a boolean).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/40e2a71c/attachment.pgp>

From gideon.simpson at gmail.com  Tue Mar 10 09:24:35 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Tue, 10 Mar 2015 10:24:35 -0400
Subject: [petsc-users] time stepping questions
In-Reply-To: <877fuplzkh.fsf@jedbrown.org>
References: <08CD7EE0-72A3-433F-B554-13BBFC868F0E@gmail.com>
	<877fuplzkh.fsf@jedbrown.org>
Message-ID: <036E74F8-008A-4AC0-955A-B7C2875628CC@gmail.com>


-gideon

> On Mar 10, 2015, at 10:03 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Gideon Simpson <gideon.simpson at gmail.com> writes:
> 
>> I had two questions on time stepping routines:
>> 
>> 1.  What?s the proper usage of TSMonitorSolutionBinary?  As near as I can tell from source, I need to call:
>> 
>> PetscViewerBinaryOpen(PETSC_COMM_WORLD, "sim.bin",FILE_MODE_WRITE, &viewer);
>> 
>> TSMonitorSet(ts, TSMonitorSolutionBinary, viewer,(PetscErrorCode (*)(void**))PetscViewerDestroy);
>> 
>> I?m guessing it?s necessary to include the PetscViewerDestroy command?  
> 
> If you created a viewer, you need to destroy it.


But the destruction of the viewer is handled by the TSMonitor routine, and not done manually, with a PetscViewerDestroy routine?

> 
>> Also, is there a reason why I?m not passing a pointer to the viewer,
>> but rather just the view itself?
> 
> All PETSc objects are pointers.
> 
>  ierr = PetscOptionsString("-ts_monitor_solution_binary","Save each solution to a binary file","TSMonitorSolutionBinary",0,monfilename,PETSC_MAX_PATH_LEN,&flg);CHKERRQ(ierr);
>  if (flg) {
>    PetscViewer ctx;
>    if (monfilename[0]) {
>      ierr = PetscViewerBinaryOpen(PetscObjectComm((PetscObject)ts),monfilename,FILE_MODE_WRITE,&ctx);CHKERRQ(ierr);
>      ierr = TSMonitorSet(ts,TSMonitorSolutionBinary,ctx,(PetscErrorCode (*)(void**))PetscViewerDestroy);CHKERRQ(ierr);
>    } else {
>      ctx = PETSC_VIEWER_BINARY_(PetscObjectComm((PetscObject)ts));
>      ierr = TSMonitorSet(ts,TSMonitorSolutionBinary,ctx,(PetscErrorCode (*)(void**))NULL);CHKERRQ(ierr);
>    }
>  }

What I find confusing about that, though, is that you call PetscViewerBinaryOpen with &ctx, a pointer to the viewer object, and this makes sense, because the command is:

PetscErrorCode  PetscViewerBinaryOpen(MPI_Comm comm,const char name[],PetscFileMode type,PetscViewer *binv)
but for TSMonitorSet and TSMonitorSolutionBinary, it?s written as:
PetscErrorCode  TSMonitorSet(TS ts,PetscErrorCode (*monitor)(TS,PetscInt,PetscReal,Vec,void*),void *mctx,PetscErrorCode (*mdestroy)(void**))
PetscErrorCode  TSMonitorSolutionBinary(TS ts,PetscInt step,PetscReal ptime,Vec u,void *viewer)
which makes it look like i should be passing an &ctx, and not the ct. itself.  But this is a minor point.  One works, and one doesn't

> 
>> 2.  Is there a built in way to track the time steps that the TS solver takes?  Assuming it?s using adaptivity, this could be variable.  I imagine I could use a TSMonitor routine to print it to the screen, but is there some way to get it into a file?
> 
> Do you want output from -ts_monitor or -ts_adapt_monitor?  The former
> takes a filename argument and the latter probably should (instead of
> just beeing a boolean).

I was more interested in -ts_monitor.  I suppose I just need to do string processing on that to get the actual times.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/c4fc2438/attachment.html>

From jed at jedbrown.org  Tue Mar 10 09:30:03 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 10 Mar 2015 08:30:03 -0600
Subject: [petsc-users] time stepping questions
In-Reply-To: <036E74F8-008A-4AC0-955A-B7C2875628CC@gmail.com>
References: <08CD7EE0-72A3-433F-B554-13BBFC868F0E@gmail.com>
	<877fuplzkh.fsf@jedbrown.org>
	<036E74F8-008A-4AC0-955A-B7C2875628CC@gmail.com>
Message-ID: <871tkxlyc4.fsf@jedbrown.org>

Gideon Simpson <gideon.simpson at gmail.com> writes:
>> If you created a viewer, you need to destroy it.
>
> But the destruction of the viewer is handled by the TSMonitor routine, and not done manually, with a PetscViewerDestroy routine?

TS monitoring calls the destroy function that you passed to TSMonitorSet.

> What I find confusing about that, though, is that you call PetscViewerBinaryOpen with &ctx, a pointer to the viewer object, and this makes sense, because the command is:
>
> PetscErrorCode  PetscViewerBinaryOpen(MPI_Comm comm,const char name[],PetscFileMode type,PetscViewer *binv)

You're creating the PetscViewer.  C has pass-by-value semantics, so if
you want to change the handle, you have to &viewer.

> but for TSMonitorSet and TSMonitorSolutionBinary, it?s written as:
> PetscErrorCode  TSMonitorSet(TS ts,PetscErrorCode (*monitor)(TS,PetscInt,PetscReal,Vec,void*),void *mctx,PetscErrorCode (*mdestroy)(void**))
> PetscErrorCode  TSMonitorSolutionBinary(TS ts,PetscInt step,PetscReal ptime,Vec u,void *viewer)
> which makes it look like i should be passing an &ctx, and not the ct. itself.  But this is a minor point.  One works, and one doesn't

The context can be any (pointer) type, so we write void*.  In this case,
it's a

  typedef struct _p_PetscViewer *PetscViewer;

> I was more interested in -ts_monitor.  I suppose I just need to do string processing on that to get the actual times.

Sure, or have your own monitor print whatever information you want.  I
like -ts_adapt_monitor for debugging -- lots more information.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/1c359e08/attachment.pgp>

From gtheler at cites-gss.com  Tue Mar 10 10:30:20 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Tue, 10 Mar 2015 15:30:20 +0000
Subject: [petsc-users] saving views as images
In-Reply-To: <66F34E87-B7B5-4B31-BD98-664C3CB8A37E@mcs.anl.gov>
References: <C638F8EEDA1EC642B0B13905A2ADAC5C3A1354@SS10175.sancorseguros.net>
	<66F34E87-B7B5-4B31-BD98-664C3CB8A37E@mcs.anl.gov>
Message-ID: <1426001374.2659.19.camel@cites-gss.com>


On Sat, 2015-03-07 at 15:51 -0600, Barry Smith wrote:
>  Did you follow the directions in the manual page for PetscDrawSetSave()? 
>  "Requires that PETSc be configured with the option --with-afterimage to save the images and ffmpeg must be in your path to make the movie"

Yes. For some reason configure does not find libAfterImage in Debian's
default location, but it works if I provide both with-afterimage-lib and
with-afterimage-include. Debian Jessie provides packages
libafterimage-dev and libafterimage0, both marked as version 2.2.12-3.

gtheler at blinky:~/libs/petsc-3.5.3$ ./configure
--with-afterimage-lib=/usr/lib/x86_64-linux-gnu/libAfterImage.so
--with-afterimage-include=/usr/include/libAfterImage
===============================================================================
             Configuring PETSc to compile on your
system                       
===============================================================================
TESTING: alternateConfigureLibrary from
PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:74)
Compilers:
  C Compiler:         mpicc  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 
  C++ Compiler:       mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -g -O0   -fPIC  
  Fortran Compiler:   mpif90  -fPIC -Wall -Wno-unused-variable
-ffree-line-length-0 -Wno-unused-dummy-argument -g -O0  
Linkers:
  Shared linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
  Dynamic linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
make:
MPI:
  Includes: -I/usr/include/mpich
BLAS/LAPACK: -llapack -lblas
cmake:
X:
  Library:  -lX11
pthread:
  Library:  -lpthread
ssl:
  Library:  -lssl -lcrypto
afterimage:
  Includes: -I/usr/include/libAfterImage
  Library:  -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -lAfterImage
valgrind:
  Arch:     
PETSc:
  PETSC_ARCH: arch-linux2-c-debug
  PETSC_DIR: /home/gtheler/libs/petsc-3.5.3
  Clanguage: C
  shared libraries: enabled
  Scalar type: real
  Precision: double
  Memory alignment: 16
xxx=========================================================================xxx
 Configure stage complete. Now build PETSc libraries with (gnumake
build):
   make PETSC_DIR=/home/gtheler/libs/petsc-3.5.3
PETSC_ARCH=arch-linux2-c-debug all
xxx=========================================================================xxx
gtheler at blinky:~/libs/petsc-3.5.3$                      


Now, consider the attached source file (based on
mat/examples/tutorials/ex16.c). It does compile and gets linked to
libAfterImage:



gtheler at blinky:~/codigos/mat2img$ make
mpicc -c -o mat2img.o mat2img.c -Wall
-I/home/gtheler/libs/petsc-3.5.3/include
-I/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/include
-I/usr/include/libAfterImage -I/usr/include/mpich
mpicc -o mat2img mat2img.o
-Wl,-rpath,/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib
-L/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib  -lpetsc
-llapack -lblas -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -lAfterImage -lX11 -lpthread -lssl -lcrypto
-lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm
-lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread
-lgcc_s -ldl  
gtheler at blinky:~/codigos/mat2img$ ldd mat2img | grep After
        libAfterImage.so.0
=> /usr/lib/x86_64-linux-gnu/libAfterImage.so.0 (0x00007f95bc4b7000)
        libAfterBase.so.0 => /usr/lib/x86_64-linux-gnu/libAfterBase.so.0
(0x00007f95b7ad4000)
gtheler at blinky:~/codigos/mat2img$ 


However, an empty directory called "test" is created when running the
code:

gtheler at blinky:~/codigos/mat2img$ ./mat2img -info
[0] PetscInitialize(): PETSc successfully started: number of processors
= 1
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscInitialize(): Running on machine: blinky
[0] PetscCommDuplicate(): Duplicating a communicator 1140850688
-2080374784 max tags = 536870911
[0] PetscGetFileStat(): System call stat() succeeded on file matrix.bin
[0] PetscTestOwnership(): System call access() succeeded on file
matrix.bin
[0] PetscFileRetrieve(): Found file matrix.bin
[0] PetscGetFileStat(): System call stat() failed on file
matrix.bin.info
[0] PetscFileRetrieve(): Did not find file matrix.bin.info
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0
unneeded,58 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
[0] Mat_CheckInode(): Found 20 nodes out of 20 rows. Not using Inode
routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] PetscDrawSetUpColormap_Shared(): Successfully allocated colors
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :rm -fr matrix matrix.m4v
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :mkdir matrix
[0] PetscFinalize(): PetscFinalize() called
gtheler at blinky:~/codigos/mat2img$ 



I cannot find out what I am missing.




>  So you have to install afterimage on your machine and then ./configure PETSc so it knows about it. If you have done all that and PETSc is being linked with the afterimage libraries then run your code with -info and send all the output; it may indicate why the images are not being saved.
> 
>   Barry
> 
> 
> > On Mar 7, 2015, at 3:38 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> > 
> > Hi
> > 
> > Last week I asked how to dump a matrix structure into an image and Barry kindly pointed me out the  PetscDrawSetSave function. But I cannot figure out how to use it. So far, I can show my matrix structure into an X window with something like
> > 
> >    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
> >    MatView(milonga.R, viewer);
> > 
> > I tried with this snippet of code but had no luck:
> > 
> >    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
> >    MatView(milonga.R, viewer);
> >    PetscViewerDrawGetDraw(viewer, 0, &draw);
> >    PetscDrawFlush(draw);
> >    PetscDrawSetSave(draw, "test.gif", PETSC_FALSE);
> >    PetscDrawSetSaveFinalImage(draw, "test2.gif");
> > 
> > The result is that test.gif and test2.gif are created but they are empty directories.
> > What am I missing?
> > 
> > --
> > jeremy
> > 
> > ________________________________
> > Imprima este mensaje s?lo si es absolutamente necesario.
> > Para imprimir, en lo posible utilice el papel de ambos lados.
> > El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> > 
> > 
> > 
> > ************AVISO DE CONFIDENCIALIDAD************
> > 
> > El Grupo Sancor Seguros comunica que:
> > 
> > Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> > 
> > Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mat2img.c
Type: text/x-csrc
Size: 1804 bytes
Desc: mat2img.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/45b1d8e5/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: text/x-makefile
Size: 144 bytes
Desc: makefile
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/45b1d8e5/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: matrix.bin
Type: application/octet-stream
Size: 792 bytes
Desc: matrix.bin
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/45b1d8e5/attachment-0003.bin>

From scanmail at anl.gov  Tue Mar 10 10:31:05 2015
From: scanmail at anl.gov (Administrator)
Date: Tue, 10 Mar 2015 10:31:05 -0500
Subject: [petsc-users] [MailServer Notification]Argonne Antivirus Quarantine
	Notification - DO NOT REPLY
Message-ID: <89BC023A4CFB4B6990114796B43375E6@anl.gov>

Do not reply to this message. The reply address is not monitored.

The message below has been quarantined by the Argonne National Laboratory 
Antivirus filtering system.  The message was filtered for having been 
detected of having malicious content or an attachment that matches the 
laboratory?s filtering criteria.

From: gtheler at cites-gss.com;
To:  petsc-users at mcs.anl.gov;
Subject:  Re: [petsc-users] saving views as images
Attachment:  matrix.bin
Date:  3/10/2015 10:30:28 AM

If you have any questions regarding the Argonne's antivirus filtering 
product, or feel that the attachment was incorrectly identified, please 
contact the CIS Service Desk at help at anl.gov or x-9999 option 2.

From scanmail at anl.gov  Tue Mar 10 10:31:05 2015
From: scanmail at anl.gov (Administrator)
Date: Tue, 10 Mar 2015 10:31:05 -0500
Subject: [petsc-users] [MailServer Notification]Argonne Antivirus Quarantine
	Notification - DO NOT REPLY
Message-ID: <BE130DBF4E6B4436AF98F7B0DC9FB949@anl.gov>

Do not reply to this message. The reply address is not monitored.

The message below has been quarantined by the Argonne National Laboratory 
Antivirus filtering system.  The message was filtered for having been 
detected of having malicious content or an attachment that matches the 
laboratory?s filtering criteria.

From: gtheler at cites-gss.com;
To:  petsc-users at mcs.anl.gov;
Subject:  Re: [petsc-users] saving views as images
Attachment:  matrix.bin
Date:  3/10/2015 10:30:28 AM

If you have any questions regarding the Argonne's antivirus filtering 
product, or feel that the attachment was incorrectly identified, please 
contact the CIS Service Desk at help at anl.gov or x-9999 option 2.

From scanmail at anl.gov  Tue Mar 10 10:31:18 2015
From: scanmail at anl.gov (Administrator)
Date: Tue, 10 Mar 2015 10:31:18 -0500
Subject: [petsc-users] [MailServer Notification]Argonne Antivirus Quarantine
	Notification - DO NOT REPLY
Message-ID: <588473A572354AD08FF368602F13AC81@anl.gov>

Do not reply to this message. The reply address is not monitored.

The message below has been quarantined by the Argonne National Laboratory 
Antivirus filtering system.  The message was filtered for having been 
detected of having malicious content or an attachment that matches the 
laboratory?s filtering criteria.

From: gtheler at cites-gss.com;
To:  petsc-users at mcs.anl.gov;
Subject:  Re: [petsc-users] saving views as images
Attachment:  matrix.bin
Date:  3/10/2015 10:30:28 AM

If you have any questions regarding the Argonne's antivirus filtering 
product, or feel that the attachment was incorrectly identified, please 
contact the CIS Service Desk at help at anl.gov or x-9999 option 2.

From gtheler at cites-gss.com  Tue Mar 10 10:47:04 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Tue, 10 Mar 2015 15:47:04 +0000
Subject: [petsc-users] [Fwd: Re:  saving views as images]
Message-ID: <1426002378.2659.21.camel@cites-gss.com>


On Sat, 2015-03-07 at 15:51 -0600, Barry Smith wrote:
>  Did you follow the directions in the manual page for PetscDrawSetSave()? 
>  "Requires that PETSc be configured with the option --with-afterimage to save the images and ffmpeg must be in your path to make the movie"

Yes. For some reason configure does not find libAfterImage in Debian's
default location, but it works if I provide both with-afterimage-lib and
with-afterimage-include. Debian Jessie provides packages
libafterimage-dev and libafterimage0, both marked as version 2.2.12-3.

gtheler at blinky:~/libs/petsc-3.5.3$ ./configure
--with-afterimage-lib=/usr/lib/x86_64-linux-gnu/libAfterImage.so
--with-afterimage-include=/usr/include/libAfterImage
===============================================================================
             Configuring PETSc to compile on your
system                       
===============================================================================
TESTING: alternateConfigureLibrary from
PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:74)
Compilers:
  C Compiler:         mpicc  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 
  C++ Compiler:       mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -g -O0   -fPIC  
  Fortran Compiler:   mpif90  -fPIC -Wall -Wno-unused-variable
-ffree-line-length-0 -Wno-unused-dummy-argument -g -O0  
Linkers:
  Shared linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
  Dynamic linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
make:
MPI:
  Includes: -I/usr/include/mpich
BLAS/LAPACK: -llapack -lblas
cmake:
X:
  Library:  -lX11
pthread:
  Library:  -lpthread
ssl:
  Library:  -lssl -lcrypto
afterimage:
  Includes: -I/usr/include/libAfterImage
  Library:  -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -lAfterImage
valgrind:
  Arch:     
PETSc:
  PETSC_ARCH: arch-linux2-c-debug
  PETSC_DIR: /home/gtheler/libs/petsc-3.5.3
  Clanguage: C
  shared libraries: enabled
  Scalar type: real
  Precision: double
  Memory alignment: 16
xxx=========================================================================xxx
 Configure stage complete. Now build PETSc libraries with (gnumake
build):
   make PETSC_DIR=/home/gtheler/libs/petsc-3.5.3
PETSC_ARCH=arch-linux2-c-debug all
xxx=========================================================================xxx
gtheler at blinky:~/libs/petsc-3.5.3$                      


Now, consider the attached source file (based on
mat/examples/tutorials/ex16.c). It does compile and gets linked to
libAfterImage:



gtheler at blinky:~/codigos/mat2img$ make
mpicc -c -o mat2img.o mat2img.c -Wall
-I/home/gtheler/libs/petsc-3.5.3/include
-I/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/include
-I/usr/include/libAfterImage -I/usr/include/mpich
mpicc -o mat2img mat2img.o
-Wl,-rpath,/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib
-L/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib  -lpetsc
-llapack -lblas -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -lAfterImage -lX11 -lpthread -lssl -lcrypto
-lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm
-lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread
-lgcc_s -ldl  
gtheler at blinky:~/codigos/mat2img$ ldd mat2img | grep After
        libAfterImage.so.0
=> /usr/lib/x86_64-linux-gnu/libAfterImage.so.0 (0x00007f95bc4b7000)
        libAfterBase.so.0 => /usr/lib/x86_64-linux-gnu/libAfterBase.so.0
(0x00007f95b7ad4000)
gtheler at blinky:~/codigos/mat2img$ 


However, an empty directory called "test" is created when running the
code:

gtheler at blinky:~/codigos/mat2img$ ./mat2img -info
[0] PetscInitialize(): PETSc successfully started: number of processors
= 1
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscInitialize(): Running on machine: blinky
[0] PetscCommDuplicate(): Duplicating a communicator 1140850688
-2080374784 max tags = 536870911
[0] PetscGetFileStat(): System call stat() succeeded on file matrix.bin
[0] PetscTestOwnership(): System call access() succeeded on file
matrix.bin
[0] PetscFileRetrieve(): Found file matrix.bin
[0] PetscGetFileStat(): System call stat() failed on file
matrix.bin.info
[0] PetscFileRetrieve(): Did not find file matrix.bin.info
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0
unneeded,58 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
[0] Mat_CheckInode(): Found 20 nodes out of 20 rows. Not using Inode
routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] PetscDrawSetUpColormap_Shared(): Successfully allocated colors
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :rm -fr matrix matrix.m4v
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :mkdir matrix
[0] PetscFinalize(): PetscFinalize() called
gtheler at blinky:~/codigos/mat2img$ 



I cannot find out what I am missing.




>  So you have to install afterimage on your machine and then ./configure PETSc so it knows about it. If you have done all that and PETSc is being linked with the afterimage libraries then run your code with -info and send all the output; it may indicate why the images are not being saved.
> 
>   Barry
> 
> 
> > On Mar 7, 2015, at 3:38 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> > 
> > Hi
> > 
> > Last week I asked how to dump a matrix structure into an image and Barry kindly pointed me out the  PetscDrawSetSave function. But I cannot figure out how to use it. So far, I can show my matrix structure into an X window with something like
> > 
> >    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
> >    MatView(milonga.R, viewer);
> > 
> > I tried with this snippet of code but had no luck:
> > 
> >    PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
> >    MatView(milonga.R, viewer);
> >    PetscViewerDrawGetDraw(viewer, 0, &draw);
> >    PetscDrawFlush(draw);
> >    PetscDrawSetSave(draw, "test.gif", PETSC_FALSE);
> >    PetscDrawSetSaveFinalImage(draw, "test2.gif");
> > 
> > The result is that test.gif and test2.gif are created but they are empty directories.
> > What am I missing?
> > 
> > --
> > jeremy
> > 
> > ________________________________
> > Imprima este mensaje s?lo si es absolutamente necesario.
> > Para imprimir, en lo posible utilice el papel de ambos lados.
> > El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> > 
> > 
> > 
> > ************AVISO DE CONFIDENCIALIDAD************
> > 
> > El Grupo Sancor Seguros comunica que:
> > 
> > Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> > 
> > Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mat2img.c
Type: text/x-csrc
Size: 1804 bytes
Desc: mat2img.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/59858ec6/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: text/x-makefile
Size: 144 bytes
Desc: makefile
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/59858ec6/attachment.bin>

From bsmith at mcs.anl.gov  Tue Mar 10 11:35:14 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 10 Mar 2015 11:35:14 -0500
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <1426002378.2659.21.camel@cites-gss.com>
References: <1426002378.2659.21.camel@cites-gss.com>
Message-ID: <78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>


   You must call 

  call(PetscDrawSetSave(draw, "matrix", PETSC_FALSE));
  call(PetscDrawSetSaveFinalImage(draw, "matrix2"));

BEFORE you view the image.  PetscDrawSetSave() sets the flag to save future images; generally you could this immediately after you've done the PetscViewerDrawOpen


  Barry

  You don't need the flush. 
> On Mar 10, 2015, at 10:47 AM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> 
> On Sat, 2015-03-07 at 15:51 -0600, Barry Smith wrote:
>> Did you follow the directions in the manual page for PetscDrawSetSave()? 
>> "Requires that PETSc be configured with the option --with-afterimage to save the images and ffmpeg must be in your path to make the movie"
> 
> Yes. For some reason configure does not find libAfterImage in Debian's
> default location, but it works if I provide both with-afterimage-lib and
> with-afterimage-include. Debian Jessie provides packages
> libafterimage-dev and libafterimage0, both marked as version 2.2.12-3.
> 
> gtheler at blinky:~/libs/petsc-3.5.3$ ./configure
> --with-afterimage-lib=/usr/lib/x86_64-linux-gnu/libAfterImage.so
> --with-afterimage-include=/usr/include/libAfterImage
> ===============================================================================
>             Configuring PETSc to compile on your
> system                       
> ===============================================================================
> TESTING: alternateConfigureLibrary from
> PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:74)
> Compilers:
>  C Compiler:         mpicc  -fPIC -Wall -Wwrite-strings
> -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 
>  C++ Compiler:       mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -g -O0   -fPIC  
>  Fortran Compiler:   mpif90  -fPIC -Wall -Wno-unused-variable
> -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0  
> Linkers:
>  Shared linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
> -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
>  Dynamic linker:   mpicc  -shared  -fPIC -Wall -Wwrite-strings
> -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
> make:
> MPI:
>  Includes: -I/usr/include/mpich
> BLAS/LAPACK: -llapack -lblas
> cmake:
> X:
>  Library:  -lX11
> pthread:
>  Library:  -lpthread
> ssl:
>  Library:  -lssl -lcrypto
> afterimage:
>  Includes: -I/usr/include/libAfterImage
>  Library:  -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -lAfterImage
> valgrind:
>  Arch:     
> PETSc:
>  PETSC_ARCH: arch-linux2-c-debug
>  PETSC_DIR: /home/gtheler/libs/petsc-3.5.3
>  Clanguage: C
>  shared libraries: enabled
>  Scalar type: real
>  Precision: double
>  Memory alignment: 16
> xxx=========================================================================xxx
> Configure stage complete. Now build PETSc libraries with (gnumake
> build):
>   make PETSC_DIR=/home/gtheler/libs/petsc-3.5.3
> PETSC_ARCH=arch-linux2-c-debug all
> xxx=========================================================================xxx
> gtheler at blinky:~/libs/petsc-3.5.3$                      
> 
> 
> Now, consider the attached source file (based on
> mat/examples/tutorials/ex16.c). It does compile and gets linked to
> libAfterImage:
> 
> 
> 
> gtheler at blinky:~/codigos/mat2img$ make
> mpicc -c -o mat2img.o mat2img.c -Wall
> -I/home/gtheler/libs/petsc-3.5.3/include
> -I/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/include
> -I/usr/include/libAfterImage -I/usr/include/mpich
> mpicc -o mat2img mat2img.o
> -Wl,-rpath,/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib
> -L/home/gtheler/libs/petsc-3.5.3/arch-linux2-c-debug/lib  -lpetsc
> -llapack -lblas -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -lAfterImage -lX11 -lpthread -lssl -lcrypto
> -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm
> -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread
> -lgcc_s -ldl  
> gtheler at blinky:~/codigos/mat2img$ ldd mat2img | grep After
>        libAfterImage.so.0
> => /usr/lib/x86_64-linux-gnu/libAfterImage.so.0 (0x00007f95bc4b7000)
>        libAfterBase.so.0 => /usr/lib/x86_64-linux-gnu/libAfterBase.so.0
> (0x00007f95b7ad4000)
> gtheler at blinky:~/codigos/mat2img$ 
> 
> 
> However, an empty directory called "test" is created when running the
> code:
> 
> gtheler at blinky:~/codigos/mat2img$ ./mat2img -info
> [0] PetscInitialize(): PETSc successfully started: number of processors
> = 1
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscInitialize(): Running on machine: blinky
> [0] PetscCommDuplicate(): Duplicating a communicator 1140850688
> -2080374784 max tags = 536870911
> [0] PetscGetFileStat(): System call stat() succeeded on file matrix.bin
> [0] PetscTestOwnership(): System call access() succeeded on file
> matrix.bin
> [0] PetscFileRetrieve(): Found file matrix.bin
> [0] PetscGetFileStat(): System call stat() failed on file
> matrix.bin.info
> [0] PetscFileRetrieve(): Did not find file matrix.bin.info
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374784
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0
> unneeded,58 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
> [0] Mat_CheckInode(): Found 20 nodes out of 20 rows. Not using Inode
> routines
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374784
> [0] PetscDrawSetUpColormap_Shared(): Successfully allocated colors
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :rm -fr matrix matrix.m4v
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :mkdir matrix
> [0] PetscFinalize(): PetscFinalize() called
> gtheler at blinky:~/codigos/mat2img$ 
> 
> 
> 
> I cannot find out what I am missing.
> 
> 
> 
> 
>> So you have to install afterimage on your machine and then ./configure PETSc so it knows about it. If you have done all that and PETSc is being linked with the afterimage libraries then run your code with -info and send all the output; it may indicate why the images are not being saved.
>> 
>>  Barry
>> 
>> 
>>> On Mar 7, 2015, at 3:38 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
>>> 
>>> Hi
>>> 
>>> Last week I asked how to dump a matrix structure into an image and Barry kindly pointed me out the  PetscDrawSetSave function. But I cannot figure out how to use it. So far, I can show my matrix structure into an X window with something like
>>> 
>>>   PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
>>>   MatView(milonga.R, viewer);
>>> 
>>> I tried with this snippet of code but had no luck:
>>> 
>>>   PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, "R", 100, 100, size, size, &viewer);
>>>   MatView(milonga.R, viewer);
>>>   PetscViewerDrawGetDraw(viewer, 0, &draw);
>>>   PetscDrawFlush(draw);
>>>   PetscDrawSetSave(draw, "test.gif", PETSC_FALSE);
>>>   PetscDrawSetSaveFinalImage(draw, "test2.gif");
>>> 
>>> The result is that test.gif and test2.gif are created but they are empty directories.
>>> What am I missing?
>>> 
>>> --
>>> jeremy
>>> 
>>> ________________________________
>>> Imprima este mensaje s?lo si es absolutamente necesario.
>>> Para imprimir, en lo posible utilice el papel de ambos lados.
>>> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
>>> 
>>> 
>>> 
>>> ************AVISO DE CONFIDENCIALIDAD************
>>> 
>>> El Grupo Sancor Seguros comunica que:
>>> 
>>> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
>>> 
>>> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.
>> 
> <mat2img.c><makefile>


From gtheler at cites-gss.com  Tue Mar 10 12:14:08 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Tue, 10 Mar 2015 17:14:08 +0000
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
Message-ID: <1426007602.2659.25.camel@cites-gss.com>


On Tue, 2015-03-10 at 11:35 -0500, Barry Smith wrote:
>    You must call
>
>   call(PetscDrawSetSave(draw, "matrix", PETSC_FALSE));
>   call(PetscDrawSetSaveFinalImage(draw, "matrix2"));
>
> BEFORE you view the image.  PetscDrawSetSave() sets the flag to save future images; generally you could this immediately after you've done the PetscViewerDrawOpen

Got it Barry, so now I have:

  call(PetscViewerBinaryOpen(PETSC_COMM_WORLD,file,FILE_MODE_READ,&fd));
  call(MatCreate(PETSC_COMM_WORLD,&A));
  call(MatSetFromOptions(A));
  call(MatLoad(A,fd));

  call(PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, file, 100, 100,
size, size, &viewer));
  call(PetscViewerDrawGetDraw(viewer, 0, &draw));
  call(PetscDrawSetFromOptions(draw));
  call(PetscDrawSetSave(draw, "matrix", PETSC_FALSE));
  call(PetscDrawSetSaveFinalImage(draw, "matrix2"));
  call(MatView(A, viewer));

  call(PetscDrawDestroy(&draw));
  call(PetscViewerDestroy(&fd));
  call(MatDestroy(&A));
  call(PetscFinalize());

But now I get this dissapointing message from libafterimage:

gtheler at blinky:~/codigos/mat2img$ ./mat2img -info
[0] PetscInitialize(): PETSc successfully started: number of processors
= 1
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscInitialize(): Running on machine: blinky
[0] PetscCommDuplicate(): Duplicating a communicator 1140850688
-2080374784 max tags = 536870911
[0] PetscGetFileStat(): System call stat() succeeded on file matrix.bin
[0] PetscTestOwnership(): System call access() succeeded on file
matrix.bin
[0] PetscFileRetrieve(): Found file matrix.bin
[0] PetscGetFileStat(): System call stat() failed on file
matrix.bin.info
[0] PetscFileRetrieve(): Did not find file matrix.bin.info
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0
unneeded,58 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
[0] Mat_CheckInode(): Found 20 nodes out of 20 rows. Not using Inode
routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] PetscDrawSetUpColormap_Shared(): Successfully allocated colors
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :rm -fr matrix matrix.m4v
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :mkdir matrix
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :rm -fr matrix2 matrix2.m4v
[0] PetscGetHostName(): Rejecting domainname, likely is NIS
blinky.(none)
[0] PetscPOpen(): Running command :mkdir matrix2
unknown ERROR: unable to write file "matrix2/matrix2_0.Gif" - GIF image
format is not supported.

[0] PetscFinalize(): PetscFinalize() called
gtheler at blinky:~/codigos/mat2img$


________________________________
 Imprima este mensaje s?lo si es absolutamente necesario.
Para imprimir, en lo posible utilice el papel de ambos lados.
El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.



************AVISO DE CONFIDENCIALIDAD************

El Grupo Sancor Seguros comunica que:

Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.

Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.

From bsmith at mcs.anl.gov  Tue Mar 10 13:02:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 10 Mar 2015 13:02:48 -0500
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <1426007602.2659.25.camel@cites-gss.com>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
	<1426007602.2659.25.camel@cites-gss.com>
Message-ID: <E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>


  Looks like your afterimage is not installed to support .Gif files. When I install afterimage on my Mac from source http://www.afterstep.org/afterimage/getcode.php it just comes with support for .Gif files. Suggest trying that.

  Barry

> On Mar 10, 2015, at 12:14 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> 
> On Tue, 2015-03-10 at 11:35 -0500, Barry Smith wrote:
>>   You must call
>> 
>>  call(PetscDrawSetSave(draw, "matrix", PETSC_FALSE));
>>  call(PetscDrawSetSaveFinalImage(draw, "matrix2"));
>> 
>> BEFORE you view the image.  PetscDrawSetSave() sets the flag to save future images; generally you could this immediately after you've done the PetscViewerDrawOpen
> 
> Got it Barry, so now I have:
> 
>  call(PetscViewerBinaryOpen(PETSC_COMM_WORLD,file,FILE_MODE_READ,&fd));
>  call(MatCreate(PETSC_COMM_WORLD,&A));
>  call(MatSetFromOptions(A));
>  call(MatLoad(A,fd));
> 
>  call(PetscViewerDrawOpen(PETSC_COMM_WORLD, PETSC_NULL, file, 100, 100,
> size, size, &viewer));
>  call(PetscViewerDrawGetDraw(viewer, 0, &draw));
>  call(PetscDrawSetFromOptions(draw));
>  call(PetscDrawSetSave(draw, "matrix", PETSC_FALSE));
>  call(PetscDrawSetSaveFinalImage(draw, "matrix2"));
>  call(MatView(A, viewer));
> 
>  call(PetscDrawDestroy(&draw));
>  call(PetscViewerDestroy(&fd));
>  call(MatDestroy(&A));
>  call(PetscFinalize());
> 
> But now I get this dissapointing message from libafterimage:
> 
> gtheler at blinky:~/codigos/mat2img$ ./mat2img -info
> [0] PetscInitialize(): PETSc successfully started: number of processors
> = 1
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscInitialize(): Running on machine: blinky
> [0] PetscCommDuplicate(): Duplicating a communicator 1140850688
> -2080374784 max tags = 536870911
> [0] PetscGetFileStat(): System call stat() succeeded on file matrix.bin
> [0] PetscTestOwnership(): System call access() succeeded on file
> matrix.bin
> [0] PetscFileRetrieve(): Found file matrix.bin
> [0] PetscGetFileStat(): System call stat() failed on file
> matrix.bin.info
> [0] PetscFileRetrieve(): Did not find file matrix.bin.info
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374784
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 20 X 20; storage space: 0
> unneeded,58 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 20) < 0.6. Do not use CompressedRow routines.
> [0] Mat_CheckInode(): Found 20 nodes out of 20 rows. Not using Inode
> routines
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374784
> [0] PetscDrawSetUpColormap_Shared(): Successfully allocated colors
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :rm -fr matrix matrix.m4v
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :mkdir matrix
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :rm -fr matrix2 matrix2.m4v
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS
> blinky.(none)
> [0] PetscPOpen(): Running command :mkdir matrix2
> unknown ERROR: unable to write file "matrix2/matrix2_0.Gif" - GIF image
> format is not supported.
> 
> [0] PetscFinalize(): PetscFinalize() called
> gtheler at blinky:~/codigos/mat2img$
> 
> 
> ________________________________
> Imprima este mensaje s?lo si es absolutamente necesario.
> Para imprimir, en lo posible utilice el papel de ambos lados.
> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> 
> 
> 
> ************AVISO DE CONFIDENCIALIDAD************
> 
> El Grupo Sancor Seguros comunica que:
> 
> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> 
> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.


From gtheler at cites-gss.com  Tue Mar 10 13:58:01 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Tue, 10 Mar 2015 18:58:01 +0000
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
	<1426007602.2659.25.camel@cites-gss.com>
	<E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>
Message-ID: <1426013835.2659.37.camel@cites-gss.com>


On Tue, 2015-03-10 at 13:02 -0500, Barry Smith wrote:
>   Looks like your afterimage is not installed to support .Gif files. When I install afterimage on my Mac from source http://www.afterstep.org/afterimage/getcode.php it just comes with support for .Gif files. Suggest trying that.

Yes, I realized that. Perhaps there is something about the GIF license
that the Debian team had to remove GIF support. I have tried to search
in the changelogs but could not find any reference. BTW, as I wrote in
my earlier emails, Debian provides version AfterImage 2.2.12 whilst in
the URL you quote (which is the only one that Google returns) the latest
version is 1.20, which I cannot have PETSc to use (see the configure.log
transcript below).

My workwaround was to change the format in which PETSc asks AfterImage
to dump the draw, i.e. line 395
of /home/gtheler/libs/petsc-3.5.3/src/sys/classes/draw/impls/x/xinit.c:

ASImage2file(asimage, 0, filename,ASIT_Gif,0);

So far I tried ASIT_Png but the library complains about not being able
to initialize zlib, but I think I should be able to fix. ASIT_Bmp works
fine.

Is there anything special about GIF within PETSc? IIRC, there are indeed
some issues with the license and this format was virtually replaced with
PNG almost all over the place. Maybe it would be a good idea to add a
database key so as to select the format in PetscDrawSetSave().


According to AfterImage API documentation, valid file types are

typedef enum
{
    ASIT_Xpm = 0,
    ASIT_ZCompressedXpm,
    ASIT_GZCompressedXpm,
    ASIT_Png,
    ASIT_Jpeg,
    ASIT_Xcf,
    ASIT_Ppm,
    ASIT_Pnm,
    ASIT_Bmp,
    ASIT_Ico,
    ASIT_Cur,
    ASIT_Gif,
    ASIT_Tiff,
    ASIT_XMLScript,
    ASIT_SVG,
    /* reserved for future implementation : */
    ASIT_Xbm,
    ASIT_Targa,
    ASIT_Supported = ASIT_Targa,
    ASIT_Pcx,
    ASIT_HTML,
    ASIT_XML,
    ASIT_Unknown
}ASImageFileTypes;


Output of configure.log with libAfterImage 1.20 as downloaded from
http://www.afterstep.org/afterimage/getcode.php:


Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char picture_ximage2asimage();
static void _check_picture_ximage2asimage()
{ picture_ximage2asimage(); }

int main() {
_check_picture_ximage2asimage();;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-s8uvAW/config.libraries/conftest
-fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-g3 -O0 /tmp/petsc-s8uvAW/config.libraries/conftest.o
-Wl,-rpath,/home/gtheler/libs/libAfterImage-1.20
-L/home/gtheler/libs/libAfterImage-1.20 -lAfterImage -lX11
-Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread
-lgcc_s -ldl
Possible ERROR while running linker: exit code 256
stderr:
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
function `asimage2drawable_gl':
ximage.c:(.text+0x1178): undefined reference to `show_warning'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
function `asimage2drawable':
ximage.c:(.text+0x1292): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
function `asimage2alpha_drawable':
ximage.c:(.text+0x13ac): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
function `asimage2alpha':
ximage.c:(.text+0x157c): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `asimage_start':
asimage.c:(.text+0x1bb): undefined reference to `safecalloc'
asimage.c:(.text+0x22e): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `create_asimage':
asimage.c:(.text+0x38d): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `destroy_asimage':
asimage.c:(.text+0x865): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `create_image_manager':
asimage.c:(.text+0x9bf): undefined reference to `mystrdup'
asimage.c:(.text+0x9f9): undefined reference to `string_compare'
asimage.c:(.text+0xa00): undefined reference to `string_hash_value'
asimage.c:(.text+0xa1e): undefined reference to `create_ashash'
asimage.c:(.text+0xa43): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `destroy_image_manager':
asimage.c:(.text+0xa74): undefined reference to `destroy_ashash'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `store_asimage':
asimage.c:(.text+0xb19): undefined reference to `mystrdup'
asimage.c:(.text+0xb43): undefined reference to `add_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `query_asimage':
asimage.c:(.text+0xbad): undefined reference to `get_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `fetch_asimage':
asimage.c:(.text+0xc0d): undefined reference to `get_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `dup_asimage':
asimage.c:(.text+0xc71): undefined reference to `show_error'
asimage.c:(.text+0xca4): undefined reference to `show_debug'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `release_asimage':
asimage.c:(.text+0xd0b): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `forget_asimage':
asimage.c:(.text+0xd77): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `relocate_asimage':
asimage.c:(.text+0xdb9): undefined reference to `remove_hash_item'
asimage.c:(.text+0xde3): undefined reference to `add_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `safe_asimage_destroy':
asimage.c:(.text+0xe9a): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `release_asimage_by_name':
asimage.c:(.text+0xedd): undefined reference to `get_hash_item'
asimage.c:(.text+0xf36): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `flip_gradient':
asimage.c:(.text+0x1006): undefined reference to `safecalloc'
asimage.c:(.text+0x1031): undefined reference to `safemalloc'
asimage.c:(.text+0x103d): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `create_image_layers':
asimage.c:(.text+0x1226): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `destroy_image_layers':
asimage.c:(.text+0x12e7): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `asimage_compare_line':
asimage.c:(.text+0x1a9e): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `set_asimage_vector':
asimage.c:(.text+0x1dac): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `vectorize_asimage':
asimage.c:(.text+0x1f6a): undefined reference to `safecalloc'
asimage.c:(.text+0x1f80): undefined reference to `safemalloc'
asimage.c:(.text+0x1f90): undefined reference to `safemalloc'
asimage.c:(.text+0x1fa0): undefined reference to `safemalloc'
asimage.c:(.text+0x1fb0): undefined reference to `safemalloc'
asimage.c:(.text+0x1fc0): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o):asimage.c:(.text+0x2143): more undefined references to `safemalloc' follow
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
function `forget_asimage_name':
asimage.c:(.text+0xe22): undefined reference to `remove_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ascmap.o): In
function `color_hash2colormap':
ascmap.c:(.text+0x564): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ascmap.o): In
function `colormap_asimage':
ascmap.c:(.text+0x858): undefined reference to `safemalloc'
ascmap.c:(.text+0x88b): undefined reference to `safecalloc'
ascmap.c:(.text+0x89f): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `store_data_in_block':
asstorage.c:(.text+0x530f): undefined reference to `show_error'
asstorage.c:(.text+0x532d): undefined reference to `show_error'
asstorage.c:(.text+0x5359): undefined reference to `show_error'
asstorage.c:(.text+0x5380): undefined reference to `show_error'
asstorage.c:(.text+0x53a9): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o):asstorage.c:(.text+0x57cd): more undefined references to `show_error' follow
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `store_data_in_block':
asstorage.c:(.text+0x5afd): undefined reference to `show_warning'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `store_compressed_data':
asstorage.c:(.text+0x5dcf): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `query_storage_slot':
asstorage.c:(.text+0x8172): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `print_storage_slot':
asstorage.c:(.text+0x82f4): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `forget_data':
asstorage.c:(.text+0x8642): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `dup_data':
asstorage.c:(.text+0x87cc): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o):asstorage.c:(.text+0x89f5): more undefined references to `show_error' follow
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
function `dup_data':
asstorage.c:(.text+0x8b4c): undefined reference to `show_warning'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `ximage2scanline_pseudo12bpp':
asvisual.c:(.text+0x2305): undefined reference to `get_hash_item'
asvisual.c:(.text+0x2402): undefined reference to `get_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `query_screen_visual_id':
asvisual.c:(.text+0x26f3): undefined reference to `get_output_threshold'
asvisual.c:(.text+0x27ca): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `destroy_asvisual':
asvisual.c:(.text+0x28a8): undefined reference to `destroy_ashash'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `visual2visual_prop':
asvisual.c:(.text+0x293b): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `visual_prop2visual':
asvisual.c:(.text+0x2be5): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `make_reverse_colormap':
asvisual.c:(.text+0x2fc6): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `make_reverse_colorhash':
asvisual.c:(.text+0x3062): undefined reference to `create_ashash'
asvisual.c:(.text+0x30c7): undefined reference to `add_hash_item'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `setup_as_colormap':
asvisual.c:(.text+0x338e): undefined reference to `safemalloc'
asvisual.c:(.text+0x34ca): undefined reference to `safemalloc'
asvisual.c:(.text+0x3642): undefined reference to `safemalloc'
asvisual.c:(.text+0x37ef): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `create_asvisual_for_id':
asvisual.c:(.text+0x3e81): undefined reference to `safecalloc'
asvisual.c:(.text+0x3f0c): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `create_visual_window':
asvisual.c:(.text+0x4118): undefined reference to `show_warning'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
function `create_visual_ximage':
asvisual.c:(.text+0x4654): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(blender.o): In
function `blend_scanlines_name2func':
blender.c:(.text+0x1821): undefined reference to `mystrncasecmp'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
function `create_image_xim':
imencdec.c:(.text+0xe71): undefined reference to `show_error'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
function `create_image_argb32':
imencdec.c:(.text+0x1e20): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
function `start_image_decoding':
imencdec.c:(.text+0x9218): undefined reference to `safecalloc'
imencdec.c:(.text+0x9529): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
function `start_image_output':
imencdec.c:(.text+0x97e2): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
function `prepare_scanline':
scanline.c:(.text+0x179): undefined reference to `safecalloc'
scanline.c:(.text+0x25b): undefined reference to `safecalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o):scanline.c:(.text+0x3a8): more undefined references to `safecalloc' follow
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
function `calculate_green_diff':
scanline.c:(.text+0x16b1): undefined reference to `safemalloc'
/home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
function `interpolate_green_diff':
scanline.c:(.text+0x19a3): undefined reference to `safemalloc'
collect2: error: ld returned 1 exit status
              Popping language C


________________________________
 Imprima este mensaje s?lo si es absolutamente necesario.
Para imprimir, en lo posible utilice el papel de ambos lados.
El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.



************AVISO DE CONFIDENCIALIDAD************

El Grupo Sancor Seguros comunica que:

Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.

Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.

From knepley at gmail.com  Tue Mar 10 15:15:46 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 10 Mar 2015 15:15:46 -0500
Subject: [petsc-users] DMView and DMLoad
In-Reply-To: <CAC+VmGeK8+4NOXKj+9CerkkJxVNCorz7D_fM-fPjmSuJv6p63g@mail.gmail.com>
References: <CAC+VmGeK8+4NOXKj+9CerkkJxVNCorz7D_fM-fPjmSuJv6p63g@mail.gmail.com>
Message-ID: <CAMYG4G=iwUeYA_FfJEy0rsyzG9LtjP8oSZTJar0sC73+sPiF6g@mail.gmail.com>

On Mon, Mar 9, 2015 at 11:28 AM, Ataollah Mesgarnejad <
amesga1 at tigers.lsu.edu> wrote:

> Dear all,
>
> I'm trying to save and load distributed DMPlex in HDF5 format with DMView
> and DMLoad. The problem is even though there are no errors while loading
> the DM when I look at the DM loaded from file is not equal to the DM that
> was saved. Also, when loading the DM if DMSetType is called before the
> DMLoad, DMLoad produces errors (like line 316
> <https://bitbucket.org/petsc/petsc/src/236f5a4d227fe9d0affddb8701edb9509ad39525/src/snes/examples/tutorials/ex12.c?at=master#cl-316>
> in src/snes/examples/tutorials/ex12.c ).
>
> I'm including a small code to replicate the problem here plus an input
> mesh file.
>

Here is a simpler test, which I am going to make DMPlex ex5. The file you
sent does not make sense
to me since it has cells with only two vertices. Run it with

  -dm_view ::ascii_info_detail

  Thanks,

    Matt


> Best,
> Ata
>
> /**
>
>  * @file
>
>  *
>
>  * Created by Ataollah Mesgarnejad on 3/6/15.
>
>  *
>
>  * This is a test to read and write a distributed DM.
>
>  *
>
>  */
>
>
>
> static char help[] = "An example of the usage of PetscSF to scatter data
> back and forth between a \
>
> a serial DM and its parallel counterpart.\n";
>
>
> #include <petscdmplex.h>
>
> #include <petscsnes.h>
>
> #include <petscsf.h>
>
> #include <exodusII.h>
>
> #include <petsc-private/dmimpl.h>     /*I      "petscdm.h"     I*/
>
> #include <petscsf.h>
>
> #include <petscviewerhdf5.h>
>
>
> #undef __FUNCT__
>
> #define __FUNCT__ "main"
>
> int main(int argc,char **argv)
>
> {
>
>     PetscErrorCode      ierr;
>
>     DM                  dm, distDM,distDMold;
>
>     char                ifilename[PETSC_MAX_PATH_LEN];
>
>     PetscBool           flg;
>
>     int                 CPU_word_size = 0,IO_word_size = 0,exoid;
>
>     float               version;
>
>     PetscInt            numproc,rank;
>
>     PetscViewer         stdoutViewer,DMPlexHDF5View;
>
>     PetscReal           *VArray;
>
>     PetscInt            dim;
>
>     PetscInt            numFields = 2;
>
>     PetscInt            numComp[2] = {1,1};
>
>     PetscInt            numDof[6] = {1, 0, 0,
>
>         0, 0, 1}; /*{Vertex, Edge, Cell} */
>
>     PetscInt            bcFields[1] = {0}, numBC=0;
>
>     //PetscInt          *remoteOffsets;
>
>     char**              namelist;
>
>     PetscInt            off;
>
>     IS                  bcPoints[1] = {NULL};
>
>     IS                  *ISlist;
>
>     PetscSection        seqSection, distSection;
>
>     Vec                 distV, V;
>
>     PetscSF             pointSF;//,pointSFold;
>
>     PetscInt            pStart, pEnd, dof;
>
>     PetscBool           load_files = PETSC_FALSE, save_files = PETSC_FALSE
> ;
>
>     MPI_Comm            comm;
>
>
>
>     ierr = PetscInitialize(&argc, &argv, (char*)0, help);CHKERRQ(ierr);
>
>     ierr = PetscViewerASCIIGetStdout(PETSC_COMM_SELF,&stdoutViewer);
> CHKERRQ(ierr);
>
>     comm = PETSC_COMM_WORLD;
>
>     // PetscErrorCode DMCreateSubDM(DM dm, PetscInt numFields, PetscInt
> fields[], IS *is, DM *subdm)
>
>
>
>
>
>
>
>     ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","PetscSF Test Options",
> "none");CHKERRQ(ierr);
>
>     ierr = PetscOptionsBool("-save_files","save the distributed vector in
> HDF5 format","",PETSC_FALSE,&save_files,NULL);CHKERRQ(ierr);
>
>     ierr = PetscOptionsBool("-load_files","Load the distributed vector in
> HDF5 format","",PETSC_FALSE,&load_files,NULL);CHKERRQ(ierr);
>
>     ierr = PetscOptionsEnd();
>
>     ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof
> ifilename,&flg);CHKERRQ(ierr);
>
>
>
>     ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc);
>
>     ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
>
>     if (!rank)
>
>     {
>
>         exoid = ex_open(ifilename,EX_READ
> ,&CPU_word_size,&IO_word_size,&version);
>
>         if (exoid <= 0) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"ex_open(\"%s\",...)
> did not return a valid file ID",ifilename);
>
>     }
>
>     else
>
>     {
>
>         exoid = -1;                 /* Not used */
>
>     }
>
>     ierr = DMPlexCreateExodus(PETSC_COMM_WORLD,exoid,PETSC_FALSE,&dm);
>                                               CHKERRQ(ierr);
>
>     ierr = PetscObjectSetName((PetscObject) dm,"sequential-DM");
>
>     ierr = DMGetDimension(dm, &dim);
>                                               CHKERRQ(ierr);
>
>
>
>     if (numproc < 2 ) distDM = dm;
>
>     else ierr = DMPlexDistribute(dm, 0, &pointSF, &distDM);
>                                               CHKERRQ(ierr);
>
>     ierr = PetscObjectSetName((PetscObject) distDM,"Distributed-DM");
>
>
>     ierr = DMPlexCreateSection(dm,      dim, numFields, numComp, numDof,
>
>                                numBC, bcFields, bcPoints,PETSC_NULL,
> &seqSection);                                      CHKERRQ(ierr);
>
>     ierr = DMPlexCreateSection(distDM,  dim, numFields, numComp, numDof,
>
>                                numBC, bcFields, bcPoints,PETSC_NULL,
> &distSection);                                     CHKERRQ(ierr);
>
>
>
>     ierr = DMSetDefaultSection(dm, seqSection);
>                                               CHKERRQ(ierr);
>
>     ierr = DMSetDefaultSection(distDM, distSection);
>                                               CHKERRQ(ierr);
>
>
>
>     ierr = DMCreateGlobalVector(dm, &V);
>                                               CHKERRQ(ierr);
>
>     ierr = VecCreate(comm, &distV);
>                                               CHKERRQ(ierr);
>
>     ierr = PetscObjectSetName((PetscObject) distV,"Distributed-V");
>
>
>     ierr = VecGetArray(V, &VArray);
>                                               CHKERRQ(ierr);
>
>
>
>     // fill vertex data
>
>     ierr = DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
>                                               CHKERRQ(ierr);
>
>     for (int p = pStart; p < pEnd; p++)
>
>     {
>
>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>                                               CHKERRQ(ierr);
>
>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>                                               CHKERRQ(ierr);
>
>         for (int d = 0; d < dof; d++)
>
>         {
>
>             VArray[off + d] = p;
>
>         }
>
>     }
>
>
>
>     // fill cell data
>
>     ierr = DMPlexGetDepthStratum(dm, 1, &pStart, &pEnd);
>                                               CHKERRQ(ierr);
>
>     for (int p = pStart; p < pEnd; p++)
>
>     {
>
>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>                                               CHKERRQ(ierr);
>
>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>                                               CHKERRQ(ierr);
>
>         for (int d = 0; d < dof; d++)
>
>         {
>
>             VArray[off + d] = -p-1;
>
>         }
>
>     }
>
>
>
>     ierr = VecRestoreArray(V, &VArray); CHKERRQ(ierr);
>
>     ierr = DMCreateFieldIS(dm, &numFields, &namelist, &ISlist);
>                                               CHKERRQ(ierr);
>
>     ierr = DMPlexDistributeField(dm, pointSF, seqSection, V, distSection,
> distV);                                       CHKERRQ(ierr);
>
> #if defined(PETSC_HAVE_HDF5)
>
>     if (save_files)
>
>     {
>
>         // distribute fields
>
>
>
>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Distrubuted DM\n");
>                                               CHKERRQ(ierr);
>
>         ierr = DMView(distDM, PETSC_VIEWER_STDOUT_WORLD);
>                                               CHKERRQ(ierr);
>
>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>
>         // Save distributed data
>
>
>
>         ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject) distDM),
> "distDM.h5", FILE_MODE_WRITE, &DMPlexHDF5View);CHKERRQ(ierr);
>
>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Writing dist DM ...\n");
>
>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>
>         ierr = DMView(distDM, DMPlexHDF5View);
>                                               CHKERRQ(ierr);
>
>
>
>         PetscViewerDestroy(&DMPlexHDF5View);
>
>
>
>     }
>
>     else if (load_files)
>
>     {
>
>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Loading dist vectors ...\n");
>                                             CHKERRQ(ierr);
>
>         ierr = PetscViewerHDF5Open(PETSC_COMM_SELF,"distDM.h5",
> FILE_MODE_READ, &DMPlexHDF5View);                       CHKERRQ(ierr);
>
>         ierr = DMCreate(comm,&distDMold);
>                                               CHKERRQ(ierr);
>
>
>
>         ierr = DMLoad(distDMold,DMPlexHDF5View);
>                                               CHKERRQ(ierr);
>
>         ierr = DMSetType(distDMold, DMPLEX);
>                                               CHKERRQ(ierr);
>
>         /*ierr = DMPlexCreateSection(distDMold,  dim, numFields, numComp,
> numDof,
>
>                                    numBC, bcFields, bcPoints,PETSC_NULL,
> &distSectionold);                              CHKERRQ(ierr);
>
>         ierr = DMSetDefaultSection(distDMold,
> distSectionold);CHKERRQ(ierr);
>
>         ierr = DMGetPointSF(distDMold,&pointSFold);
>                                               CHKERRQ(ierr);*/
>
>         PetscViewerDestroy(&DMPlexHDF5View);
>
>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Read DM\n");
>                                               CHKERRQ(ierr);
>
>         ierr = DMView(distDMold, PETSC_VIEWER_STDOUT_WORLD);
>                                               CHKERRQ(ierr);
>
>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>                                               CHKERRQ(ierr);
>
>         DMPlexEqual(distDM, distDMold, &flg);
>
>         if (flg)
>
>         {
>
>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs equal\n");
>
>         }
>
>         else
>
>         {
>
>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs are not equal\n\n");
>
>         }
>
>     }
>
> #endif
>
>
>
>     ierr = VecDestroy(&V); CHKERRQ(ierr);
>
>     ierr = VecDestroy(&distV); CHKERRQ(ierr);
>
>     //ierr = VecDestroy(&seqV); CHKERRQ(ierr);
>
>
>
>     ierr = DMDestroy(&dm);CHKERRQ(ierr);
>
>     ierr = DMDestroy(&distDM);CHKERRQ(ierr);
>
>
>
>     ierr = PetscFinalize();
>
>     return 0;
>
> }
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/09da59dc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5.c
Type: text/x-csrc
Size: 582 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/09da59dc/attachment-0001.c>

From mfadams at lbl.gov  Tue Mar 10 16:50:42 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 10 Mar 2015 17:50:42 -0400
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CAJCWK9B9YO-w7+Xxf1NyZ_V9cNtuKS8TAsyTjzmPFzJXqTXECg@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
	<CADOhEh4hP=oEnJrnbdBt-cLLVDb+Czy0y0COEncUdQdkodFCJQ@mail.gmail.com>
	<CAJCWK9B9YO-w7+Xxf1NyZ_V9cNtuKS8TAsyTjzmPFzJXqTXECg@mail.gmail.com>
Message-ID: <CADOhEh5C3hWcMDrUAZdGY17rAy2vQtt8KjbwFPmYv26u_Jz9+w@mail.gmail.com>

Penalty methods are hell on solvers.  There is a good change that GAMG will
do the right thing with a finite threshold. and keep this stiff section
separate.  You might want to start wit a small number of levels and a large
coarse grid and get it working.  Then add more levels and see if it breaks.

On Sun, Mar 8, 2015 at 11:38 AM, David Knezevic <david.knezevic at akselos.com>
wrote:

> On Sun, Mar 8, 2015 at 10:31 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>>
>>> Another situation I've been meaning to ask about: If the elasticity
>>> model includes some "near rigid" regions (very high Young's modulus)
>>>
>>
>> Do you mean Poisson ratio?
>>
>
>
> No, just high Young's modulus in subregions. Engineers sometimes implement
> a rotation boundary condition about a node by including rigid (or, in
> practice, very high Young's modulus) parts in a model. For example, imagine
> including a rigid pyramid in a model, in which the tip of the pyramid is is
> clamped so that the base of the pyramid (which is connected to a surface in
> the model) can rotate about the tip.
>
> This works fine with a direct solver, so I was curious about getting a
> good iterative solver for this case too.
>
>
>
>> Nearly incompressible is hard.  bigger smoothers, like ASM, can help.
>>
>
> Agreed. But the Poisson ratio is still just 0.3 or so in this context, so
> locking and near incompressibility aren't an issue here.
>
> Thanks,
> David
>
>
>
>
>
>> On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> FYI, st?ben used classical AMG for elasticity but he has articulated his
>>> code for elasticity more than Hypre as I understand it.  Hypre can work OK
>>> for elasticity in my experience.  Its worth a try.
>>>
>>> Mark
>>>
>>> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <
>>> david.knezevic at akselos.com> wrote:
>>>
>>>> OK, got it, thanks!
>>>>
>>>> David
>>>>
>>>>
>>>> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> David Knezevic <david.knezevic at akselos.com> writes:
>>>>> > I was just wondering if its possible to achieve the same sort of
>>>>> thing with
>>>>> > other AMG solvers (e.g. BoomerAMG)? I assume that
>>>>> MatSetNearNullSpace does
>>>>> > nothing for external solvers like hypre, right?
>>>>>
>>>>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>>>>> AMG) which uses an algorithm that doesn't have a natural place for such
>>>>> information.  To my knowledge, classical AMG is not widely used for
>>>>> elasticity.  It is very robust for M-matrices.
>>>>>
>>>>
>>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/507d0c46/attachment.html>

From david.knezevic at akselos.com  Tue Mar 10 16:52:46 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Tue, 10 Mar 2015 17:52:46 -0400
Subject: [petsc-users] Near nullspace for preconditioners other than
	GAMG?
In-Reply-To: <CADOhEh5C3hWcMDrUAZdGY17rAy2vQtt8KjbwFPmYv26u_Jz9+w@mail.gmail.com>
References: <CAJCWK9AkG6O=mNNd4s4JvBkKfqgb6yndtxkn+ikzJwV+QGCQSw@mail.gmail.com>
	<87h9tzumfi.fsf@jedbrown.org>
	<CAJCWK9AugMpWXnYiWq0zRnAz89qi6HqnYyVyQs+b1yqKwS35ow@mail.gmail.com>
	<CADOhEh5yDwi1vEZmL09cSSbdk6mOYfsdTGbviJDh5faGe9KLOw@mail.gmail.com>
	<CAJCWK9CYQ0aL3oCm27n0==ZH9+SVNZDcw0RdGBcMDAf3PcQwcA@mail.gmail.com>
	<CADOhEh4hP=oEnJrnbdBt-cLLVDb+Czy0y0COEncUdQdkodFCJQ@mail.gmail.com>
	<CAJCWK9B9YO-w7+Xxf1NyZ_V9cNtuKS8TAsyTjzmPFzJXqTXECg@mail.gmail.com>
	<CADOhEh5C3hWcMDrUAZdGY17rAy2vQtt8KjbwFPmYv26u_Jz9+w@mail.gmail.com>
Message-ID: <CAJCWK9AXBChTqZ2w+typt3hKLC4O0Dv4FtNKALPjwTQkB1KbfA@mail.gmail.com>

On Tue, Mar 10, 2015 at 5:50 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Penalty methods are hell on solvers.  There is a good change that GAMG
> will do the right thing with a finite threshold. and keep this stiff
> section separate.  You might want to start wit a small number of levels and
> a large coarse grid and get it working.  Then add more levels and see if it
> breaks.
>


OK, thanks for the help, much appreciated. I'll look into this.

Best regards,
David







>
> On Sun, Mar 8, 2015 at 11:38 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
>
>> On Sun, Mar 8, 2015 at 10:31 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>>
>>>
>>>
>>>> Another situation I've been meaning to ask about: If the elasticity
>>>> model includes some "near rigid" regions (very high Young's modulus)
>>>>
>>>
>>> Do you mean Poisson ratio?
>>>
>>
>>
>> No, just high Young's modulus in subregions. Engineers sometimes
>> implement a rotation boundary condition about a node by including rigid
>> (or, in practice, very high Young's modulus) parts in a model. For example,
>> imagine including a rigid pyramid in a model, in which the tip of the
>> pyramid is is clamped so that the base of the pyramid (which is connected
>> to a surface in the model) can rotate about the tip.
>>
>> This works fine with a direct solver, so I was curious about getting a
>> good iterative solver for this case too.
>>
>>
>>
>>> Nearly incompressible is hard.  bigger smoothers, like ASM, can help.
>>>
>>
>> Agreed. But the Poisson ratio is still just 0.3 or so in this context, so
>> locking and near incompressibility aren't an issue here.
>>
>> Thanks,
>> David
>>
>>
>>
>>
>>
>>> On Sat, Mar 7, 2015 at 5:57 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> FYI, st?ben used classical AMG for elasticity but he has articulated
>>>> his code for elasticity more than Hypre as I understand it.  Hypre can work
>>>> OK for elasticity in my experience.  Its worth a try.
>>>>
>>>> Mark
>>>>
>>>> On Thu, Mar 5, 2015 at 5:27 PM, David Knezevic <
>>>> david.knezevic at akselos.com> wrote:
>>>>
>>>>> OK, got it, thanks!
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Thu, Mar 5, 2015 at 5:08 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>>> David Knezevic <david.knezevic at akselos.com> writes:
>>>>>> > I was just wondering if its possible to achieve the same sort of
>>>>>> thing with
>>>>>> > other AMG solvers (e.g. BoomerAMG)? I assume that
>>>>>> MatSetNearNullSpace does
>>>>>> > nothing for external solvers like hypre, right?
>>>>>>
>>>>>> It is used by ML (smoothed aggregation), but not BoomerAMG (classical
>>>>>> AMG) which uses an algorithm that doesn't have a natural place for
>>>>>> such
>>>>>> information.  To my knowledge, classical AMG is not widely used for
>>>>>> elasticity.  It is very robust for M-matrices.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/41827f6e/attachment.html>

From amesga1 at tigers.lsu.edu  Tue Mar 10 17:02:53 2015
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Tue, 10 Mar 2015 17:02:53 -0500
Subject: [petsc-users] DMView and DMLoad
In-Reply-To: <CAMYG4G=iwUeYA_FfJEy0rsyzG9LtjP8oSZTJar0sC73+sPiF6g@mail.gmail.com>
References: <CAC+VmGeK8+4NOXKj+9CerkkJxVNCorz7D_fM-fPjmSuJv6p63g@mail.gmail.com>
	<CAMYG4G=iwUeYA_FfJEy0rsyzG9LtjP8oSZTJar0sC73+sPiF6g@mail.gmail.com>
Message-ID: <CAC+VmGd46Ac-Ns9+tMwdMvVaZzHhsDE0Pw5h3=Sq=ZUgXo3rEA@mail.gmail.com>

Thanks Matt,

I changed your file to reproduce the error. If you run it on more than one
processor you will get something similar to:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------

[0]PETSC ERROR: Invalid argument

[0]PETSC ERROR: Number of coordinates loaded 9 does not match number of
vertices 13

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.

[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2186-g052b31f  GIT
Date: 2015-03-04 11:15:59 -0600

[0]PETSC ERROR: /work/01624/amesga/Code/dmcomplex/ex5 on a
sandybridge-cxx-dbg named c560-303.stampede.tacc.utexas.edu by amesga Tue
Mar 10 16:57:01 2015

[0]PETSC ERROR: Configure options --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS=
--download-chaco=1 --download-exodusii=1 --download-hdf5=1
--download-metis=1 --download-netcdf=1 --download-parmetis=1 --with-blacs=1
--with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013_sp1.1.106/mkl/lib/intel64
--with-clanguage=C++ --with-debugging=yes --with-hdf5=1 --with-metis=1
--with-valgrind=1 --download-triangle=1
--with-valgrind-dir=/opt/apps/valgrind/3.8.1/ --with-mpi-compilers=1
--with-mpi-dir=/opt/apps/intel13/impi/4.1.3.049/intel64
--with-scalar-type=real --with-shared-libraries=1
--with-vendor-compilers=intel --with-x11=1 -with-pic
PETSC_ARCH=sandybridge-cxx-dbg

[0]PETSC ERROR: #1 DMPlexLoad_HDF5() line 741 in
/work/01624/amesga/Software/petsc/src/dm/impls/plex/plexhdf5.c

[0]PETSC ERROR: #2 DMLoad_Plex() line 605 in
/work/01624/amesga/Software/petsc/src/dm/impls/plex/plex.c

[0]PETSC ERROR: #3 DMLoad() line 2939 in
/work/01624/amesga/Software/petsc/src/dm/interface/dm.c

[0]PETSC ERROR: #4 main() line 31 in /work/01624/amesga/Code/dmcomplex/ex5.c

[0]PETSC ERROR: No PETSc Option Table entries

[0]PETSC ERROR: ----------------End of Error Message -------


It seems like HDF5 version of DMView and DMLoad work fine for a serial
DMPlex but DMLoad breaks in parallel.


Many thanks,

Ata

On Tue, Mar 10, 2015 at 3:15 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Mar 9, 2015 at 11:28 AM, Ataollah Mesgarnejad <
> amesga1 at tigers.lsu.edu> wrote:
>
>> Dear all,
>>
>> I'm trying to save and load distributed DMPlex in HDF5 format with DMView
>> and DMLoad. The problem is even though there are no errors while loading
>> the DM when I look at the DM loaded from file is not equal to the DM that
>> was saved. Also, when loading the DM if DMSetType is called before the
>> DMLoad, DMLoad produces errors (like line 316
>> <https://bitbucket.org/petsc/petsc/src/236f5a4d227fe9d0affddb8701edb9509ad39525/src/snes/examples/tutorials/ex12.c?at=master#cl-316>
>> in src/snes/examples/tutorials/ex12.c ).
>>
>> I'm including a small code to replicate the problem here plus an input
>> mesh file.
>>
>
> Here is a simpler test, which I am going to make DMPlex ex5. The file you
> sent does not make sense
> to me since it has cells with only two vertices. Run it with
>
>   -dm_view ::ascii_info_detail
>
>   Thanks,
>
>     Matt
>
>
>> Best,
>> Ata
>>
>> /**
>>
>>  * @file
>>
>>  *
>>
>>  * Created by Ataollah Mesgarnejad on 3/6/15.
>>
>>  *
>>
>>  * This is a test to read and write a distributed DM.
>>
>>  *
>>
>>  */
>>
>>
>>
>> static char help[] = "An example of the usage of PetscSF to scatter data
>> back and forth between a \
>>
>> a serial DM and its parallel counterpart.\n";
>>
>>
>> #include <petscdmplex.h>
>>
>> #include <petscsnes.h>
>>
>> #include <petscsf.h>
>>
>> #include <exodusII.h>
>>
>> #include <petsc-private/dmimpl.h>     /*I      "petscdm.h"     I*/
>>
>> #include <petscsf.h>
>>
>> #include <petscviewerhdf5.h>
>>
>>
>> #undef __FUNCT__
>>
>> #define __FUNCT__ "main"
>>
>> int main(int argc,char **argv)
>>
>> {
>>
>>     PetscErrorCode      ierr;
>>
>>     DM                  dm, distDM,distDMold;
>>
>>     char                ifilename[PETSC_MAX_PATH_LEN];
>>
>>     PetscBool           flg;
>>
>>     int                 CPU_word_size = 0,IO_word_size = 0,exoid;
>>
>>     float               version;
>>
>>     PetscInt            numproc,rank;
>>
>>     PetscViewer         stdoutViewer,DMPlexHDF5View;
>>
>>     PetscReal           *VArray;
>>
>>     PetscInt            dim;
>>
>>     PetscInt            numFields = 2;
>>
>>     PetscInt            numComp[2] = {1,1};
>>
>>     PetscInt            numDof[6] = {1, 0, 0,
>>
>>         0, 0, 1}; /*{Vertex, Edge, Cell} */
>>
>>     PetscInt            bcFields[1] = {0}, numBC=0;
>>
>>     //PetscInt          *remoteOffsets;
>>
>>     char**              namelist;
>>
>>     PetscInt            off;
>>
>>     IS                  bcPoints[1] = {NULL};
>>
>>     IS                  *ISlist;
>>
>>     PetscSection        seqSection, distSection;
>>
>>     Vec                 distV, V;
>>
>>     PetscSF             pointSF;//,pointSFold;
>>
>>     PetscInt            pStart, pEnd, dof;
>>
>>     PetscBool           load_files = PETSC_FALSE, save_files =
>> PETSC_FALSE;
>>
>>     MPI_Comm            comm;
>>
>>
>>
>>     ierr = PetscInitialize(&argc, &argv, (char*)0, help);CHKERRQ(ierr);
>>
>>     ierr = PetscViewerASCIIGetStdout(PETSC_COMM_SELF,&stdoutViewer);
>> CHKERRQ(ierr);
>>
>>     comm = PETSC_COMM_WORLD;
>>
>>     // PetscErrorCode DMCreateSubDM(DM dm, PetscInt numFields, PetscInt
>> fields[], IS *is, DM *subdm)
>>
>>
>>
>>
>>
>>
>>
>>     ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","PetscSF Test Options",
>> "none");CHKERRQ(ierr);
>>
>>     ierr = PetscOptionsBool("-save_files","save the distributed vector
>> in HDF5 format","",PETSC_FALSE,&save_files,NULL);CHKERRQ(ierr);
>>
>>     ierr = PetscOptionsBool("-load_files","Load the distributed vector
>> in HDF5 format","",PETSC_FALSE,&load_files,NULL);CHKERRQ(ierr);
>>
>>     ierr = PetscOptionsEnd();
>>
>>     ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof
>> ifilename,&flg);CHKERRQ(ierr);
>>
>>
>>
>>     ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc);
>>
>>     ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
>>
>>     if (!rank)
>>
>>     {
>>
>>         exoid = ex_open(ifilename,EX_READ
>> ,&CPU_word_size,&IO_word_size,&version);
>>
>>         if (exoid <= 0) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"ex_open(\"%s\",...)
>> did not return a valid file ID",ifilename);
>>
>>     }
>>
>>     else
>>
>>     {
>>
>>         exoid = -1;                 /* Not used */
>>
>>     }
>>
>>     ierr = DMPlexCreateExodus(PETSC_COMM_WORLD,exoid,PETSC_FALSE,&dm);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = PetscObjectSetName((PetscObject) dm,"sequential-DM");
>>
>>     ierr = DMGetDimension(dm, &dim);
>>                                                 CHKERRQ(ierr);
>>
>>
>>
>>     if (numproc < 2 ) distDM = dm;
>>
>>     else ierr = DMPlexDistribute(dm, 0, &pointSF, &distDM);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = PetscObjectSetName((PetscObject) distDM,"Distributed-DM");
>>
>>
>>     ierr = DMPlexCreateSection(dm,      dim, numFields, numComp, numDof,
>>
>>                                numBC, bcFields, bcPoints,PETSC_NULL,
>> &seqSection);                                      CHKERRQ(ierr);
>>
>>     ierr = DMPlexCreateSection(distDM,  dim, numFields, numComp, numDof,
>>
>>                                numBC, bcFields, bcPoints,PETSC_NULL,
>> &distSection);                                     CHKERRQ(ierr);
>>
>>
>>
>>     ierr = DMSetDefaultSection(dm, seqSection);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = DMSetDefaultSection(distDM, distSection);
>>                                                 CHKERRQ(ierr);
>>
>>
>>
>>     ierr = DMCreateGlobalVector(dm, &V);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = VecCreate(comm, &distV);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = PetscObjectSetName((PetscObject) distV,"Distributed-V");
>>
>>
>>     ierr = VecGetArray(V, &VArray);
>>                                                 CHKERRQ(ierr);
>>
>>
>>
>>     // fill vertex data
>>
>>     ierr = DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
>>                                                 CHKERRQ(ierr);
>>
>>     for (int p = pStart; p < pEnd; p++)
>>
>>     {
>>
>>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>>                                                 CHKERRQ(ierr);
>>
>>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>>                                                 CHKERRQ(ierr);
>>
>>         for (int d = 0; d < dof; d++)
>>
>>         {
>>
>>             VArray[off + d] = p;
>>
>>         }
>>
>>     }
>>
>>
>>
>>     // fill cell data
>>
>>     ierr = DMPlexGetDepthStratum(dm, 1, &pStart, &pEnd);
>>                                                 CHKERRQ(ierr);
>>
>>     for (int p = pStart; p < pEnd; p++)
>>
>>     {
>>
>>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>>                                                 CHKERRQ(ierr);
>>
>>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>>                                                 CHKERRQ(ierr);
>>
>>         for (int d = 0; d < dof; d++)
>>
>>         {
>>
>>             VArray[off + d] = -p-1;
>>
>>         }
>>
>>     }
>>
>>
>>
>>     ierr = VecRestoreArray(V, &VArray); CHKERRQ(ierr);
>>
>>     ierr = DMCreateFieldIS(dm, &numFields, &namelist, &ISlist);
>>                                                 CHKERRQ(ierr);
>>
>>     ierr = DMPlexDistributeField(dm, pointSF, seqSection, V,
>> distSection, distV);                                       CHKERRQ(ierr);
>>
>> #if defined(PETSC_HAVE_HDF5)
>>
>>     if (save_files)
>>
>>     {
>>
>>         // distribute fields
>>
>>
>>
>>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Distrubuted DM\n");
>>                                                 CHKERRQ(ierr);
>>
>>         ierr = DMView(distDM, PETSC_VIEWER_STDOUT_WORLD);
>>                                                 CHKERRQ(ierr);
>>
>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>
>>         // Save distributed data
>>
>>
>>
>>         ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject) distDM),
>> "distDM.h5", FILE_MODE_WRITE, &DMPlexHDF5View);CHKERRQ(ierr);
>>
>>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Writing dist DM ...\n");
>>
>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>
>>         ierr = DMView(distDM, DMPlexHDF5View);
>>                                                 CHKERRQ(ierr);
>>
>>
>>
>>         PetscViewerDestroy(&DMPlexHDF5View);
>>
>>
>>
>>     }
>>
>>     else if (load_files)
>>
>>     {
>>
>>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Loading dist vectors ...\n");
>>                                             CHKERRQ(ierr);
>>
>>         ierr = PetscViewerHDF5Open(PETSC_COMM_SELF,"distDM.h5",
>> FILE_MODE_READ, &DMPlexHDF5View);                       CHKERRQ(ierr);
>>
>>         ierr = DMCreate(comm,&distDMold);
>>                                                 CHKERRQ(ierr);
>>
>>
>>
>>         ierr = DMLoad(distDMold,DMPlexHDF5View);
>>                                                 CHKERRQ(ierr);
>>
>>         ierr = DMSetType(distDMold, DMPLEX);
>>                                                 CHKERRQ(ierr);
>>
>>         /*ierr = DMPlexCreateSection(distDMold,  dim, numFields,
>> numComp, numDof,
>>
>>                                    numBC, bcFields, bcPoints,PETSC_NULL,
>> &distSectionold);                              CHKERRQ(ierr);
>>
>>         ierr = DMSetDefaultSection(distDMold,
>> distSectionold);CHKERRQ(ierr);
>>
>>         ierr = DMGetPointSF(distDMold,&pointSFold);
>>                                               CHKERRQ(ierr);*/
>>
>>         PetscViewerDestroy(&DMPlexHDF5View);
>>
>>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Read DM\n");
>>                                                 CHKERRQ(ierr);
>>
>>         ierr = DMView(distDMold, PETSC_VIEWER_STDOUT_WORLD);
>>                                                 CHKERRQ(ierr);
>>
>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>                                                 CHKERRQ(ierr);
>>
>>         DMPlexEqual(distDM, distDMold, &flg);
>>
>>         if (flg)
>>
>>         {
>>
>>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs equal\n");
>>
>>         }
>>
>>         else
>>
>>         {
>>
>>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs are not equal\n\n");
>>
>>         }
>>
>>     }
>>
>> #endif
>>
>>
>>
>>     ierr = VecDestroy(&V); CHKERRQ(ierr);
>>
>>     ierr = VecDestroy(&distV); CHKERRQ(ierr);
>>
>>     //ierr = VecDestroy(&seqV); CHKERRQ(ierr);
>>
>>
>>
>>     ierr = DMDestroy(&dm);CHKERRQ(ierr);
>>
>>     ierr = DMDestroy(&distDM);CHKERRQ(ierr);
>>
>>
>>
>>     ierr = PetscFinalize();
>>
>>     return 0;
>>
>> }
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/716cf4db/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5.c
Type: text/x-csrc
Size: 2630 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/716cf4db/attachment-0001.c>

From knepley at gmail.com  Tue Mar 10 17:05:37 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 10 Mar 2015 17:05:37 -0500
Subject: [petsc-users] DMView and DMLoad
In-Reply-To: <CAC+VmGd46Ac-Ns9+tMwdMvVaZzHhsDE0Pw5h3=Sq=ZUgXo3rEA@mail.gmail.com>
References: <CAC+VmGeK8+4NOXKj+9CerkkJxVNCorz7D_fM-fPjmSuJv6p63g@mail.gmail.com>
	<CAMYG4G=iwUeYA_FfJEy0rsyzG9LtjP8oSZTJar0sC73+sPiF6g@mail.gmail.com>
	<CAC+VmGd46Ac-Ns9+tMwdMvVaZzHhsDE0Pw5h3=Sq=ZUgXo3rEA@mail.gmail.com>
Message-ID: <CAMYG4G=kz2a1kajDH+Psy1y31JHbhRRxBry0bCx3qV5gVcXQqQ@mail.gmail.com>

On Tue, Mar 10, 2015 at 5:02 PM, Ataollah Mesgarnejad <
amesga1 at tigers.lsu.edu> wrote:

> Thanks Matt,
>
> I changed your file to reproduce the error. If you run it on more than one
> processor you will get something similar to:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Invalid argument
>
> [0]PETSC ERROR: Number of coordinates loaded 9 does not match number of
> vertices 13
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2186-g052b31f  GIT
> Date: 2015-03-04 11:15:59 -0600
>
> [0]PETSC ERROR: /work/01624/amesga/Code/dmcomplex/ex5 on a
> sandybridge-cxx-dbg named c560-303.stampede.tacc.utexas.edu by amesga Tue
> Mar 10 16:57:01 2015
>
> [0]PETSC ERROR: Configure options --COPTFLAGS= --CXXOPTFLAGS= --FOPTFLAGS=
> --download-chaco=1 --download-exodusii=1 --download-hdf5=1
> --download-metis=1 --download-netcdf=1 --download-parmetis=1 --with-blacs=1
> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013_sp1.1.106/mkl/lib/intel64
> --with-clanguage=C++ --with-debugging=yes --with-hdf5=1 --with-metis=1
> --with-valgrind=1 --download-triangle=1
> --with-valgrind-dir=/opt/apps/valgrind/3.8.1/ --with-mpi-compilers=1
> --with-mpi-dir=/opt/apps/intel13/impi/4.1.3.049/intel64
> --with-scalar-type=real --with-shared-libraries=1
> --with-vendor-compilers=intel --with-x11=1 -with-pic
> PETSC_ARCH=sandybridge-cxx-dbg
>
> [0]PETSC ERROR: #1 DMPlexLoad_HDF5() line 741 in
> /work/01624/amesga/Software/petsc/src/dm/impls/plex/plexhdf5.c
>
> [0]PETSC ERROR: #2 DMLoad_Plex() line 605 in
> /work/01624/amesga/Software/petsc/src/dm/impls/plex/plex.c
>
> [0]PETSC ERROR: #3 DMLoad() line 2939 in
> /work/01624/amesga/Software/petsc/src/dm/interface/dm.c
>
> [0]PETSC ERROR: #4 main() line 31 in
> /work/01624/amesga/Code/dmcomplex/ex5.c
>
> [0]PETSC ERROR: No PETSc Option Table entries
>
> [0]PETSC ERROR: ----------------End of Error Message -------
>
>
> It seems like HDF5 version of DMView and DMLoad work fine for a serial
> DMPlex but DMLoad breaks in parallel.
>

Cool, will get it fixed. I have only been loading Vecs since DMLoad()
cannot recreate hierarchies, and thus is not
compatible with the FAS tests I am mostly running.

  Thanks,

    Matt


> Many thanks,
>
> Ata
>
> On Tue, Mar 10, 2015 at 3:15 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Mon, Mar 9, 2015 at 11:28 AM, Ataollah Mesgarnejad <
>> amesga1 at tigers.lsu.edu> wrote:
>>
>>> Dear all,
>>>
>>> I'm trying to save and load distributed DMPlex in HDF5 format with
>>> DMView and DMLoad. The problem is even though there are no errors while
>>> loading the DM when I look at the DM loaded from file is not equal to the
>>> DM that was saved. Also, when loading the DM if DMSetType is called before
>>> the DMLoad, DMLoad produces errors (like line 316
>>> <https://bitbucket.org/petsc/petsc/src/236f5a4d227fe9d0affddb8701edb9509ad39525/src/snes/examples/tutorials/ex12.c?at=master#cl-316>
>>> in src/snes/examples/tutorials/ex12.c ).
>>>
>>> I'm including a small code to replicate the problem here plus an input
>>> mesh file.
>>>
>>
>> Here is a simpler test, which I am going to make DMPlex ex5. The file you
>> sent does not make sense
>> to me since it has cells with only two vertices. Run it with
>>
>>   -dm_view ::ascii_info_detail
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Best,
>>> Ata
>>>
>>> /**
>>>
>>>  * @file
>>>
>>>  *
>>>
>>>  * Created by Ataollah Mesgarnejad on 3/6/15.
>>>
>>>  *
>>>
>>>  * This is a test to read and write a distributed DM.
>>>
>>>  *
>>>
>>>  */
>>>
>>>
>>>
>>> static char help[] = "An example of the usage of PetscSF to scatter
>>> data back and forth between a \
>>>
>>> a serial DM and its parallel counterpart.\n";
>>>
>>>
>>> #include <petscdmplex.h>
>>>
>>> #include <petscsnes.h>
>>>
>>> #include <petscsf.h>
>>>
>>> #include <exodusII.h>
>>>
>>> #include <petsc-private/dmimpl.h>     /*I      "petscdm.h"     I*/
>>>
>>> #include <petscsf.h>
>>>
>>> #include <petscviewerhdf5.h>
>>>
>>>
>>> #undef __FUNCT__
>>>
>>> #define __FUNCT__ "main"
>>>
>>> int main(int argc,char **argv)
>>>
>>> {
>>>
>>>     PetscErrorCode      ierr;
>>>
>>>     DM                  dm, distDM,distDMold;
>>>
>>>     char                ifilename[PETSC_MAX_PATH_LEN];
>>>
>>>     PetscBool           flg;
>>>
>>>     int                 CPU_word_size = 0,IO_word_size = 0,exoid;
>>>
>>>     float               version;
>>>
>>>     PetscInt            numproc,rank;
>>>
>>>     PetscViewer         stdoutViewer,DMPlexHDF5View;
>>>
>>>     PetscReal           *VArray;
>>>
>>>     PetscInt            dim;
>>>
>>>     PetscInt            numFields = 2;
>>>
>>>     PetscInt            numComp[2] = {1,1};
>>>
>>>     PetscInt            numDof[6] = {1, 0, 0,
>>>
>>>         0, 0, 1}; /*{Vertex, Edge, Cell} */
>>>
>>>     PetscInt            bcFields[1] = {0}, numBC=0;
>>>
>>>     //PetscInt          *remoteOffsets;
>>>
>>>     char**              namelist;
>>>
>>>     PetscInt            off;
>>>
>>>     IS                  bcPoints[1] = {NULL};
>>>
>>>     IS                  *ISlist;
>>>
>>>     PetscSection        seqSection, distSection;
>>>
>>>     Vec                 distV, V;
>>>
>>>     PetscSF             pointSF;//,pointSFold;
>>>
>>>     PetscInt            pStart, pEnd, dof;
>>>
>>>     PetscBool           load_files = PETSC_FALSE, save_files =
>>> PETSC_FALSE;
>>>
>>>     MPI_Comm            comm;
>>>
>>>
>>>
>>>     ierr = PetscInitialize(&argc, &argv, (char*)0, help);CHKERRQ(ierr);
>>>
>>>     ierr = PetscViewerASCIIGetStdout(PETSC_COMM_SELF,&stdoutViewer);
>>> CHKERRQ(ierr);
>>>
>>>     comm = PETSC_COMM_WORLD;
>>>
>>>     // PetscErrorCode DMCreateSubDM(DM dm, PetscInt numFields, PetscInt
>>> fields[], IS *is, DM *subdm)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>     ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","PetscSF Test Options",
>>> "none");CHKERRQ(ierr);
>>>
>>>     ierr = PetscOptionsBool("-save_files","save the distributed vector
>>> in HDF5 format","",PETSC_FALSE,&save_files,NULL);CHKERRQ(ierr);
>>>
>>>     ierr = PetscOptionsBool("-load_files","Load the distributed vector
>>> in HDF5 format","",PETSC_FALSE,&load_files,NULL);CHKERRQ(ierr);
>>>
>>>     ierr = PetscOptionsEnd();
>>>
>>>     ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof
>>> ifilename,&flg);CHKERRQ(ierr);
>>>
>>>
>>>
>>>     ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc);
>>>
>>>     ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
>>>
>>>     if (!rank)
>>>
>>>     {
>>>
>>>         exoid = ex_open(ifilename,EX_READ
>>> ,&CPU_word_size,&IO_word_size,&version);
>>>
>>>         if (exoid <= 0) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"ex_open(\"%s\",...)
>>> did not return a valid file ID",ifilename);
>>>
>>>     }
>>>
>>>     else
>>>
>>>     {
>>>
>>>         exoid = -1;                 /* Not used */
>>>
>>>     }
>>>
>>>     ierr = DMPlexCreateExodus(PETSC_COMM_WORLD,exoid,PETSC_FALSE,&dm);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = PetscObjectSetName((PetscObject) dm,"sequential-DM");
>>>
>>>     ierr = DMGetDimension(dm, &dim);
>>>                                                 CHKERRQ(ierr);
>>>
>>>
>>>
>>>     if (numproc < 2 ) distDM = dm;
>>>
>>>     else ierr = DMPlexDistribute(dm, 0, &pointSF, &distDM);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = PetscObjectSetName((PetscObject) distDM,"Distributed-DM");
>>>
>>>
>>>     ierr = DMPlexCreateSection(dm,      dim, numFields, numComp, numDof,
>>>
>>>                                numBC, bcFields, bcPoints,PETSC_NULL,
>>> &seqSection);                                      CHKERRQ(ierr);
>>>
>>>     ierr = DMPlexCreateSection(distDM,  dim, numFields, numComp, numDof,
>>>
>>>                                numBC, bcFields, bcPoints,PETSC_NULL,
>>> &distSection);                                     CHKERRQ(ierr);
>>>
>>>
>>>
>>>     ierr = DMSetDefaultSection(dm, seqSection);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = DMSetDefaultSection(distDM, distSection);
>>>                                                 CHKERRQ(ierr);
>>>
>>>
>>>
>>>     ierr = DMCreateGlobalVector(dm, &V);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = VecCreate(comm, &distV);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = PetscObjectSetName((PetscObject) distV,"Distributed-V");
>>>
>>>
>>>     ierr = VecGetArray(V, &VArray);
>>>                                                 CHKERRQ(ierr);
>>>
>>>
>>>
>>>     // fill vertex data
>>>
>>>     ierr = DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     for (int p = pStart; p < pEnd; p++)
>>>
>>>     {
>>>
>>>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         for (int d = 0; d < dof; d++)
>>>
>>>         {
>>>
>>>             VArray[off + d] = p;
>>>
>>>         }
>>>
>>>     }
>>>
>>>
>>>
>>>     // fill cell data
>>>
>>>     ierr = DMPlexGetDepthStratum(dm, 1, &pStart, &pEnd);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     for (int p = pStart; p < pEnd; p++)
>>>
>>>     {
>>>
>>>         ierr = PetscSectionGetDof(seqSection, p, &dof);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         ierr = PetscSectionGetOffset(seqSection, p, &off);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         for (int d = 0; d < dof; d++)
>>>
>>>         {
>>>
>>>             VArray[off + d] = -p-1;
>>>
>>>         }
>>>
>>>     }
>>>
>>>
>>>
>>>     ierr = VecRestoreArray(V, &VArray); CHKERRQ(ierr);
>>>
>>>     ierr = DMCreateFieldIS(dm, &numFields, &namelist, &ISlist);
>>>                                                 CHKERRQ(ierr);
>>>
>>>     ierr = DMPlexDistributeField(dm, pointSF, seqSection, V,
>>> distSection, distV);                                       CHKERRQ
>>> (ierr);
>>>
>>> #if defined(PETSC_HAVE_HDF5)
>>>
>>>     if (save_files)
>>>
>>>     {
>>>
>>>         // distribute fields
>>>
>>>
>>>
>>>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Distrubuted DM\n");
>>>                                                 CHKERRQ(ierr);
>>>
>>>         ierr = DMView(distDM, PETSC_VIEWER_STDOUT_WORLD);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>>
>>>         // Save distributed data
>>>
>>>
>>>
>>>         ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject)
>>> distDM),"distDM.h5", FILE_MODE_WRITE, &DMPlexHDF5View);CHKERRQ(ierr);
>>>
>>>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Writing dist DM ...\n");
>>>
>>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>>
>>>         ierr = DMView(distDM, DMPlexHDF5View);
>>>                                                 CHKERRQ(ierr);
>>>
>>>
>>>
>>>         PetscViewerDestroy(&DMPlexHDF5View);
>>>
>>>
>>>
>>>     }
>>>
>>>     else if (load_files)
>>>
>>>     {
>>>
>>>         ierr = PetscPrintf(PETSC_COMM_WORLD,"Loading dist vectors ...\n");
>>>                                             CHKERRQ(ierr);
>>>
>>>         ierr = PetscViewerHDF5Open(PETSC_COMM_SELF,"distDM.h5",
>>> FILE_MODE_READ, &DMPlexHDF5View);                       CHKERRQ(ierr);
>>>
>>>         ierr = DMCreate(comm,&distDMold);
>>>                                                 CHKERRQ(ierr);
>>>
>>>
>>>
>>>         ierr = DMLoad(distDMold,DMPlexHDF5View);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         ierr = DMSetType(distDMold, DMPLEX);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         /*ierr = DMPlexCreateSection(distDMold,  dim, numFields,
>>> numComp, numDof,
>>>
>>>                                    numBC, bcFields, bcPoints,PETSC_NULL,
>>> &distSectionold);                              CHKERRQ(ierr);
>>>
>>>         ierr = DMSetDefaultSection(distDMold,
>>> distSectionold);CHKERRQ(ierr);
>>>
>>>         ierr = DMGetPointSF(distDMold,&pointSFold);
>>>                                                 CHKERRQ(ierr);*/
>>>
>>>         PetscViewerDestroy(&DMPlexHDF5View);
>>>
>>>         ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Read DM\n");
>>>                                                 CHKERRQ(ierr);
>>>
>>>         ierr = DMView(distDMold, PETSC_VIEWER_STDOUT_WORLD);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT);
>>>                                                 CHKERRQ(ierr);
>>>
>>>         DMPlexEqual(distDM, distDMold, &flg);
>>>
>>>         if (flg)
>>>
>>>         {
>>>
>>>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs equal\n");
>>>
>>>         }
>>>
>>>         else
>>>
>>>         {
>>>
>>>             PetscPrintf(PETSC_COMM_WORLD,"\n DMs are not equal\n\n");
>>>
>>>         }
>>>
>>>     }
>>>
>>> #endif
>>>
>>>
>>>
>>>     ierr = VecDestroy(&V); CHKERRQ(ierr);
>>>
>>>     ierr = VecDestroy(&distV); CHKERRQ(ierr);
>>>
>>>     //ierr = VecDestroy(&seqV); CHKERRQ(ierr);
>>>
>>>
>>>
>>>     ierr = DMDestroy(&dm);CHKERRQ(ierr);
>>>
>>>     ierr = DMDestroy(&distDM);CHKERRQ(ierr);
>>>
>>>
>>>
>>>     ierr = PetscFinalize();
>>>
>>>     return 0;
>>>
>>> }
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/96c33c6f/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Mar 10 18:20:04 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 10 Mar 2015 18:20:04 -0500
Subject: [petsc-users] GMRES stability
In-Reply-To: <CAOYX9gz7hcZ3C+js8YYTmzTUwqtxdz9PRkGGOz0UjBtY0LCKEg@mail.gmail.com>
References: <CAOYX9gzxrd7-tHWLS6RZZK2AaO_O7tVbDw1s1b6vF=YewwCSuQ@mail.gmail.com>
	<0A826E90-DD1A-450B-BDE2-D5A4867B682B@mcs.anl.gov>
	<CAOYX9gx5e=TigrB3A0hE8G-qXJX9h83mRB+GSYwD3YE9oVATtg@mail.gmail.com>
	<239AD9F8-8DBE-42D6-B185-7BF246FC4900@mcs.anl.gov>
	<CAOYX9gzSfgswMAPnxPMNHab8As9MdkOj3g-9ZJGqPc6Rr8M4pQ@mail.gmail.com>
	<041B1984-4BFB-4A6C-92F1-8CC8DBBF0B07@mcs.anl.gov>
	<CAOYX9gyG6Pu9S3JTVTtYxTWn8NcS6K5S1pMrvXdYer0-nV8aPw@mail.gmail.com>
	<B5979305-38D7-4399-973D-45EF95296C36@mcs.anl.gov>
	<CAOYX9gz7hcZ3C+js8YYTmzTUwqtxdz9PRkGGOz0UjBtY0LCKEg@mail.gmail.com>
Message-ID: <930FB097-E73F-45D9-8B8C-21544BE4DB95@mcs.anl.gov>


  Is the matrix positive definite or semi-positive definite ? Then you can use CG instead of GMRES.

  What about the convergence of SOR plus GMRES? Was that not good? 

  Is this a "Stoke's" like problem?

  What makes up the block size of 5? That is what physically do those five variables represent?


  Barry

> On Mar 10, 2015, at 6:12 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> 
> You are right. So which PC do you suggest with GMRES for sequential and parallel executions? Seems like Jacobi PC converges very slowly. The matrix is symmetric and sparse and its block size is 5. Is there a better solver than GMRES for my case?
> 
> 
> On Mon, Mar 9, 2015 at 5:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 9, 2015, at 5:23 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> >
> > I use proactive point GS which updates the RHS of neighbours instead of updating the variable itself in subsequent iterations. I use natural ordering. I am sure that the matrices are the same for both solvers.
> 
>    Ok, so you are doing some very strange custom GS in your code that I have never heard of and yet you are surprised that the PETSc GS doesn't converge exactly the same? Unless that exact same "proactive point GS" was implemented in PETSc of course you are going to get different convergence behavior.
> 
>   Barry
> 
> >
> > On Sun, Mar 1, 2015 at 4:48 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Ok, we need to understand what is algorithmically different between what your code is doing and PETSc is doing.
> >
> >   From below PETSc is running a symmetric point block SOR with a block size of 5. Are you using a point or point block GS? Are you using any particular ordering of the unknowns in your GS, or just natural ordering? Are you sure the matrices are the same between the two codes? Print them out for a tiny grid and compare.
> >
> >   Barry
> >
> > > On Mar 1, 2015, at 4:33 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> > >
> > > I don't run in parallel. I tried -ksp_richardson -pc_type but still
> > > cannot get over CFL=5 (with my own GS code CFL=40 was converging).
> > > Also, lowering damping factor does not help. Default number of
> > > iterations (10000) does not help as well. -ksp_view outputs:
> > >
> > > KSP Object: 1 MPI processes
> > >  type: richardson
> > >    Richardson: damping factor=0.5
> > >  maximum iterations=10, initial guess is zero
> > >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> > >  left preconditioning
> > >  using PRECONDITIONED norm type for convergence test
> > > PC Object: 1 MPI processes
> > >  type: sor
> > >    SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
> > >  linear system matrix = precond matrix:
> > >  Mat Object:   1 MPI processes
> > >    type: seqbaij
> > >    rows=13250, cols=13250, bs=5
> > >    total: nonzeros=261150, allocated nonzeros=1.325e+06
> > >    total number of mallocs used during MatSetValues calls =0
> > >        block size is 5
> > >
> > > On Fri, Feb 27, 2015 at 4:30 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >>
> > >>  Ok, please provide the rest of the information I asked for.
> > >>
> > >>  Barry
> > >>
> > >>> On Feb 27, 2015, at 2:33 AM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> > >>>
> > >>> No. It does not converge at all or iow it diverges.
> > >>>
> > >>> On Thu, Feb 26, 2015 at 9:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >>>>
> > >>>> By stability I assume you mean the the GMRES does not converge (or converges too slowly)?
> > >>>>
> > >>>> The way to improve GMRES convergence is with a preconditioner better suited to your problem. By default PETSc uses GMRES with a block Jacobi preconditioner with one block per process and ILU(0) on each block. For some problems this is fine, but for many problems it will give bad convergence.
> > >>>>
> > >>>>  What do you get for -ksp_view  (are you using the default?) Are you running yet in parallel?
> > >>>>
> > >>>> As a test on one process you can use GS in PETSc as the preconditioner and make sure you get similar convergence to your code. For example -ksp_richardson -pc_type sor on one processor will give you a GS solver.
> > >>>>
> > >>>>  Once we know a bit more about your problem we can suggest better preconditioners.
> > >>>>
> > >>>> Barry
> > >>>>
> > >>>>
> > >>>>> On Feb 26, 2015, at 10:25 PM, Orxan Shibliyev <orxan.shibli at gmail.com> wrote:
> > >>>>>
> > >>>>> Hi
> > >>>>>
> > >>>>> I tried to solve Ax=b with my own Gauss-Seidel code and Petsc's GMRES.
> > >>>>> With my GS, for a steady state problem I can set CFL=40 and for
> > >>>>> unsteady case can set dt=0.1. However, for GMRES I can't set CFL more
> > >>>>> than 5 and for unsteady case dt more than 0.00001. I need GMRES for
> > >>>>> parallel computations so I cannot use GS for this purpose. Is there a
> > >>>>> way to improve the stability of GMRES?
> > >>>>
> > >>
> >
> >
> 
> 


From bsmith at mcs.anl.gov  Tue Mar 10 21:02:59 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 10 Mar 2015 21:02:59 -0500
Subject: [petsc-users] PETSc conference and tutorials June 15 - 18,
	2015 at ANL
References: <AE77F513-B9B0-47FA-A3DF-3A6C6F08058C@mcs.anl.gov>
Message-ID: <B123D1C0-3251-4B55-9E9A-3875C1252746@mcs.anl.gov>


   This is a reminder to please register for the PETSc 20th anniversary conference and tutorial from June 15 to June 18 2014 at Argonne National Laboratory as soon as possible. The conference announcement can be found at http://www.mcs.anl.gov/petsc-20 and in the attached PDE document. There is still some travel support available for students and post-docs. 

  Hope to see you there,

   Barry

For those who have already applied for travel support we plan to be in touch with you soon with the current status.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc-20.pdf
Type: application/pdf
Size: 2548842 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/8c98b372/attachment-0002.pdf>
-------------- next part --------------


> Begin forwarded message:
> 
> From: Barry Smith <bsmith at mcs.anl.gov>
> Reply-To: petsc-20 at mcs.anl.gov
> Subject: PETSc conference and tutorials June 15 - 18, 2015 at ANL
> Date: December 21, 2014 at 6:09:16 PM CST
> Cc: petsc-20 at mcs.anl.gov
> To: petsc-dev <petsc-dev at mcs.anl.gov>
> 
> We are excited to invite you to the PETSc 20th anniversary conference and tutorial from June 15 to June 18 2014 at Argonne National Laboratory. The conference announcement can be found at http://www.mcs.anl.gov/petsc-20 and in the attached PDE document. Thanks to the generosity of the Computing, Environment, and Life Sciences (CELS) directorate of Argonne, there are no registration fees, and we can provide some travel support for students, post-docs and those with limited travel budgets. Please register soon so we can include you in the program.
> 
> Hope to see you there,
> 
> Barry 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc-20.pdf
Type: application/pdf
Size: 2548842 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150310/8c98b372/attachment-0003.pdf>
-------------- next part --------------
> 


From bhatiamanav at gmail.com  Wed Mar 11 13:12:50 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 11 Mar 2015 13:12:50 -0500
Subject: [petsc-users] make check error
Message-ID: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>

Greetings! 

   I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 

Talon-login[114] bhatia$ make check
gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
Running test examples to verify correct installation
Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
*******************Error detected during compile or link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
*********************************************************************************
gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/bin/rm -f ex19.o
gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
CMA: unable to open /dev/infiniband/rdma_cm
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
[Talon-login:18281] *** Process received signal ***
[Talon-login:18281] Signal: Segmentation fault (11)
[Talon-login:18281] Signal code: Address not mapped (1)
[Talon-login:18281] Failing at address: 0x100002990
[Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
[Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
[Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
[Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
[Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
[Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
[Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
[Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
[Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
[Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
[Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
[Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
[Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
[Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
[Talon-login:18281] [14] ./ex19 [0x482639]
[Talon-login:18281] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
CMA: unable to open /dev/infiniband/rdma_cm
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
[Talon-login:18286] *** Process received signal ***
[Talon-login:18286] Signal: Segmentation fault (11)
[Talon-login:18286] Signal code: Address not mapped (1)
[Talon-login:18286] Failing at address: 0x100002990
[Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
[Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
[Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
[Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
[Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
[Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
[Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
[Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
[Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
[Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
[Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
[Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
[Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
[Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
[Talon-login:18286] [14] ./ex19 [0x482639]
[Talon-login:18286] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
*******************Error detected during compile or link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
*********************************************************
gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
/bin/rm -f ex5f.o
gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
CMA: unable to open /dev/infiniband/rdma_cm
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2B6CD4A6F4D7
#1  0x2B6CD4A6FADE
#2  0x2B6CD5805E1F
#3  0x2B6CD716F4CE
#4  0x2B6CD716E49D
#5  0x2B6CD7280159
#6  0x2B6CD46E5165
#7  0x2B6CD6E59820
#8  0x2B6CD46E4971
#9  0x2B6CD6C41C8E
#10  0x2B6CD46EE742
#11  0x2B6CD46B204F
#12  0x2B6CD46CF99F
#13  0x2B6CD494AA84
#14  0x489A6D in petscinitialize_
#15  0x487CEC in MAIN__ at ex5f.F:?
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
Completed test examples
gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
=========================================
Now to evaluate the computer systems you plan use - do:
make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
Talon-login[115] bhatia$ 


Thanks,
Manav

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/2e340b76/attachment-0001.html>

From bhatiamanav at gmail.com  Wed Mar 11 13:20:54 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 11 Mar 2015 13:20:54 -0500
Subject: [petsc-users] make check error
In-Reply-To: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
Message-ID: <BE627BCA-7BB8-4A32-BB0B-890AA678A47E@gmail.com>

One a different machine, but with same version of code, I am getting the following error. There is some complaint about unable to open /dev/infiniband/rdma_cm. 

Any pointers would be greatly appreciated. 

Thanks,
Manav


shadow-login[238] bhatia$ make check
Running test examples to verify correct installation
Using PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'shadow-login', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           shadow-login
  Local device:         mlx4_1
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              shadow-login
  Registerable memory:     24576 MiB
  Total memory:            65457 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
Number of SNES iterations = 2
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'shadow-login', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           shadow-login
  Local device:         mlx4_1
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              shadow-login
  Registerable memory:     24576 MiB
  Total memory:            65457 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
Number of SNES iterations = 2
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 196567 on node shadow-login exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
[shadow-login:196564] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / reg mem limit low
Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'shadow-login', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           shadow-login
  Local device:         mlx4_1
  Local port:           1
  CPCs attempted:       udcm
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              shadow-login
  Registerable memory:     24576 MiB
  Total memory:            65457 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
Number of SNES iterations =     4
Completed test examples
=========================================
Now to evaluate the computer systems you plan use - do:
make PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
shadow-login[239] bhatia$ 



> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Greetings! 
> 
>    I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
> 
> Talon-login[114] bhatia$ make check
> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> Running test examples to verify correct installation
> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html>
> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
> *********************************************************************************
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /bin/rm -f ex19.o
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html>
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> [Talon-login:18281] *** Process received signal ***
> [Talon-login:18281] Signal: Segmentation fault (11)
> [Talon-login:18281] Signal code: Address not mapped (1)
> [Talon-login:18281] Failing at address: 0x100002990
> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
> [Talon-login:18281] [14] ./ex19 [0x482639]
> [Talon-login:18281] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu <http://talon-login.hpc.msstate.edu/> exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html>
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> [Talon-login:18286] *** Process received signal ***
> [Talon-login:18286] Signal: Segmentation fault (11)
> [Talon-login:18286] Signal code: Address not mapped (1)
> [Talon-login:18286] Failing at address: 0x100002990
> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
> [Talon-login:18286] [14] ./ex19 [0x482639]
> [Talon-login:18286] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu <http://talon-login.hpc.msstate.edu/> exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html>
> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
> *********************************************************
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /bin/rm -f ex5f.o
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html>
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> 
> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
> 
> Backtrace for this error:
> #0  0x2B6CD4A6F4D7
> #1  0x2B6CD4A6FADE
> #2  0x2B6CD5805E1F
> #3  0x2B6CD716F4CE
> #4  0x2B6CD716E49D
> #5  0x2B6CD7280159
> #6  0x2B6CD46E5165
> #7  0x2B6CD6E59820
> #8  0x2B6CD46E4971
> #9  0x2B6CD6C41C8E
> #10  0x2B6CD46EE742
> #11  0x2B6CD46B204F
> #12  0x2B6CD46CF99F
> #13  0x2B6CD494AA84
> #14  0x489A6D in petscinitialize_
> #15  0x487CEC in MAIN__ at ex5f.F:?
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu <http://talon-login.hpc.msstate.edu/> exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Completed test examples
> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> =========================================
> Now to evaluate the computer systems you plan use - do:
> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> Talon-login[115] bhatia$ 
> 
> 
> Thanks,
> Manav
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/638d5278/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Mar 11 13:21:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 13:21:17 -0500
Subject: [petsc-users] make check error
In-Reply-To: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
Message-ID: <D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>


  Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.

  Barry


> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Greetings! 
> 
>    I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
> 
> Talon-login[114] bhatia$ make check
> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> Running test examples to verify correct installation
> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
> *********************************************************************************
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /bin/rm -f ex19.o
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> [Talon-login:18281] *** Process received signal ***
> [Talon-login:18281] Signal: Segmentation fault (11)
> [Talon-login:18281] Signal code: Address not mapped (1)
> [Talon-login:18281] Failing at address: 0x100002990
> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
> [Talon-login:18281] [14] ./ex19 [0x482639]
> [Talon-login:18281] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> [Talon-login:18286] *** Process received signal ***
> [Talon-login:18286] Signal: Segmentation fault (11)
> [Talon-login:18286] Signal code: Address not mapped (1)
> [Talon-login:18286] Failing at address: 0x100002990
> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
> [Talon-login:18286] [14] ./ex19 [0x482639]
> [Talon-login:18286] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
> *********************************************************
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> /bin/rm -f ex5f.o
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> CMA: unable to open /dev/infiniband/rdma_cm
> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>     This will severely limit memory registrations.
> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> 
> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
> 
> Backtrace for this error:
> #0  0x2B6CD4A6F4D7
> #1  0x2B6CD4A6FADE
> #2  0x2B6CD5805E1F
> #3  0x2B6CD716F4CE
> #4  0x2B6CD716E49D
> #5  0x2B6CD7280159
> #6  0x2B6CD46E5165
> #7  0x2B6CD6E59820
> #8  0x2B6CD46E4971
> #9  0x2B6CD6C41C8E
> #10  0x2B6CD46EE742
> #11  0x2B6CD46B204F
> #12  0x2B6CD46CF99F
> #13  0x2B6CD494AA84
> #14  0x489A6D in petscinitialize_
> #15  0x487CEC in MAIN__ at ex5f.F:?
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> Completed test examples
> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> =========================================
> Now to evaluate the computer systems you plan use - do:
> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> Talon-login[115] bhatia$ 
> 
> 
> Thanks,
> Manav
> 


From bsmith at mcs.anl.gov  Wed Mar 11 13:24:03 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 13:24:03 -0500
Subject: [petsc-users] make check error
In-Reply-To: <BE627BCA-7BB8-4A32-BB0B-890AA678A47E@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<BE627BCA-7BB8-4A32-BB0B-890AA678A47E@gmail.com>
Message-ID: <A6D19B70-B3C5-4B6F-912D-5468DFF043B8@mcs.anl.gov>


  This is a problem with your MPI installation. You need to talk to a local systems person or check the documentation on how to set up your environment to run MPI jobs on that system.

  Barry

> On Mar 11, 2015, at 1:20 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> One a different machine, but with same version of code, I am getting the following error. There is some complaint about unable to open /dev/infiniband/rdma_cm. 
> 
> Any pointers would be greatly appreciated. 
> 
> Thanks,
> Manav
> 
> 
> shadow-login[238] bhatia$ make check
> Running test examples to verify correct installation
> Using PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> librdmacm: Warning: couldn't read ABI version.
> librdmacm: Warning: assuming: 4
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> --------------------------------------------------------------------------
> WARNING: There are more than one active ports on host 'shadow-login', but the
> default subnet GID prefix was detected on more than one of these
> ports.  If these ports are connected to different physical IB
> networks, this configuration will fail in Open MPI.  This version of
> Open MPI requires that every physically separate IB subnet that is
> used between connected MPI processes must have different subnet ID
> values.
> 
> Please see this FAQ entry for more details:
> 
>   http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid
> 
> NOTE: You can turn off this warning by setting the MCA parameter
>       btl_openib_warn_default_gid_prefix to 0.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> No OpenFabrics connection schemes reported that they were able to be
> used on a specific port.  As such, the openib BTL (OpenFabrics
> support) will be disabled for this port.
> 
>   Local host:           shadow-login
>   Local device:         mlx4_1
>   Local port:           1
>   CPCs attempted:       udcm
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.  This can cause MPI jobs to
> run with erratic performance, hang, and/or crash.
> 
> This may be caused by your OpenFabrics vendor limiting the amount of
> physical memory that can be registered.  You should investigate the
> relevant Linux kernel module parameters that control how much physical
> memory can be registered, and increase them to allow registering all
> physical memory on your machine.
> 
> See this Open MPI FAQ item for more information on these Linux kernel module
> parameters:
> 
>     http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
> 
>   Local host:              shadow-login
>   Registerable memory:     24576 MiB
>   Total memory:            65457 MiB
> 
> Your MPI job will continue, but may be behave poorly and/or hang.
> --------------------------------------------------------------------------
> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> Number of SNES iterations = 2
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> librdmacm: Warning: couldn't read ABI version.
> librdmacm: Warning: assuming: 4
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> librdmacm: Warning: couldn't read ABI version.
> librdmacm: Warning: assuming: 4
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> --------------------------------------------------------------------------
> WARNING: There are more than one active ports on host 'shadow-login', but the
> default subnet GID prefix was detected on more than one of these
> ports.  If these ports are connected to different physical IB
> networks, this configuration will fail in Open MPI.  This version of
> Open MPI requires that every physically separate IB subnet that is
> used between connected MPI processes must have different subnet ID
> values.
> 
> Please see this FAQ entry for more details:
> 
>   http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid
> 
> NOTE: You can turn off this warning by setting the MCA parameter
>       btl_openib_warn_default_gid_prefix to 0.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> No OpenFabrics connection schemes reported that they were able to be
> used on a specific port.  As such, the openib BTL (OpenFabrics
> support) will be disabled for this port.
> 
>   Local host:           shadow-login
>   Local device:         mlx4_1
>   Local port:           1
>   CPCs attempted:       udcm
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.  This can cause MPI jobs to
> run with erratic performance, hang, and/or crash.
> 
> This may be caused by your OpenFabrics vendor limiting the amount of
> physical memory that can be registered.  You should investigate the
> relevant Linux kernel module parameters that control how much physical
> memory can be registered, and increase them to allow registering all
> physical memory on your machine.
> 
> See this Open MPI FAQ item for more information on these Linux kernel module
> parameters:
> 
>     http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
> 
>   Local host:              shadow-login
>   Registerable memory:     24576 MiB
>   Total memory:            65457 MiB
> 
> Your MPI job will continue, but may be behave poorly and/or hang.
> --------------------------------------------------------------------------
> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> Number of SNES iterations = 2
> --------------------------------------------------------------------------
> mpirun noticed that process rank 1 with PID 196567 on node shadow-login exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> [shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
> [shadow-login:196564] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> [shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
> [shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / reg mem limit low
> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> librdmacm: Warning: couldn't read ABI version.
> librdmacm: Warning: assuming: 4
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm
> --------------------------------------------------------------------------
> WARNING: There are more than one active ports on host 'shadow-login', but the
> default subnet GID prefix was detected on more than one of these
> ports.  If these ports are connected to different physical IB
> networks, this configuration will fail in Open MPI.  This version of
> Open MPI requires that every physically separate IB subnet that is
> used between connected MPI processes must have different subnet ID
> values.
> 
> Please see this FAQ entry for more details:
> 
>   http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid
> 
> NOTE: You can turn off this warning by setting the MCA parameter
>       btl_openib_warn_default_gid_prefix to 0.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> No OpenFabrics connection schemes reported that they were able to be
> used on a specific port.  As such, the openib BTL (OpenFabrics
> support) will be disabled for this port.
> 
>   Local host:           shadow-login
>   Local device:         mlx4_1
>   Local port:           1
>   CPCs attempted:       udcm
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.  This can cause MPI jobs to
> run with erratic performance, hang, and/or crash.
> 
> This may be caused by your OpenFabrics vendor limiting the amount of
> physical memory that can be registered.  You should investigate the
> relevant Linux kernel module parameters that control how much physical
> memory can be registered, and increase them to allow registering all
> physical memory on your machine.
> 
> See this Open MPI FAQ item for more information on these Linux kernel module
> parameters:
> 
>     http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
> 
>   Local host:              shadow-login
>   Registerable memory:     24576 MiB
>   Total memory:            65457 MiB
> 
> Your MPI job will continue, but may be behave poorly and/or hang.
> --------------------------------------------------------------------------
> Number of SNES iterations =     4
> Completed test examples
> =========================================
> Now to evaluate the computer systems you plan use - do:
> make PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> shadow-login[239] bhatia$ 
> 
> 
> 
>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Greetings! 
>> 
>>    I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>> 
>> Talon-login[114] bhatia$ make check
>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>> Running test examples to verify correct installation
>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>> *********************************************************************************
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /bin/rm -f ex19.o
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> [Talon-login:18281] *** Process received signal ***
>> [Talon-login:18281] Signal: Segmentation fault (11)
>> [Talon-login:18281] Signal code: Address not mapped (1)
>> [Talon-login:18281] Failing at address: 0x100002990
>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>> [Talon-login:18281] [14] ./ex19 [0x482639]
>> [Talon-login:18281] *** End of error message ***
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> [Talon-login:18286] *** Process received signal ***
>> [Talon-login:18286] Signal: Segmentation fault (11)
>> [Talon-login:18286] Signal code: Address not mapped (1)
>> [Talon-login:18286] Failing at address: 0x100002990
>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>> [Talon-login:18286] [14] ./ex19 [0x482639]
>> [Talon-login:18286] *** End of error message ***
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>> *********************************************************
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /bin/rm -f ex5f.o
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>     This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> 
>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>> 
>> Backtrace for this error:
>> #0  0x2B6CD4A6F4D7
>> #1  0x2B6CD4A6FADE
>> #2  0x2B6CD5805E1F
>> #3  0x2B6CD716F4CE
>> #4  0x2B6CD716E49D
>> #5  0x2B6CD7280159
>> #6  0x2B6CD46E5165
>> #7  0x2B6CD6E59820
>> #8  0x2B6CD46E4971
>> #9  0x2B6CD6C41C8E
>> #10  0x2B6CD46EE742
>> #11  0x2B6CD46B204F
>> #12  0x2B6CD46CF99F
>> #13  0x2B6CD494AA84
>> #14  0x489A6D in petscinitialize_
>> #15  0x487CEC in MAIN__ at ex5f.F:?
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Completed test examples
>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>> =========================================
>> Now to evaluate the computer systems you plan use - do:
>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>> Talon-login[115] bhatia$ 
>> 
>> 
>> Thanks,
>> Manav
>> 
> 


From bhatiamanav at gmail.com  Wed Mar 11 13:24:51 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 11 Mar 2015 13:24:51 -0500
Subject: [petsc-users] make check error
In-Reply-To: <D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
Message-ID: <1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>

I apologize for not sending these files along. Here they are for the two machines. 

Really appreciate your help. 

-Manav

-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure_shadow.log
Type: application/octet-stream
Size: 6842513 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/51919fc7/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure_talon.log
Type: application/octet-stream
Size: 6600478 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/51919fc7/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make_shadow.log
Type: application/octet-stream
Size: 92286 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/51919fc7/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make_talon.log
Type: application/octet-stream
Size: 932604 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/51919fc7/attachment-0007.obj>
-------------- next part --------------

> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
> 
>  Barry
> 
> 
>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Greetings! 
>> 
>>   I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>> 
>> Talon-login[114] bhatia$ make check
>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>> Running test examples to verify correct installation
>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>> *********************************************************************************
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /bin/rm -f ex19.o
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> [Talon-login:18281] *** Process received signal ***
>> [Talon-login:18281] Signal: Segmentation fault (11)
>> [Talon-login:18281] Signal code: Address not mapped (1)
>> [Talon-login:18281] Failing at address: 0x100002990
>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>> [Talon-login:18281] [14] ./ex19 [0x482639]
>> [Talon-login:18281] *** End of error message ***
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> [Talon-login:18286] *** Process received signal ***
>> [Talon-login:18286] Signal: Segmentation fault (11)
>> [Talon-login:18286] Signal code: Address not mapped (1)
>> [Talon-login:18286] Failing at address: 0x100002990
>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>> [Talon-login:18286] [14] ./ex19 [0x482639]
>> [Talon-login:18286] *** End of error message ***
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>> *********************************************************
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> /bin/rm -f ex5f.o
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>> CMA: unable to open /dev/infiniband/rdma_cm
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>    This will severely limit memory registrations.
>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>> 
>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>> 
>> Backtrace for this error:
>> #0  0x2B6CD4A6F4D7
>> #1  0x2B6CD4A6FADE
>> #2  0x2B6CD5805E1F
>> #3  0x2B6CD716F4CE
>> #4  0x2B6CD716E49D
>> #5  0x2B6CD7280159
>> #6  0x2B6CD46E5165
>> #7  0x2B6CD6E59820
>> #8  0x2B6CD46E4971
>> #9  0x2B6CD6C41C8E
>> #10  0x2B6CD46EE742
>> #11  0x2B6CD46B204F
>> #12  0x2B6CD46CF99F
>> #13  0x2B6CD494AA84
>> #14  0x489A6D in petscinitialize_
>> #15  0x487CEC in MAIN__ at ex5f.F:?
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>> Completed test examples
>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>> =========================================
>> Now to evaluate the computer systems you plan use - do:
>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>> Talon-login[115] bhatia$ 
>> 
>> 
>> Thanks,
>> Manav
>> 
> 


From bsmith at mcs.anl.gov  Wed Mar 11 13:36:58 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 13:36:58 -0500
Subject: [petsc-users] make check error
In-Reply-To: <1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
Message-ID: <1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>


  Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=

  I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue

  Barry



> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> I apologize for not sending these files along. Here they are for the two machines. 
> 
> Really appreciate your help. 
> 
> -Manav
> 
> 
> > On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > 
> > 
> >  Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
> > 
> >  Barry
> > 
> > 
> >> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> >> 
> >> Greetings! 
> >> 
> >>   I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
> >> 
> >> Talon-login[114] bhatia$ make check
> >> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> >> Running test examples to verify correct installation
> >> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> *******************Error detected during compile or link!*******************
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
> >> *********************************************************************************
> >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
> >> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /bin/rm -f ex19.o
> >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >> CMA: unable to open /dev/infiniband/rdma_cm
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> [Talon-login:18281] *** Process received signal ***
> >> [Talon-login:18281] Signal: Segmentation fault (11)
> >> [Talon-login:18281] Signal code: Address not mapped (1)
> >> [Talon-login:18281] Failing at address: 0x100002990
> >> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
> >> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
> >> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
> >> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
> >> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
> >> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
> >> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
> >> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
> >> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
> >> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
> >> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
> >> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> >> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
> >> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
> >> [Talon-login:18281] [14] ./ex19 [0x482639]
> >> [Talon-login:18281] *** End of error message ***
> >> --------------------------------------------------------------------------
> >> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >> --------------------------------------------------------------------------
> >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >> CMA: unable to open /dev/infiniband/rdma_cm
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> [Talon-login:18286] *** Process received signal ***
> >> [Talon-login:18286] Signal: Segmentation fault (11)
> >> [Talon-login:18286] Signal code: Address not mapped (1)
> >> [Talon-login:18286] Failing at address: 0x100002990
> >> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
> >> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
> >> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
> >> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
> >> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
> >> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
> >> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
> >> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
> >> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
> >> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
> >> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
> >> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> >> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
> >> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
> >> [Talon-login:18286] [14] ./ex19 [0x482639]
> >> [Talon-login:18286] *** End of error message ***
> >> --------------------------------------------------------------------------
> >> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >> --------------------------------------------------------------------------
> >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> *******************Error detected during compile or link!*******************
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
> >> *********************************************************
> >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
> >> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >> /bin/rm -f ex5f.o
> >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >> CMA: unable to open /dev/infiniband/rdma_cm
> >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>    This will severely limit memory registrations.
> >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >> 
> >> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
> >> 
> >> Backtrace for this error:
> >> #0  0x2B6CD4A6F4D7
> >> #1  0x2B6CD4A6FADE
> >> #2  0x2B6CD5805E1F
> >> #3  0x2B6CD716F4CE
> >> #4  0x2B6CD716E49D
> >> #5  0x2B6CD7280159
> >> #6  0x2B6CD46E5165
> >> #7  0x2B6CD6E59820
> >> #8  0x2B6CD46E4971
> >> #9  0x2B6CD6C41C8E
> >> #10  0x2B6CD46EE742
> >> #11  0x2B6CD46B204F
> >> #12  0x2B6CD46CF99F
> >> #13  0x2B6CD494AA84
> >> #14  0x489A6D in petscinitialize_
> >> #15  0x487CEC in MAIN__ at ex5f.F:?
> >> --------------------------------------------------------------------------
> >> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >> --------------------------------------------------------------------------
> >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >> Completed test examples
> >> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> >> =========================================
> >> Now to evaluate the computer systems you plan use - do:
> >> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> >> Talon-login[115] bhatia$ 
> >> 
> >> 
> >> Thanks,
> >> Manav
> >> 
> > 
> 
> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>


From bsmith at mcs.anl.gov  Wed Mar 11 13:47:22 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 13:47:22 -0500
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <1426013835.2659.37.camel@cites-gss.com>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
	<1426007602.2659.25.camel@cites-gss.com>
	<E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>
	<1426013835.2659.37.camel@cites-gss.com>
Message-ID: <94C36ABC-BFF9-4D90-9529-1567DB569DB4@mcs.anl.gov>


   Thanks for the suggestions and observations. I have made a new branch barry/improve-afterimage 
that allows one to set the file format saved by passing it in the filename. On my Mac with the --download-afterimage only .Gif and .Jpeg formats are supported (by Afterimage) but that will depend on your install of Afterimage I guess (Png gives me the same error message you get).

  Barry

> On Mar 10, 2015, at 1:58 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> 
> On Tue, 2015-03-10 at 13:02 -0500, Barry Smith wrote:
>>  Looks like your afterimage is not installed to support .Gif files. When I install afterimage on my Mac from source http://www.afterstep.org/afterimage/getcode.php it just comes with support for .Gif files. Suggest trying that.
> 
> Yes, I realized that. Perhaps there is something about the GIF license
> that the Debian team had to remove GIF support. I have tried to search
> in the changelogs but could not find any reference. BTW, as I wrote in
> my earlier emails, Debian provides version AfterImage 2.2.12 whilst in
> the URL you quote (which is the only one that Google returns) the latest
> version is 1.20, which I cannot have PETSc to use (see the configure.log
> transcript below).
> 
> My workwaround was to change the format in which PETSc asks AfterImage
> to dump the draw, i.e. line 395
> of /home/gtheler/libs/petsc-3.5.3/src/sys/classes/draw/impls/x/xinit.c:
> 
> ASImage2file(asimage, 0, filename,ASIT_Gif,0);
> 
> So far I tried ASIT_Png but the library complains about not being able
> to initialize zlib, but I think I should be able to fix. ASIT_Bmp works
> fine.
> 
> Is there anything special about GIF within PETSc? IIRC, there are indeed
> some issues with the license and this format was virtually replaced with
> PNG almost all over the place. Maybe it would be a good idea to add a
> database key so as to select the format in PetscDrawSetSave().
> 
> 
> According to AfterImage API documentation, valid file types are
> 
> typedef enum
> {
>    ASIT_Xpm = 0,
>    ASIT_ZCompressedXpm,
>    ASIT_GZCompressedXpm,
>    ASIT_Png,
>    ASIT_Jpeg,
>    ASIT_Xcf,
>    ASIT_Ppm,
>    ASIT_Pnm,
>    ASIT_Bmp,
>    ASIT_Ico,
>    ASIT_Cur,
>    ASIT_Gif,
>    ASIT_Tiff,
>    ASIT_XMLScript,
>    ASIT_SVG,
>    /* reserved for future implementation : */
>    ASIT_Xbm,
>    ASIT_Targa,
>    ASIT_Supported = ASIT_Targa,
>    ASIT_Pcx,
>    ASIT_HTML,
>    ASIT_XML,
>    ASIT_Unknown
> }ASImageFileTypes;
> 
> 
> Output of configure.log with libAfterImage 1.20 as downloaded from
> http://www.afterstep.org/afterimage/getcode.php:
> 
> 
> Source:
> #include "confdefs.h"
> #include "conffix.h"
> /* Override any gcc2 internal prototype to avoid an error. */
> char picture_ximage2asimage();
> static void _check_picture_ximage2asimage()
> { picture_ximage2asimage(); }
> 
> int main() {
> _check_picture_ximage2asimage();;
>  return 0;
> }
>                      Pushing language C
>                      Popping language C
> Executing: mpicc  -o /tmp/petsc-s8uvAW/config.libraries/conftest
> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -g3 -O0 /tmp/petsc-s8uvAW/config.libraries/conftest.o
> -Wl,-rpath,/home/gtheler/libs/libAfterImage-1.20
> -L/home/gtheler/libs/libAfterImage-1.20 -lAfterImage -lX11
> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.9
> -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lcr -lpthread
> -lgcc_s -ldl
> Possible ERROR while running linker: exit code 256
> stderr:
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
> function `asimage2drawable_gl':
> ximage.c:(.text+0x1178): undefined reference to `show_warning'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
> function `asimage2drawable':
> ximage.c:(.text+0x1292): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
> function `asimage2alpha_drawable':
> ximage.c:(.text+0x13ac): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ximage.o): In
> function `asimage2alpha':
> ximage.c:(.text+0x157c): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `asimage_start':
> asimage.c:(.text+0x1bb): undefined reference to `safecalloc'
> asimage.c:(.text+0x22e): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `create_asimage':
> asimage.c:(.text+0x38d): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `destroy_asimage':
> asimage.c:(.text+0x865): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `create_image_manager':
> asimage.c:(.text+0x9bf): undefined reference to `mystrdup'
> asimage.c:(.text+0x9f9): undefined reference to `string_compare'
> asimage.c:(.text+0xa00): undefined reference to `string_hash_value'
> asimage.c:(.text+0xa1e): undefined reference to `create_ashash'
> asimage.c:(.text+0xa43): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `destroy_image_manager':
> asimage.c:(.text+0xa74): undefined reference to `destroy_ashash'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `store_asimage':
> asimage.c:(.text+0xb19): undefined reference to `mystrdup'
> asimage.c:(.text+0xb43): undefined reference to `add_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `query_asimage':
> asimage.c:(.text+0xbad): undefined reference to `get_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `fetch_asimage':
> asimage.c:(.text+0xc0d): undefined reference to `get_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `dup_asimage':
> asimage.c:(.text+0xc71): undefined reference to `show_error'
> asimage.c:(.text+0xca4): undefined reference to `show_debug'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `release_asimage':
> asimage.c:(.text+0xd0b): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `forget_asimage':
> asimage.c:(.text+0xd77): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `relocate_asimage':
> asimage.c:(.text+0xdb9): undefined reference to `remove_hash_item'
> asimage.c:(.text+0xde3): undefined reference to `add_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `safe_asimage_destroy':
> asimage.c:(.text+0xe9a): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `release_asimage_by_name':
> asimage.c:(.text+0xedd): undefined reference to `get_hash_item'
> asimage.c:(.text+0xf36): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `flip_gradient':
> asimage.c:(.text+0x1006): undefined reference to `safecalloc'
> asimage.c:(.text+0x1031): undefined reference to `safemalloc'
> asimage.c:(.text+0x103d): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `create_image_layers':
> asimage.c:(.text+0x1226): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `destroy_image_layers':
> asimage.c:(.text+0x12e7): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `asimage_compare_line':
> asimage.c:(.text+0x1a9e): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `set_asimage_vector':
> asimage.c:(.text+0x1dac): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `vectorize_asimage':
> asimage.c:(.text+0x1f6a): undefined reference to `safecalloc'
> asimage.c:(.text+0x1f80): undefined reference to `safemalloc'
> asimage.c:(.text+0x1f90): undefined reference to `safemalloc'
> asimage.c:(.text+0x1fa0): undefined reference to `safemalloc'
> asimage.c:(.text+0x1fb0): undefined reference to `safemalloc'
> asimage.c:(.text+0x1fc0): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o):asimage.c:(.text+0x2143): more undefined references to `safemalloc' follow
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asimage.o): In
> function `forget_asimage_name':
> asimage.c:(.text+0xe22): undefined reference to `remove_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ascmap.o): In
> function `color_hash2colormap':
> ascmap.c:(.text+0x564): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(ascmap.o): In
> function `colormap_asimage':
> ascmap.c:(.text+0x858): undefined reference to `safemalloc'
> ascmap.c:(.text+0x88b): undefined reference to `safecalloc'
> ascmap.c:(.text+0x89f): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `store_data_in_block':
> asstorage.c:(.text+0x530f): undefined reference to `show_error'
> asstorage.c:(.text+0x532d): undefined reference to `show_error'
> asstorage.c:(.text+0x5359): undefined reference to `show_error'
> asstorage.c:(.text+0x5380): undefined reference to `show_error'
> asstorage.c:(.text+0x53a9): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o):asstorage.c:(.text+0x57cd): more undefined references to `show_error' follow
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `store_data_in_block':
> asstorage.c:(.text+0x5afd): undefined reference to `show_warning'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `store_compressed_data':
> asstorage.c:(.text+0x5dcf): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `query_storage_slot':
> asstorage.c:(.text+0x8172): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `print_storage_slot':
> asstorage.c:(.text+0x82f4): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `forget_data':
> asstorage.c:(.text+0x8642): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `dup_data':
> asstorage.c:(.text+0x87cc): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o):asstorage.c:(.text+0x89f5): more undefined references to `show_error' follow
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asstorage.o): In
> function `dup_data':
> asstorage.c:(.text+0x8b4c): undefined reference to `show_warning'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `ximage2scanline_pseudo12bpp':
> asvisual.c:(.text+0x2305): undefined reference to `get_hash_item'
> asvisual.c:(.text+0x2402): undefined reference to `get_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `query_screen_visual_id':
> asvisual.c:(.text+0x26f3): undefined reference to `get_output_threshold'
> asvisual.c:(.text+0x27ca): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `destroy_asvisual':
> asvisual.c:(.text+0x28a8): undefined reference to `destroy_ashash'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `visual2visual_prop':
> asvisual.c:(.text+0x293b): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `visual_prop2visual':
> asvisual.c:(.text+0x2be5): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `make_reverse_colormap':
> asvisual.c:(.text+0x2fc6): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `make_reverse_colorhash':
> asvisual.c:(.text+0x3062): undefined reference to `create_ashash'
> asvisual.c:(.text+0x30c7): undefined reference to `add_hash_item'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `setup_as_colormap':
> asvisual.c:(.text+0x338e): undefined reference to `safemalloc'
> asvisual.c:(.text+0x34ca): undefined reference to `safemalloc'
> asvisual.c:(.text+0x3642): undefined reference to `safemalloc'
> asvisual.c:(.text+0x37ef): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `create_asvisual_for_id':
> asvisual.c:(.text+0x3e81): undefined reference to `safecalloc'
> asvisual.c:(.text+0x3f0c): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `create_visual_window':
> asvisual.c:(.text+0x4118): undefined reference to `show_warning'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(asvisual.o): In
> function `create_visual_ximage':
> asvisual.c:(.text+0x4654): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(blender.o): In
> function `blend_scanlines_name2func':
> blender.c:(.text+0x1821): undefined reference to `mystrncasecmp'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
> function `create_image_xim':
> imencdec.c:(.text+0xe71): undefined reference to `show_error'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
> function `create_image_argb32':
> imencdec.c:(.text+0x1e20): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
> function `start_image_decoding':
> imencdec.c:(.text+0x9218): undefined reference to `safecalloc'
> imencdec.c:(.text+0x9529): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(imencdec.o): In
> function `start_image_output':
> imencdec.c:(.text+0x97e2): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
> function `prepare_scanline':
> scanline.c:(.text+0x179): undefined reference to `safecalloc'
> scanline.c:(.text+0x25b): undefined reference to `safecalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o):scanline.c:(.text+0x3a8): more undefined references to `safecalloc' follow
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
> function `calculate_green_diff':
> scanline.c:(.text+0x16b1): undefined reference to `safemalloc'
> /home/gtheler/libs/libAfterImage-1.20/libAfterImage.a(scanline.o): In
> function `interpolate_green_diff':
> scanline.c:(.text+0x19a3): undefined reference to `safemalloc'
> collect2: error: ld returned 1 exit status
>              Popping language C
> 
> 
> ________________________________
> Imprima este mensaje s?lo si es absolutamente necesario.
> Para imprimir, en lo posible utilice el papel de ambos lados.
> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> 
> 
> 
> ************AVISO DE CONFIDENCIALIDAD************
> 
> El Grupo Sancor Seguros comunica que:
> 
> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> 
> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.


From balay at mcs.anl.gov  Wed Mar 11 13:47:35 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 11 Mar 2015 13:47:35 -0500
Subject: [petsc-users] make check error
In-Reply-To: <1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1503111340210.27891@asterix>

Actually mumps is being used - so fortran is required [by mumps]

The reason for the 'multiple' libgfortran messages is: openmpi is
compiled with an older version of gcc/gfortran

[and I suspect its harmless - as the binaries are created]

The primary issue to check: is this openmpi install functional - if so
how do you run mpi binaries? Perhaps its a cluster - and have to go
through a scheduler.. [i.e make check is perhaps inappropriate for
this case]

Satish
----------

Python version:
2.4.2 (#1, Jan 10 2008, 17:43:47)
[GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]

Executing: mpicc --version
stdout:
gcc (GCC) 4.8.3

Executing: mpicc -show
stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl

Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec


/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3



On Wed, 11 Mar 2015, Barry Smith wrote:

> 
>   Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
> 
>   I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
> 
>   Barry
> 
> 
> 
> > On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> > 
> > I apologize for not sending these files along. Here they are for the two machines. 
> > 
> > Really appreciate your help. 
> > 
> > -Manav
> > 
> > 
> > > On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > 
> > > 
> > >  Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
> > > 
> > >  Barry
> > > 
> > > 
> > >> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> > >> 
> > >> Greetings! 
> > >> 
> > >>   I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
> > >> 
> > >> Talon-login[114] bhatia$ make check
> > >> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> > >> Running test examples to verify correct installation
> > >> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> > >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> *******************Error detected during compile or link!*******************
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
> > >> *********************************************************************************
> > >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
> > >> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /bin/rm -f ex19.o
> > >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> > >> CMA: unable to open /dev/infiniband/rdma_cm
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> [Talon-login:18281] *** Process received signal ***
> > >> [Talon-login:18281] Signal: Segmentation fault (11)
> > >> [Talon-login:18281] Signal code: Address not mapped (1)
> > >> [Talon-login:18281] Failing at address: 0x100002990
> > >> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
> > >> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
> > >> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
> > >> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
> > >> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
> > >> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
> > >> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
> > >> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
> > >> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
> > >> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
> > >> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
> > >> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> > >> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
> > >> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
> > >> [Talon-login:18281] [14] ./ex19 [0x482639]
> > >> [Talon-login:18281] *** End of error message ***
> > >> --------------------------------------------------------------------------
> > >> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> > >> --------------------------------------------------------------------------
> > >> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> > >> CMA: unable to open /dev/infiniband/rdma_cm
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> [Talon-login:18286] *** Process received signal ***
> > >> [Talon-login:18286] Signal: Segmentation fault (11)
> > >> [Talon-login:18286] Signal code: Address not mapped (1)
> > >> [Talon-login:18286] Failing at address: 0x100002990
> > >> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
> > >> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
> > >> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
> > >> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
> > >> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
> > >> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
> > >> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
> > >> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
> > >> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
> > >> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
> > >> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
> > >> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> > >> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
> > >> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
> > >> [Talon-login:18286] [14] ./ex19 [0x482639]
> > >> [Talon-login:18286] *** End of error message ***
> > >> --------------------------------------------------------------------------
> > >> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> > >> --------------------------------------------------------------------------
> > >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> *******************Error detected during compile or link!*******************
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
> > >> *********************************************************
> > >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
> > >> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > >> /bin/rm -f ex5f.o
> > >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> > >> CMA: unable to open /dev/infiniband/rdma_cm
> > >> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> > >>    This will severely limit memory registrations.
> > >> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> > >> 
> > >> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
> > >> 
> > >> Backtrace for this error:
> > >> #0  0x2B6CD4A6F4D7
> > >> #1  0x2B6CD4A6FADE
> > >> #2  0x2B6CD5805E1F
> > >> #3  0x2B6CD716F4CE
> > >> #4  0x2B6CD716E49D
> > >> #5  0x2B6CD7280159
> > >> #6  0x2B6CD46E5165
> > >> #7  0x2B6CD6E59820
> > >> #8  0x2B6CD46E4971
> > >> #9  0x2B6CD6C41C8E
> > >> #10  0x2B6CD46EE742
> > >> #11  0x2B6CD46B204F
> > >> #12  0x2B6CD46CF99F
> > >> #13  0x2B6CD494AA84
> > >> #14  0x489A6D in petscinitialize_
> > >> #15  0x487CEC in MAIN__ at ex5f.F:?
> > >> --------------------------------------------------------------------------
> > >> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> > >> --------------------------------------------------------------------------
> > >> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> > >> Completed test examples
> > >> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> > >> =========================================
> > >> Now to evaluate the computer systems you plan use - do:
> > >> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> > >> Talon-login[115] bhatia$ 
> > >> 
> > >> 
> > >> Thanks,
> > >> Manav
> > >> 
> > > 
> > 
> > <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
> 
> 

From bhatiamanav at gmail.com  Wed Mar 11 13:50:43 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 11 Mar 2015 13:50:43 -0500
Subject: [petsc-users] make check error
In-Reply-To: <alpine.LFD.2.11.1503111340210.27891@asterix>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
	<alpine.LFD.2.11.1503111340210.27891@asterix>
Message-ID: <3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>


> On Mar 11, 2015, at 1:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> Actually mumps is being used - so fortran is required [by mumps]
> 
> The reason for the 'multiple' libgfortran messages is: openmpi is
> compiled with an older version of gcc/gfortran
> 

You are right about openmpi being build with an older version of the compiler. I am trying to mix and match codes.

> [and I suspect its harmless - as the binaries are created]
> 
> The primary issue to check: is this openmpi install functional - if so
> how do you run mpi binaries? Perhaps its a cluster - and have to go
> through a scheduler.. [i.e make check is perhaps inappropriate for
> this case]

This might be true. This is my first time working on this system, so I need to find out the details. 

> 
> Satish
> ----------
> 
> Python version:
> 2.4.2 (#1, Jan 10 2008, 17:43:47)
> [GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]
> 
> Executing: mpicc --version
> stdout:
> gcc (GCC) 4.8.3
> 
> Executing: mpicc -show
> stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
> 
> Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
> erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec
> 
> 
> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> 
> 
> 
> On Wed, 11 Mar 2015, Barry Smith wrote:
> 
>> 
>>  Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
>> 
>>  I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
>> 
>>  Barry
>> 
>> 
>> 
>>> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> 
>>> I apologize for not sending these files along. Here they are for the two machines. 
>>> 
>>> Really appreciate your help. 
>>> 
>>> -Manav
>>> 
>>> 
>>>> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>> 
>>>>> Greetings! 
>>>>> 
>>>>>  I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>>>>> 
>>>>> Talon-login[114] bhatia$ make check
>>>>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>> Running test examples to verify correct installation
>>>>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> *******************Error detected during compile or link!*******************
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>>>>> *********************************************************************************
>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>>>>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /bin/rm -f ex19.o
>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> [Talon-login:18281] *** Process received signal ***
>>>>> [Talon-login:18281] Signal: Segmentation fault (11)
>>>>> [Talon-login:18281] Signal code: Address not mapped (1)
>>>>> [Talon-login:18281] Failing at address: 0x100002990
>>>>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>>>>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>>>>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>>>>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>>>>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>>>>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>>>>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>>>>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>>>>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>>>>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>>>>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>>>>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>>>>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>>>>> [Talon-login:18281] [14] ./ex19 [0x482639]
>>>>> [Talon-login:18281] *** End of error message ***
>>>>> --------------------------------------------------------------------------
>>>>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>> --------------------------------------------------------------------------
>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> [Talon-login:18286] *** Process received signal ***
>>>>> [Talon-login:18286] Signal: Segmentation fault (11)
>>>>> [Talon-login:18286] Signal code: Address not mapped (1)
>>>>> [Talon-login:18286] Failing at address: 0x100002990
>>>>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>>>>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>>>>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>>>>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>>>>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>>>>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>>>>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>>>>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>>>>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>>>>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>>>>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>>>>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>>>>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>>>>> [Talon-login:18286] [14] ./ex19 [0x482639]
>>>>> [Talon-login:18286] *** End of error message ***
>>>>> --------------------------------------------------------------------------
>>>>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>> --------------------------------------------------------------------------
>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> *******************Error detected during compile or link!*******************
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>>>>> *********************************************************
>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>>>>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>> /bin/rm -f ex5f.o
>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>   This will severely limit memory registrations.
>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>> 
>>>>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>>>>> 
>>>>> Backtrace for this error:
>>>>> #0  0x2B6CD4A6F4D7
>>>>> #1  0x2B6CD4A6FADE
>>>>> #2  0x2B6CD5805E1F
>>>>> #3  0x2B6CD716F4CE
>>>>> #4  0x2B6CD716E49D
>>>>> #5  0x2B6CD7280159
>>>>> #6  0x2B6CD46E5165
>>>>> #7  0x2B6CD6E59820
>>>>> #8  0x2B6CD46E4971
>>>>> #9  0x2B6CD6C41C8E
>>>>> #10  0x2B6CD46EE742
>>>>> #11  0x2B6CD46B204F
>>>>> #12  0x2B6CD46CF99F
>>>>> #13  0x2B6CD494AA84
>>>>> #14  0x489A6D in petscinitialize_
>>>>> #15  0x487CEC in MAIN__ at ex5f.F:?
>>>>> --------------------------------------------------------------------------
>>>>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>> --------------------------------------------------------------------------
>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>> Completed test examples
>>>>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>> =========================================
>>>>> Now to evaluate the computer systems you plan use - do:
>>>>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>>>>> Talon-login[115] bhatia$ 
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Manav
>>>>> 
>>>> 
>>> 
>>> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
>> 
>> 


From balay at mcs.anl.gov  Wed Mar 11 13:55:21 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 11 Mar 2015 13:55:21 -0500
Subject: [petsc-users] make check error
In-Reply-To: <3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
	<alpine.LFD.2.11.1503111340210.27891@asterix>
	<3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
Message-ID: <alpine.LFD.2.11.1503111352390.27891@asterix>

Perhaps they have a different MPI install to go with "(GCC) 4.8.3"
compilers - that you should look for.

Satish

On Wed, 11 Mar 2015, Manav Bhatia wrote:

> 
> > On Mar 11, 2015, at 1:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > Actually mumps is being used - so fortran is required [by mumps]
> > 
> > The reason for the 'multiple' libgfortran messages is: openmpi is
> > compiled with an older version of gcc/gfortran
> > 
> 
> You are right about openmpi being build with an older version of the compiler. I am trying to mix and match codes.
> 
> > [and I suspect its harmless - as the binaries are created]
> > 
> > The primary issue to check: is this openmpi install functional - if so
> > how do you run mpi binaries? Perhaps its a cluster - and have to go
> > through a scheduler.. [i.e make check is perhaps inappropriate for
> > this case]
> 
> This might be true. This is my first time working on this system, so I need to find out the details. 
> 
> > 
> > Satish
> > ----------
> > 
> > Python version:
> > 2.4.2 (#1, Jan 10 2008, 17:43:47)
> > [GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]
> > 
> > Executing: mpicc --version
> > stdout:
> > gcc (GCC) 4.8.3
> > 
> > Executing: mpicc -show
> > stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
> > 
> > Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
> > erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec
> > 
> > 
> > /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> > 
> > 
> > 
> > On Wed, 11 Mar 2015, Barry Smith wrote:
> > 
> >> 
> >>  Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
> >> 
> >>  I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
> >> 
> >>  Barry
> >> 
> >> 
> >> 
> >>> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> >>> 
> >>> I apologize for not sending these files along. Here they are for the two machines. 
> >>> 
> >>> Really appreciate your help. 
> >>> 
> >>> -Manav
> >>> 
> >>> 
> >>>> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>>> 
> >>>> 
> >>>> Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
> >>>> 
> >>>> Barry
> >>>> 
> >>>> 
> >>>>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> >>>>> 
> >>>>> Greetings! 
> >>>>> 
> >>>>>  I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
> >>>>> 
> >>>>> Talon-login[114] bhatia$ make check
> >>>>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> >>>>> Running test examples to verify correct installation
> >>>>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
> >>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> *******************Error detected during compile or link!*******************
> >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
> >>>>> *********************************************************************************
> >>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
> >>>>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /bin/rm -f ex19.o
> >>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >>>>> CMA: unable to open /dev/infiniband/rdma_cm
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> [Talon-login:18281] *** Process received signal ***
> >>>>> [Talon-login:18281] Signal: Segmentation fault (11)
> >>>>> [Talon-login:18281] Signal code: Address not mapped (1)
> >>>>> [Talon-login:18281] Failing at address: 0x100002990
> >>>>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
> >>>>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
> >>>>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
> >>>>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
> >>>>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
> >>>>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
> >>>>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
> >>>>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
> >>>>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
> >>>>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
> >>>>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
> >>>>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> >>>>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
> >>>>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
> >>>>> [Talon-login:18281] [14] ./ex19 [0x482639]
> >>>>> [Talon-login:18281] *** End of error message ***
> >>>>> --------------------------------------------------------------------------
> >>>>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >>>>> --------------------------------------------------------------------------
> >>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >>>>> CMA: unable to open /dev/infiniband/rdma_cm
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> [Talon-login:18286] *** Process received signal ***
> >>>>> [Talon-login:18286] Signal: Segmentation fault (11)
> >>>>> [Talon-login:18286] Signal code: Address not mapped (1)
> >>>>> [Talon-login:18286] Failing at address: 0x100002990
> >>>>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
> >>>>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
> >>>>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
> >>>>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
> >>>>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
> >>>>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
> >>>>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
> >>>>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
> >>>>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
> >>>>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
> >>>>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
> >>>>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
> >>>>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
> >>>>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
> >>>>> [Talon-login:18286] [14] ./ex19 [0x482639]
> >>>>> [Talon-login:18286] *** End of error message ***
> >>>>> --------------------------------------------------------------------------
> >>>>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >>>>> --------------------------------------------------------------------------
> >>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> *******************Error detected during compile or link!*******************
> >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
> >>>>> *********************************************************
> >>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
> >>>>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
> >>>>> /bin/rm -f ex5f.o
> >>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
> >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
> >>>>> CMA: unable to open /dev/infiniband/rdma_cm
> >>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
> >>>>>   This will severely limit memory registrations.
> >>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
> >>>>> 
> >>>>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
> >>>>> 
> >>>>> Backtrace for this error:
> >>>>> #0  0x2B6CD4A6F4D7
> >>>>> #1  0x2B6CD4A6FADE
> >>>>> #2  0x2B6CD5805E1F
> >>>>> #3  0x2B6CD716F4CE
> >>>>> #4  0x2B6CD716E49D
> >>>>> #5  0x2B6CD7280159
> >>>>> #6  0x2B6CD46E5165
> >>>>> #7  0x2B6CD6E59820
> >>>>> #8  0x2B6CD46E4971
> >>>>> #9  0x2B6CD6C41C8E
> >>>>> #10  0x2B6CD46EE742
> >>>>> #11  0x2B6CD46B204F
> >>>>> #12  0x2B6CD46CF99F
> >>>>> #13  0x2B6CD494AA84
> >>>>> #14  0x489A6D in petscinitialize_
> >>>>> #15  0x487CEC in MAIN__ at ex5f.F:?
> >>>>> --------------------------------------------------------------------------
> >>>>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
> >>>>> --------------------------------------------------------------------------
> >>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
> >>>>> Completed test examples
> >>>>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
> >>>>> =========================================
> >>>>> Now to evaluate the computer systems you plan use - do:
> >>>>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
> >>>>> Talon-login[115] bhatia$ 
> >>>>> 
> >>>>> 
> >>>>> Thanks,
> >>>>> Manav
> >>>>> 
> >>>> 
> >>> 
> >>> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
> >> 
> >> 
> 
> 

From elbueler at alaska.edu  Wed Mar 11 14:21:12 2015
From: elbueler at alaska.edu (Ed Bueler)
Date: Wed, 11 Mar 2015 13:21:12 -0600
Subject: [petsc-users] evidence that VINEWTONRSLS is broken (in next,
	but not in maint)
Message-ID: <CAOHboJ96=+AMnvjvKfPeu-a1yCLmHbe0-FL1S7x5BExU-Q8ugg@mail.gmail.com>

Dear PETSc --

I sought advice at the FAQ page on non-convergence.  It is a nice page, and
I like the range of advice!

But both
  -pc_type svd
  -pc_type none
seem to break the RSLS SNESVI in the next branch.

I tested with ex9.c (below) and ex58.c in next branch
src/snes/examples/tutorials/, and get the same broken results.  The other
SNESVI-using examples (ex60.c, ex61.c, ex63.c, ex64.c, ex65.c) give
compiler errors for me.

Note that both of the above PC types work fine with non-VI examples (e.g.
ex1.c, ex3.c, ex5.c), of course.  Furthermore, in the maint branch, these
pc_types DO work (very bottom) with ex9, and with ex58 (not shown).

I am suspicious of around lines 503-505 in virs.c.  That has been the
failure location for a lot of my tries on various RSLS VI experiments.  I
have looked at the source (i.e. virs.c) itself, and recognize roughly what
is going on.  Objects are being reset because of a change in the active
set; it's nontrivial.  I don't see how to fix it.

Thanks for help!

Ed

**** next branch

~/petsc-next[next]$ export PETSC_ARCH=linux-c-dbg
~/petsc-next[next]$ cd src/snes/examples/tutorials/
~/petsc-next/src/snes/examples/tutorials[next]$ make ex9
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make
rule.
/home/ed/petsc-next/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
-I/home/ed/petsc-next/include -I/home/ed/petsc-next/linux-c-dbg/include
 `pwd`/ex9.c
/home/ed/petsc-next/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o
-Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib
-L/home/ed/petsc-next/linux-c-dbg/lib  -lpetsc
-Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lflapack -lfblas -lX11
-lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath
-lm -lmpicxx -lstdc++ -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib
-L/home/ed/petsc-next/linux-c-dbg/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -ldl
-Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lmpi -lgcc_s -ldl
/bin/rm -f ex9.o
~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type lu
setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9
setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type svd
setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Corrupt argument:
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT
Date: 2015-03-10 22:35:11 -0500
[0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar
11 12:47:08 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-fblaslapack --download-mpich
--with-debugging=1
[0]PETSC ERROR: #1 VecDestroy() line 424 in
/home/ed/petsc-next/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #2 PCReset_SVD() line 295 in
/home/ed/petsc-next/src/ksp/pc/impls/svd/svd.c
[0]PETSC ERROR: #3 PCReset() line 81 in
/home/ed/petsc-next/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPReset() line 861 in
/home/ed/petsc-next/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 SNESVIResetPCandKSP() line 305 in
/home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
[0]PETSC ERROR: #6 SNESSolve_VINEWTONRSLS() line 503 in
/home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
[0]PETSC ERROR: #7 SNESSolve() line 3846 in
/home/ed/petsc-next/src/snes/interface/snes.c
[0]PETSC ERROR: #8 main() line 100 in
/home/ed/petsc-next/src/snes/examples/tutorials/ex9.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -pc_type svd
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type none
setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PCMGSetGalerkin line 1028
/home/ed/petsc-next/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 348
/home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
[0]PETSC ERROR: [0] SNESSolve line 3799
/home/ed/petsc-next/src/snes/interface/snes.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT
Date: 2015-03-10 22:35:11 -0500
[0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar
11 12:47:11 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-fblaslapack --download-mpich
--with-debugging=1
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
~/petsc-next/src/snes/examples/tutorials[next]$

******* maint branch

~/petsc-maint/src/snes/examples/tutorials[maint]$ make ex9
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make
rule.
/home/ed/petsc-maint/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0
-I/home/ed/petsc-maint/include -I/home/ed/petsc-maint/linux-c-dbg/include
 `pwd`/ex9.c
/home/ed/petsc-maint/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o
-Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib
-L/home/ed/petsc-maint/linux-c-dbg/lib  -lpetsc
-Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lflapack -lfblas -lX11
-lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath
-lm -lmpichcxx -lstdc++ -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib
-L/home/ed/petsc-maint/linux-c-dbg/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -ldl
-Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lmpich -lopa -lmpl -lrt
-lpthread -lgcc_s -ldl
/bin/rm -f ex9.o
~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9
setup done: square       side length = 4.000
            grid               Mx,My = 11,11
            spacing            dx,dy = 0.400,0.400
number of Newton iterations = 3; result = CONVERGED_FNORM_RELATIVE
errors:    av |u-uexact|  = 7.156e-03
           |u-uexact|_inf = 2.550e-02
~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type svd
setup done: square       side length = 4.000
            grid               Mx,My = 11,11
            spacing            dx,dy = 0.400,0.400
number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
errors:    av |u-uexact|  = 7.156e-03
           |u-uexact|_inf = 2.550e-02
~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type none
setup done: square       side length = 4.000
            grid               Mx,My = 11,11
            spacing            dx,dy = 0.400,0.400
number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
errors:    av |u-uexact|  = 7.156e-03
           |u-uexact|_inf = 2.550e-02


-- 
Ed Bueler
Dept of Math and Stat and Geophysical Institute
University of Alaska Fairbanks
Fairbanks, AK 99775-6660
301C Chapman and 410D Elvey
907 474-7693 and 907 474-7199  (fax 907 474-5394)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/5aeeb875/attachment.html>

From song.gao2 at mail.mcgill.ca  Wed Mar 11 14:33:24 2015
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Wed, 11 Mar 2015 15:33:24 -0400
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
Message-ID: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>

Hello,

I'm solving Navier-Stokes equations by finite element method.  I use KSP as
the linear system solver. I run with 2 cpu. The code runs fine in non-debug
version. But when I switch to the debug version, the code gives the
following error.

I output the matrix and rhs before calling KSPSolve to make sure no NAN or
INF in them. The condition number of matrix is ~2e4.  Seems okay.

I also run the code with valgrind, but didn't find any other errors. The
valgrind output is attached. Any ideas of what I can do next? Thanks in
advance.


[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
CDT 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR:
/home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
[0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
[0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
[0]PETSC ERROR: Configure options
--with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
--with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
[0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPGMRESCycle() line 206 in src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/eead4b97/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: valgrindout
Type: application/octet-stream
Size: 9775 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/eead4b97/attachment-0001.obj>

From bsmith at mcs.anl.gov  Wed Mar 11 14:35:47 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 14:35:47 -0500
Subject: [petsc-users] make check error
In-Reply-To: <3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
	<alpine.LFD.2.11.1503111340210.27891@asterix>
	<3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
Message-ID: <373DD4E0-7B14-4681-8C6C-47E964ADA53E@mcs.anl.gov>


> On Mar 11, 2015, at 1:50 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> 
>> On Mar 11, 2015, at 1:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>> 
>> Actually mumps is being used - so fortran is required [by mumps]
>> 
>> The reason for the 'multiple' libgfortran messages is: openmpi is
>> compiled with an older version of gcc/gfortran
>> 
> 
> You are right about openmpi being build with an older version of the compiler. I am trying to mix and match codes.

   This is not a good idea. It is really really difficult for us to help you get things working when you have mixed and matched codes.

  Barry

> 
>> [and I suspect its harmless - as the binaries are created]
>> 
>> The primary issue to check: is this openmpi install functional - if so
>> how do you run mpi binaries? Perhaps its a cluster - and have to go
>> through a scheduler.. [i.e make check is perhaps inappropriate for
>> this case]
> 
> This might be true. This is my first time working on this system, so I need to find out the details. 
> 
>> 
>> Satish
>> ----------
>> 
>> Python version:
>> 2.4.2 (#1, Jan 10 2008, 17:43:47)
>> [GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]
>> 
>> Executing: mpicc --version
>> stdout:
>> gcc (GCC) 4.8.3
>> 
>> Executing: mpicc -show
>> stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
>> 
>> Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
>> erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec
>> 
>> 
>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>> 
>> 
>> 
>> On Wed, 11 Mar 2015, Barry Smith wrote:
>> 
>>> 
>>> Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
>>> 
>>> I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>> 
>>>> I apologize for not sending these files along. Here they are for the two machines. 
>>>> 
>>>> Really appreciate your help. 
>>>> 
>>>> -Manav
>>>> 
>>>> 
>>>>> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>> 
>>>>>> Greetings! 
>>>>>> 
>>>>>> I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>>>>>> 
>>>>>> Talon-login[114] bhatia$ make check
>>>>>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>> Running test examples to verify correct installation
>>>>>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> *******************Error detected during compile or link!*******************
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>>>>>> *********************************************************************************
>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>>>>>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /bin/rm -f ex19.o
>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> [Talon-login:18281] *** Process received signal ***
>>>>>> [Talon-login:18281] Signal: Segmentation fault (11)
>>>>>> [Talon-login:18281] Signal code: Address not mapped (1)
>>>>>> [Talon-login:18281] Failing at address: 0x100002990
>>>>>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>>>>>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>>>>>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>>>>>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>>>>>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>>>>>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>>>>>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>>>>>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>>>>>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>>>>>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>>>>>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>>>>>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>>>>>> [Talon-login:18281] [14] ./ex19 [0x482639]
>>>>>> [Talon-login:18281] *** End of error message ***
>>>>>> --------------------------------------------------------------------------
>>>>>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>> --------------------------------------------------------------------------
>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> [Talon-login:18286] *** Process received signal ***
>>>>>> [Talon-login:18286] Signal: Segmentation fault (11)
>>>>>> [Talon-login:18286] Signal code: Address not mapped (1)
>>>>>> [Talon-login:18286] Failing at address: 0x100002990
>>>>>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>>>>>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>>>>>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>>>>>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>>>>>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>>>>>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>>>>>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>>>>>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>>>>>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>>>>>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>>>>>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>>>>>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>>>>>> [Talon-login:18286] [14] ./ex19 [0x482639]
>>>>>> [Talon-login:18286] *** End of error message ***
>>>>>> --------------------------------------------------------------------------
>>>>>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>> --------------------------------------------------------------------------
>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> *******************Error detected during compile or link!*******************
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>>>>>> *********************************************************
>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>>>>>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>> /bin/rm -f ex5f.o
>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>  This will severely limit memory registrations.
>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>> 
>>>>>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>>>>>> 
>>>>>> Backtrace for this error:
>>>>>> #0  0x2B6CD4A6F4D7
>>>>>> #1  0x2B6CD4A6FADE
>>>>>> #2  0x2B6CD5805E1F
>>>>>> #3  0x2B6CD716F4CE
>>>>>> #4  0x2B6CD716E49D
>>>>>> #5  0x2B6CD7280159
>>>>>> #6  0x2B6CD46E5165
>>>>>> #7  0x2B6CD6E59820
>>>>>> #8  0x2B6CD46E4971
>>>>>> #9  0x2B6CD6C41C8E
>>>>>> #10  0x2B6CD46EE742
>>>>>> #11  0x2B6CD46B204F
>>>>>> #12  0x2B6CD46CF99F
>>>>>> #13  0x2B6CD494AA84
>>>>>> #14  0x489A6D in petscinitialize_
>>>>>> #15  0x487CEC in MAIN__ at ex5f.F:?
>>>>>> --------------------------------------------------------------------------
>>>>>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>> --------------------------------------------------------------------------
>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>> Completed test examples
>>>>>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>> =========================================
>>>>>> Now to evaluate the computer systems you plan use - do:
>>>>>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>>>>>> Talon-login[115] bhatia$ 
>>>>>> 
>>>>>> 
>>>>>> Thanks,
>>>>>> Manav
>>>>>> 
>>>>> 
>>>> 
>>>> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
>>> 
>>> 
> 


From bhatiamanav at gmail.com  Wed Mar 11 14:42:52 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 11 Mar 2015 14:42:52 -0500
Subject: [petsc-users] make check error
In-Reply-To: <373DD4E0-7B14-4681-8C6C-47E964ADA53E@mcs.anl.gov>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
	<alpine.LFD.2.11.1503111340210.27891@asterix>
	<3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
	<373DD4E0-7B14-4681-8C6C-47E964ADA53E@mcs.anl.gov>
Message-ID: <84749FA2-9644-4CCB-A794-FD01A4D22E02@gmail.com>

Agreed. 
I am attempting to recompile with a single compiler to maintain consistency. 

Hopefully, I will be able to make more progress. 

-Manav


> On Mar 11, 2015, at 2:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Mar 11, 2015, at 1:50 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> 
>>> On Mar 11, 2015, at 1:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> Actually mumps is being used - so fortran is required [by mumps]
>>> 
>>> The reason for the 'multiple' libgfortran messages is: openmpi is
>>> compiled with an older version of gcc/gfortran
>>> 
>> 
>> You are right about openmpi being build with an older version of the compiler. I am trying to mix and match codes.
> 
>   This is not a good idea. It is really really difficult for us to help you get things working when you have mixed and matched codes.
> 
>  Barry
> 
>> 
>>> [and I suspect its harmless - as the binaries are created]
>>> 
>>> The primary issue to check: is this openmpi install functional - if so
>>> how do you run mpi binaries? Perhaps its a cluster - and have to go
>>> through a scheduler.. [i.e make check is perhaps inappropriate for
>>> this case]
>> 
>> This might be true. This is my first time working on this system, so I need to find out the details. 
>> 
>>> 
>>> Satish
>>> ----------
>>> 
>>> Python version:
>>> 2.4.2 (#1, Jan 10 2008, 17:43:47)
>>> [GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]
>>> 
>>> Executing: mpicc --version
>>> stdout:
>>> gcc (GCC) 4.8.3
>>> 
>>> Executing: mpicc -show
>>> stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
>>> 
>>> Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
>>> erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec
>>> 
>>> 
>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>> 
>>> 
>>> 
>>> On Wed, 11 Mar 2015, Barry Smith wrote:
>>> 
>>>> 
>>>> Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
>>>> 
>>>> I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>> 
>>>>> I apologize for not sending these files along. Here they are for the two machines. 
>>>>> 
>>>>> Really appreciate your help. 
>>>>> 
>>>>> -Manav
>>>>> 
>>>>> 
>>>>>> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>> 
>>>>>>> Greetings! 
>>>>>>> 
>>>>>>> I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>>>>>>> 
>>>>>>> Talon-login[114] bhatia$ make check
>>>>>>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>>> Running test examples to verify correct installation
>>>>>>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> *******************Error detected during compile or link!*******************
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>>>>>>> *********************************************************************************
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>>>>>>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /bin/rm -f ex19.o
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> [Talon-login:18281] *** Process received signal ***
>>>>>>> [Talon-login:18281] Signal: Segmentation fault (11)
>>>>>>> [Talon-login:18281] Signal code: Address not mapped (1)
>>>>>>> [Talon-login:18281] Failing at address: 0x100002990
>>>>>>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>>>>>>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>>>>>>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>>>>>>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>>>>>>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>>>>>>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>>>>>>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>>>>>>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>>>>>>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>>>>>>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>>>>>>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>>>>>>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>>>>>>> [Talon-login:18281] [14] ./ex19 [0x482639]
>>>>>>> [Talon-login:18281] *** End of error message ***
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> [Talon-login:18286] *** Process received signal ***
>>>>>>> [Talon-login:18286] Signal: Segmentation fault (11)
>>>>>>> [Talon-login:18286] Signal code: Address not mapped (1)
>>>>>>> [Talon-login:18286] Failing at address: 0x100002990
>>>>>>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>>>>>>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>>>>>>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>>>>>>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>>>>>>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>>>>>>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>>>>>>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>>>>>>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>>>>>>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>>>>>>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>>>>>>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>>>>>>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>>>>>>> [Talon-login:18286] [14] ./ex19 [0x482639]
>>>>>>> [Talon-login:18286] *** End of error message ***
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> *******************Error detected during compile or link!*******************
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>>>>>>> *********************************************************
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>>>>>>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /bin/rm -f ex5f.o
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> 
>>>>>>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>>>>>>> 
>>>>>>> Backtrace for this error:
>>>>>>> #0  0x2B6CD4A6F4D7
>>>>>>> #1  0x2B6CD4A6FADE
>>>>>>> #2  0x2B6CD5805E1F
>>>>>>> #3  0x2B6CD716F4CE
>>>>>>> #4  0x2B6CD716E49D
>>>>>>> #5  0x2B6CD7280159
>>>>>>> #6  0x2B6CD46E5165
>>>>>>> #7  0x2B6CD6E59820
>>>>>>> #8  0x2B6CD46E4971
>>>>>>> #9  0x2B6CD6C41C8E
>>>>>>> #10  0x2B6CD46EE742
>>>>>>> #11  0x2B6CD46B204F
>>>>>>> #12  0x2B6CD46CF99F
>>>>>>> #13  0x2B6CD494AA84
>>>>>>> #14  0x489A6D in petscinitialize_
>>>>>>> #15  0x487CEC in MAIN__ at ex5f.F:?
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Completed test examples
>>>>>>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>>> =========================================
>>>>>>> Now to evaluate the computer systems you plan use - do:
>>>>>>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>>>>>>> Talon-login[115] bhatia$ 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Manav
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
>>>> 
>>>> 
>> 
> 


From knepley at gmail.com  Wed Mar 11 14:52:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 11 Mar 2015 14:52:39 -0500
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
Message-ID: <CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>

On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:

> Hello,
>
> I'm solving Navier-Stokes equations by finite element method.  I use KSP
> as the linear system solver. I run with 2 cpu. The code runs fine in
> non-debug version. But when I switch to the debug version, the code gives
> the following error.
>
> I output the matrix and rhs before calling KSPSolve to make sure no NAN or
> INF in them. The condition number of matrix is ~2e4.  Seems okay.
>
> I also run the code with valgrind, but didn't find any other errors. The
> valgrind output is attached. Any ideas of what I can do next?
>

Is there any chance you could spawn the debugger, -start_in_debugger, and
when you get the error,
print out the value of 'alpha' on each process?

Otherwise the best thing to do is output your Mat and RHS in binary and
send them so we can try to reproduce.

    Matt

  Thanks,

     Matt


> Thanks in advance.
>
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
> CDT 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR:
> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
> [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
> [0]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
> src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPGMRESCycle() line 206 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
> src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/474ff375/attachment.html>

From song.gao2 at mail.mcgill.ca  Wed Mar 11 15:39:30 2015
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Wed, 11 Mar 2015 16:39:30 -0400
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
Message-ID: <CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>

Thanks.

I run with two processes. When the code stop, I'm in      raise()     and
alpha is not in the current context.

(gdb)p alpha
No symbol "alpha" in current context.
(gdb)  bt
#0  0x0000003764432625 in raise () from /lib64/libc.so.6
#1  0x0000003764433e05 in abort () from /lib64/libc.so.6
#2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0,
line=1186,
    fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
    dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
    mess=0x7fff33ffa4c0 "Scalar value must be same on all processes,
argument # 3", ctx=0x0) at errabort.c:62
#3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
    func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
    dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
    mess=0x279c720 "Scalar value must be same on all processes, argument #
%d")
    at err.c:356
#4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
    x=0x3b33080) at rvector.c:1186
#5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
    vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345


But I break at VecMAXPY. then print out alpha on both processes. For the
first few times that the breakpoint is hit , I checked the values on both
processes and they are the same.


(gdb) b VecMAXPY
Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
(gdb) c
Continuing.
Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
    at rvector.c:1174
1174      PetscFunctionBegin;
(gdb) p alpha
$1 = (const PetscScalar *) 0x3f374e0
(gdb) p *alpha
$2 = -0.54285016977140765
(gdb)

2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:

> On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca>
> wrote:
>
>> Hello,
>>
>> I'm solving Navier-Stokes equations by finite element method.  I use KSP
>> as the linear system solver. I run with 2 cpu. The code runs fine in
>> non-debug version. But when I switch to the debug version, the code gives
>> the following error.
>>
>> I output the matrix and rhs before calling KSPSolve to make sure no NAN
>> or INF in them. The condition number of matrix is ~2e4.  Seems okay.
>>
>> I also run the code with valgrind, but didn't find any other errors. The
>> valgrind output is attached. Any ideas of what I can do next?
>>
>
> Is there any chance you could spawn the debugger, -start_in_debugger, and
> when you get the error,
> print out the value of 'alpha' on each process?
>
> Otherwise the best thing to do is output your Mat and RHS in binary and
> send them so we can try to reproduce.
>
>     Matt
>
>   Thanks,
>
>      Matt
>
>
>> Thanks in advance.
>>
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Invalid argument!
>> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
>> CDT 2013
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR:
>> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
>> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
>> [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
>> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
>> [0]PETSC ERROR: Configure options
>> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
>> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
>> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
>> src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: KSPGMRESCycle() line 206 in
>> src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
>> src/ksp/ksp/impls/gmres/gmres.c
>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/c8cb0585/attachment.html>

From song.gao2 at mail.mcgill.ca  Wed Mar 11 15:50:34 2015
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Wed, 11 Mar 2015 16:50:34 -0400
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
Message-ID: <CAJitgPUhh26FDCMB_c7SCe68fynez6MJALbxm+jSf=AFyFhkjw@mail.gmail.com>

And The matrix and rhs is attached. Thanks.

2015-03-11 16:39 GMT-04:00 Song Gao <song.gao2 at mail.mcgill.ca>:

> Thanks.
>
> I run with two processes. When the code stop, I'm in      raise()     and
> alpha is not in the current context.
>
> (gdb)p alpha
> No symbol "alpha" in current context.
> (gdb)  bt
> #0  0x0000003764432625 in raise () from /lib64/libc.so.6
> #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
> #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0,
> line=1186,
>     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes,
> argument # 3", ctx=0x0) at errabort.c:62
> #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
>     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>     mess=0x279c720 "Scalar value must be same on all processes, argument #
> %d")
>     at err.c:356
> #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
>     x=0x3b33080) at rvector.c:1186
> #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
>     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
>
>
> But I break at VecMAXPY. then print out alpha on both processes. For the
> first few times that the breakpoint is hit , I checked the values on both
> processes and they are the same.
>
>
> (gdb) b VecMAXPY
> Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
> (gdb) c
> Continuing.
> Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
>     at rvector.c:1174
> 1174      PetscFunctionBegin;
> (gdb) p alpha
> $1 = (const PetscScalar *) 0x3f374e0
> (gdb) p *alpha
> $2 = -0.54285016977140765
> (gdb)
>
> 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
>
> On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm solving Navier-Stokes equations by finite element method.  I use KSP
>>> as the linear system solver. I run with 2 cpu. The code runs fine in
>>> non-debug version. But when I switch to the debug version, the code gives
>>> the following error.
>>>
>>> I output the matrix and rhs before calling KSPSolve to make sure no NAN
>>> or INF in them. The condition number of matrix is ~2e4.  Seems okay.
>>>
>>> I also run the code with valgrind, but didn't find any other errors. The
>>> valgrind output is attached. Any ideas of what I can do next?
>>>
>>
>> Is there any chance you could spawn the debugger, -start_in_debugger, and
>> when you get the error,
>> print out the value of 'alpha' on each process?
>>
>> Otherwise the best thing to do is output your Mat and RHS in binary and
>> send them so we can try to reproduce.
>>
>>     Matt
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks in advance.
>>>
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Invalid argument!
>>> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
>>> 22:15:24 CDT 2013
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR:
>>> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
>>> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
>>> [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
>>> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
>>> [0]PETSC ERROR: Configure options
>>> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
>>> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPGMRESCycle() line 206 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/ada1f894/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mat.data
Type: application/octet-stream
Size: 301296 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/ada1f894/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rhs.data
Type: application/octet-stream
Size: 2568 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/ada1f894/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mat.data.info
Type: application/octet-stream
Size: 21 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/ada1f894/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rhs.data.info
Type: application/octet-stream
Size: 21 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/ada1f894/attachment-0007.obj>

From knepley at gmail.com  Wed Mar 11 16:35:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 11 Mar 2015 16:35:10 -0500
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
Message-ID: <CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>

On Wed, Mar 11, 2015 at 3:39 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:

> Thanks.
>
> I run with two processes. When the code stop, I'm in      raise()     and
> alpha is not in the current context.
>

Here you would use:

  (gdb) up 4

  (gdb) p *alpha

   Matt


> (gdb)p alpha
> No symbol "alpha" in current context.
> (gdb)  bt
> #0  0x0000003764432625 in raise () from /lib64/libc.so.6
> #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
> #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0,
> line=1186,
>     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes,
> argument # 3", ctx=0x0) at errabort.c:62
> #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
>     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>     mess=0x279c720 "Scalar value must be same on all processes, argument #
> %d")
>     at err.c:356
> #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
>     x=0x3b33080) at rvector.c:1186
> #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
>     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
>
>
> But I break at VecMAXPY. then print out alpha on both processes. For the
> first few times that the breakpoint is hit , I checked the values on both
> processes and they are the same.
>
>
> (gdb) b VecMAXPY
> Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
> (gdb) c
> Continuing.
> Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
>     at rvector.c:1174
> 1174      PetscFunctionBegin;
> (gdb) p alpha
> $1 = (const PetscScalar *) 0x3f374e0
> (gdb) p *alpha
> $2 = -0.54285016977140765
> (gdb)
>
> 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
>
>> On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm solving Navier-Stokes equations by finite element method.  I use KSP
>>> as the linear system solver. I run with 2 cpu. The code runs fine in
>>> non-debug version. But when I switch to the debug version, the code gives
>>> the following error.
>>>
>>> I output the matrix and rhs before calling KSPSolve to make sure no NAN
>>> or INF in them. The condition number of matrix is ~2e4.  Seems okay.
>>>
>>> I also run the code with valgrind, but didn't find any other errors. The
>>> valgrind output is attached. Any ideas of what I can do next?
>>>
>>
>> Is there any chance you could spawn the debugger, -start_in_debugger, and
>> when you get the error,
>> print out the value of 'alpha' on each process?
>>
>> Otherwise the best thing to do is output your Mat and RHS in binary and
>> send them so we can try to reproduce.
>>
>>     Matt
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks in advance.
>>>
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Invalid argument!
>>> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
>>> 22:15:24 CDT 2013
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR:
>>> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
>>> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
>>> [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
>>> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
>>> [0]PETSC ERROR: Configure options
>>> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
>>> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPGMRESCycle() line 206 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
>>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/edeb7e55/attachment.html>

From song.gao2 at mail.mcgill.ca  Wed Mar 11 19:05:05 2015
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Wed, 11 Mar 2015 20:05:05 -0400
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
	<CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>
Message-ID: <CAJitgPWiGJa3fDRh3VkGx64cE3aXDAsgK9nBgfpTdUcafBT41Q@mail.gmail.com>

Thank you. That's cool. Sorry, I'm not good at gdb.

I did that. They are the same. One gave
(gdb) p *alpha
$1 = 0.013169008988739605

and another gave
(gdb) p *alpha
$1 = 0.013169008988739673

2015-03-11 17:35 GMT-04:00 Matthew Knepley <knepley at gmail.com>:

> On Wed, Mar 11, 2015 at 3:39 PM, Song Gao <song.gao2 at mail.mcgill.ca>
> wrote:
>
>> Thanks.
>>
>> I run with two processes. When the code stop, I'm in      raise()     and
>> alpha is not in the current context.
>>
>
> Here you would use:
>
>   (gdb) up 4
>
>   (gdb) p *alpha
>
>    Matt
>
>
>> (gdb)p alpha
>> No symbol "alpha" in current context.
>> (gdb)  bt
>> #0  0x0000003764432625 in raise () from /lib64/libc.so.6
>> #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
>> #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0,
>> line=1186,
>>     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>>     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes,
>> argument # 3", ctx=0x0) at errabort.c:62
>> #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
>>     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
>>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
>>     mess=0x279c720 "Scalar value must be same on all processes, argument
>> # %d")
>>     at err.c:356
>> #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
>>     x=0x3b33080) at rvector.c:1186
>> #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
>>     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
>>
>>
>> But I break at VecMAXPY. then print out alpha on both processes. For the
>> first few times that the breakpoint is hit , I checked the values on both
>> processes and they are the same.
>>
>>
>> (gdb) b VecMAXPY
>> Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
>> (gdb) c
>> Continuing.
>> Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
>>     at rvector.c:1174
>> 1174      PetscFunctionBegin;
>> (gdb) p alpha
>> $1 = (const PetscScalar *) 0x3f374e0
>> (gdb) p *alpha
>> $2 = -0.54285016977140765
>> (gdb)
>>
>> 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
>>
>>> On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm solving Navier-Stokes equations by finite element method.  I use
>>>> KSP as the linear system solver. I run with 2 cpu. The code runs fine in
>>>> non-debug version. But when I switch to the debug version, the code gives
>>>> the following error.
>>>>
>>>> I output the matrix and rhs before calling KSPSolve to make sure no NAN
>>>> or INF in them. The condition number of matrix is ~2e4.  Seems okay.
>>>>
>>>> I also run the code with valgrind, but didn't find any other errors.
>>>> The valgrind output is attached. Any ideas of what I can do next?
>>>>
>>>
>>> Is there any chance you could spawn the debugger, -start_in_debugger,
>>> and when you get the error,
>>> print out the value of 'alpha' on each process?
>>>
>>> Otherwise the best thing to do is output your Mat and RHS in binary and
>>> send them so we can try to reproduce.
>>>
>>>     Matt
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks in advance.
>>>>
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [0]PETSC ERROR: Invalid argument!
>>>> [0]PETSC ERROR: Scalar value must be same on all processes, argument #
>>>> 3!
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
>>>> 22:15:24 CDT 2013
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR:
>>>> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
>>>> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /tmp/PETSC33/petsc-3.3-p7/linux/lib
>>>> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
>>>> [0]PETSC ERROR: Configure options
>>>> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
>>>> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
>>>> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>> [0]PETSC ERROR: KSPGMRESCycle() line 206 in
>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/44687300/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 11 19:16:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 19:16:17 -0500
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAJitgPWiGJa3fDRh3VkGx64cE3aXDAsgK9nBgfpTdUcafBT41Q@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
	<CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>
	<CAJitgPWiGJa3fDRh3VkGx64cE3aXDAsgK9nBgfpTdUcafBT41Q@mail.gmail.com>
Message-ID: <5ED00234-44E2-499C-821B-3D48BDAC7414@mcs.anl.gov>


  In IEEE floating point standard Nan is not equal to itself. This is what triggers these types of non-intuitive error messages.

  My guess is that the preconditioner is triggering a Nan; for example ILU sometimes triggers Nan due to very small pivots.

  You are using an old version of PETSc the first step I recommend is to upgrade http://www.mcs.anl.gov/petsc/documentation/changes/index.html; the more recent version of PETSc has more checks for Nan etc in the code.

  Barry



> On Mar 11, 2015, at 7:05 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> 
> Thank you. That's cool. Sorry, I'm not good at gdb.
> 
> I did that. They are the same. One gave
> (gdb) p *alpha
> $1 = 0.013169008988739605
> 
> and another gave
> (gdb) p *alpha
> $1 = 0.013169008988739673
> 
> 2015-03-11 17:35 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> On Wed, Mar 11, 2015 at 3:39 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> Thanks.
> 
> I run with two processes. When the code stop, I'm in      raise()     and alpha is not in the current context.
> 
> Here you would use:
> 
>   (gdb) up 4
> 
>   (gdb) p *alpha
> 
>    Matt
>  
> (gdb)p alpha
> No symbol "alpha" in current context.
> (gdb)  bt
> #0  0x0000003764432625 in raise () from /lib64/libc.so.6
> #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
> #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0, line=1186, 
>     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c", 
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL, 
>     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes, argument # 3", ctx=0x0) at errabort.c:62
> #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186, 
>     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c", 
>     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL, 
>     mess=0x279c720 "Scalar value must be same on all processes, argument # %d")
>     at err.c:356
> #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840, 
>     x=0x3b33080) at rvector.c:1186
> #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090, 
>     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
> 
> 
> But I break at VecMAXPY. then print out alpha on both processes. For the first few times that the breakpoint is hit , I checked the values on both processes and they are the same. 
> 
> 
> (gdb) b VecMAXPY
> Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
> (gdb) c
> Continuing.
> Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
>     at rvector.c:1174
> 1174      PetscFunctionBegin;
> (gdb) p alpha
> $1 = (const PetscScalar *) 0x3f374e0
> (gdb) p *alpha
> $2 = -0.54285016977140765
> (gdb) 
> 
> 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> Hello,
> 
> I'm solving Navier-Stokes equations by finite element method.  I use KSP as the linear system solver. I run with 2 cpu. The code runs fine in non-debug version. But when I switch to the debug version, the code gives the following error.
> 
> I output the matrix and rhs before calling KSPSolve to make sure no NAN or INF in them. The condition number of matrix is ~2e4.  Seems okay.
> 
> I also run the code with valgrind, but didn't find any other errors. The valgrind output is attached. Any ideas of what I can do next?
> 
> Is there any chance you could spawn the debugger, -start_in_debugger, and when you get the error,
> print out the value of 'alpha' on each process?
> 
> Otherwise the best thing to do is output your Mat and RHS in binary and send them so we can try to reproduce.
> 
>     Matt
> 
>   Thanks,
> 
>      Matt
>  
> Thanks in advance.
> 
> 
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
> [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
> [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
> [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/ --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPGMRESCycle() line 206 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From song.gao2 at mail.mcgill.ca  Wed Mar 11 20:35:39 2015
From: song.gao2 at mail.mcgill.ca (Song Gao)
Date: Wed, 11 Mar 2015 21:35:39 -0400
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <5ED00234-44E2-499C-821B-3D48BDAC7414@mcs.anl.gov>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
	<CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>
	<CAJitgPWiGJa3fDRh3VkGx64cE3aXDAsgK9nBgfpTdUcafBT41Q@mail.gmail.com>
	<5ED00234-44E2-499C-821B-3D48BDAC7414@mcs.anl.gov>
Message-ID: <CAJitgPVucXX_tLZX8-8tHb7knfaw_TB=PjJjdxpmxmvJUaK-Zg@mail.gmail.com>

Thanks for your suggestion.   Yes,  but sorry that unfortunately I'm not
allowed to upgrade it.

As for the preconditioner. I tried with -pc_type none and still have the
same error message.

Sorry, I am not clear. The code crashes
at  PetscValidLogicalCollectiveScalar(y,alpha[i],3) and I have seen that on
both processors  b2 = {-0.013169008988739605, 0.013169008988739673},
Forgive me if I ask a silly question. Is it because we compare two double
precision number with the operator '!=' ?    Looking at line 309,   Of
course, (-b2[0] != b2[1]) is true, right?

Thanks for your help.

I pasted PetscValidLogicalCollectiveScalar here:

303 #define PetscValidLogicalCollectiveScalar(a,b,c)
 \
304   do {
 \
305     PetscErrorCode _7_ierr;
\
306     PetscReal b1[2],b2[2];
 \
307     b1[0] = -PetscRealPart(b); b1[1] = PetscRealPart(b);
 \
308     _7_ierr =
MPI_Allreduce(b1,b2,2,MPIU_REAL,MPIU_MAX,((PetscObject)a)->comm);CHKERRQ(_7_ierr);
\
309     if (-b2[0] != b2[1])
SETERRQ1(((PetscObject)a)->comm,PETSC_ERR_ARG_WRONG,"Scalar value must be
same on all processes, argument # %d",c); \
310   } while (0)


Regards,
Song

2015-03-11 20:16 GMT-04:00 Barry Smith <bsmith at mcs.anl.gov>:

>
>   In IEEE floating point standard Nan is not equal to itself. This is what
> triggers these types of non-intuitive error messages.
>
>   My guess is that the preconditioner is triggering a Nan; for example ILU
> sometimes triggers Nan due to very small pivots.
>
>   You are using an old version of PETSc the first step I recommend is to
> upgrade http://www.mcs.anl.gov/petsc/documentation/changes/index.html;
> the more recent version of PETSc has more checks for Nan etc in the code.
>
>   Barry
>
>
>
> > On Mar 11, 2015, at 7:05 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> >
> > Thank you. That's cool. Sorry, I'm not good at gdb.
> >
> > I did that. They are the same. One gave
> > (gdb) p *alpha
> > $1 = 0.013169008988739605
> >
> > and another gave
> > (gdb) p *alpha
> > $1 = 0.013169008988739673
> >
> > 2015-03-11 17:35 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> > On Wed, Mar 11, 2015 at 3:39 PM, Song Gao <song.gao2 at mail.mcgill.ca>
> wrote:
> > Thanks.
> >
> > I run with two processes. When the code stop, I'm in      raise()
>  and alpha is not in the current context.
> >
> > Here you would use:
> >
> >   (gdb) up 4
> >
> >   (gdb) p *alpha
> >
> >    Matt
> >
> > (gdb)p alpha
> > No symbol "alpha" in current context.
> > (gdb)  bt
> > #0  0x0000003764432625 in raise () from /lib64/libc.so.6
> > #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
> > #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0,
> line=1186,
> >     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
> >     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
> >     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes,
> argument # 3", ctx=0x0) at errabort.c:62
> > #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
> >     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
> >     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
> >     mess=0x279c720 "Scalar value must be same on all processes, argument
> # %d")
> >     at err.c:356
> > #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
> >     x=0x3b33080) at rvector.c:1186
> > #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
> >     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
> >
> >
> > But I break at VecMAXPY. then print out alpha on both processes. For the
> first few times that the breakpoint is hit , I checked the values on both
> processes and they are the same.
> >
> >
> > (gdb) b VecMAXPY
> > Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
> > (gdb) c
> > Continuing.
> > Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
> >     at rvector.c:1174
> > 1174      PetscFunctionBegin;
> > (gdb) p alpha
> > $1 = (const PetscScalar *) 0x3f374e0
> > (gdb) p *alpha
> > $2 = -0.54285016977140765
> > (gdb)
> >
> > 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> > On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca>
> wrote:
> > Hello,
> >
> > I'm solving Navier-Stokes equations by finite element method.  I use KSP
> as the linear system solver. I run with 2 cpu. The code runs fine in
> non-debug version. But when I switch to the debug version, the code gives
> the following error.
> >
> > I output the matrix and rhs before calling KSPSolve to make sure no NAN
> or INF in them. The condition number of matrix is ~2e4.  Seems okay.
> >
> > I also run the code with valgrind, but didn't find any other errors. The
> valgrind output is attached. Any ideas of what I can do next?
> >
> > Is there any chance you could spawn the debugger, -start_in_debugger,
> and when you get the error,
> > print out the value of 'alpha' on each process?
> >
> > Otherwise the best thing to do is output your Mat and RHS in binary and
> send them so we can try to reproduce.
> >
> >     Matt
> >
> >   Thanks,
> >
> >      Matt
> >
> > Thanks in advance.
> >
> >
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Invalid argument!
> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
> 22:15:24 CDT 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR:
> /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG
> on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
> > [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
> > [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
> > [0]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/
> --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
> > [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in
> src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPGMRESCycle() line 206 in
> src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
> src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150311/acb921fe/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 11 22:22:14 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 11 Mar 2015 22:22:14 -0500
Subject: [petsc-users] The PETSC ERROR: VecMAXPY() when using GMRES
In-Reply-To: <CAJitgPVucXX_tLZX8-8tHb7knfaw_TB=PjJjdxpmxmvJUaK-Zg@mail.gmail.com>
References: <CAJitgPU85FH634KyJX9Pc_ERLBxvLpvxNLytoLzhCXorkfYvhA@mail.gmail.com>
	<CAMYG4GkbvTrZom=LtGPGRa-6GW=gapoQSC3X1uN0spOg2GqPNw@mail.gmail.com>
	<CAJitgPUR_RMtgmh73to8WsiqWw79+oi5gvj=RuUt8Fk2kbaE5g@mail.gmail.com>
	<CAMYG4GmDC_Sky64fmLiAqz2UKAQVJVftK4_Efq7PWaHe33_m4g@mail.gmail.com>
	<CAJitgPWiGJa3fDRh3VkGx64cE3aXDAsgK9nBgfpTdUcafBT41Q@mail.gmail.com>
	<5ED00234-44E2-499C-821B-3D48BDAC7414@mcs.anl.gov>
	<CAJitgPVucXX_tLZX8-8tHb7knfaw_TB=PjJjdxpmxmvJUaK-Zg@mail.gmail.com>
Message-ID: <DB440CA5-9B12-4420-B1C0-2747B7D27493@mcs.anl.gov>


  Interesting, I was wrong it has nothing to do with Nan. Either the MPI_Allreduce() in the sum or in max (for the check of consistency) is producing ever slightly different numbers thus triggering our error (though it appears no true error has occurred). I have never seen this before. We may have to rethink our consistency checker in the code.

  You can force the debugging version of PETSc to not do the consistency checking by editing include/petsc-private/petscimpl.h and locating the lines 

#if !defined(PETSC_USE_DEBUG)

#define PetscCheckSameType(a,arga,b,argb) do {} while (0)

and changing it to start with 

#if 1

then run make in the PETSc root directory and relink your program. With this change the debug version should not produce the error message you have been seeing.

  Has anyone else ever seen anything like this?

  Barry



> On Mar 11, 2015, at 8:35 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> 
> Thanks for your suggestion.   Yes,  but sorry that unfortunately I'm not allowed to upgrade it.  
> 
> As for the preconditioner. I tried with -pc_type none and still have the same error message.
> 
> Sorry, I am not clear. The code crashes at  PetscValidLogicalCollectiveScalar(y,alpha[i],3) and I have seen that on both processors  b2 = {-0.013169008988739605, 0.013169008988739673}, Forgive me if I ask a silly question. Is it because we compare two double precision number with the operator '!=' ?    Looking at line 309,   Of course, (-b2[0] != b2[1]) is true, right? 
> 
> Thanks for your help.
> 
> I pasted PetscValidLogicalCollectiveScalar here:
> 
> 303 #define PetscValidLogicalCollectiveScalar(a,b,c)                        \
> 304   do {                                                                  \
> 305     PetscErrorCode _7_ierr;                                             \
> 306     PetscReal b1[2],b2[2];                                              \
> 307     b1[0] = -PetscRealPart(b); b1[1] = PetscRealPart(b);                \
> 308     _7_ierr = MPI_Allreduce(b1,b2,2,MPIU_REAL,MPIU_MAX,((PetscObject)a)->comm);CHKERRQ(_7_ierr); \
> 309     if (-b2[0] != b2[1]) SETERRQ1(((PetscObject)a)->comm,PETSC_ERR_ARG_WRONG,"Scalar value must be same on all processes, argument # %d",c); \
> 310   } while (0)
> 
> 
> Regards,
> Song 
> 
> 2015-03-11 20:16 GMT-04:00 Barry Smith <bsmith at mcs.anl.gov>:
> 
>   In IEEE floating point standard Nan is not equal to itself. This is what triggers these types of non-intuitive error messages.
> 
>   My guess is that the preconditioner is triggering a Nan; for example ILU sometimes triggers Nan due to very small pivots.
> 
>   You are using an old version of PETSc the first step I recommend is to upgrade http://www.mcs.anl.gov/petsc/documentation/changes/index.html; the more recent version of PETSc has more checks for Nan etc in the code.
> 
>   Barry
> 
> 
> 
> > On Mar 11, 2015, at 7:05 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> >
> > Thank you. That's cool. Sorry, I'm not good at gdb.
> >
> > I did that. They are the same. One gave
> > (gdb) p *alpha
> > $1 = 0.013169008988739605
> >
> > and another gave
> > (gdb) p *alpha
> > $1 = 0.013169008988739673
> >
> > 2015-03-11 17:35 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> > On Wed, Mar 11, 2015 at 3:39 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> > Thanks.
> >
> > I run with two processes. When the code stop, I'm in      raise()     and alpha is not in the current context.
> >
> > Here you would use:
> >
> >   (gdb) up 4
> >
> >   (gdb) p *alpha
> >
> >    Matt
> >
> > (gdb)p alpha
> > No symbol "alpha" in current context.
> > (gdb)  bt
> > #0  0x0000003764432625 in raise () from /lib64/libc.so.6
> > #1  0x0000003764433e05 in abort () from /lib64/libc.so.6
> > #2  0x00000000015d02f5 in PetscAbortErrorHandler (comm=0x36e17a0, line=1186,
> >     fun=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
> >     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
> >     mess=0x7fff33ffa4c0 "Scalar value must be same on all processes, argument # 3", ctx=0x0) at errabort.c:62
> > #3  0x000000000130cf44 in PetscError (comm=0x36e17a0, line=1186,
> >     func=0x279cad4 "VecMAXPY", file=0x279c404 "rvector.c",
> >     dir=0x279c1c8 "src/vec/vec/interface/", n=62, p=PETSC_ERROR_INITIAL,
> >     mess=0x279c720 "Scalar value must be same on all processes, argument # %d")
> >     at err.c:356
> > #4  0x00000000013f8184 in VecMAXPY (y=0x3b35000, nv=20, alpha=0x3b31840,
> >     x=0x3b33080) at rvector.c:1186
> > #5  0x0000000001581062 in KSPGMRESBuildSoln (nrs=0x3b31840, vs=0x3ab2090,
> >     vdest=0x3ab2090, ksp=0x39a9700, it=19) at gmres.c:345
> >
> >
> > But I break at VecMAXPY. then print out alpha on both processes. For the first few times that the breakpoint is hit , I checked the values on both processes and they are the same.
> >
> >
> > (gdb) b VecMAXPY
> > Breakpoint 1 at 0x13f73e0: file rvector.c, line 1174.
> > (gdb) c
> > Continuing.
> > Breakpoint 1, VecMAXPY (y=0x3f2b790, nv=1, alpha=0x3f374e0, x=0x3f1fde0)
> >     at rvector.c:1174
> > 1174      PetscFunctionBegin;
> > (gdb) p alpha
> > $1 = (const PetscScalar *) 0x3f374e0
> > (gdb) p *alpha
> > $2 = -0.54285016977140765
> > (gdb)
> >
> > 2015-03-11 15:52 GMT-04:00 Matthew Knepley <knepley at gmail.com>:
> > On Wed, Mar 11, 2015 at 2:33 PM, Song Gao <song.gao2 at mail.mcgill.ca> wrote:
> > Hello,
> >
> > I'm solving Navier-Stokes equations by finite element method.  I use KSP as the linear system solver. I run with 2 cpu. The code runs fine in non-debug version. But when I switch to the debug version, the code gives the following error.
> >
> > I output the matrix and rhs before calling KSPSolve to make sure no NAN or INF in them. The condition number of matrix is ~2e4.  Seems okay.
> >
> > I also run the code with valgrind, but didn't find any other errors. The valgrind output is attached. Any ideas of what I can do next?
> >
> > Is there any chance you could spawn the debugger, -start_in_debugger, and when you get the error,
> > print out the value of 'alpha' on each process?
> >
> > Otherwise the best thing to do is output your Mat and RHS in binary and send them so we can try to reproduce.
> >
> >     Matt
> >
> >   Thanks,
> >
> >      Matt
> >
> > Thanks in advance.
> >
> >
> > [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Invalid argument!
> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: /home/cfd/sgao/mycodes/fensap_new_edge_coefficient/fensapng-mf-newmuscl-overledg_org/bin/fensapMPI_LINUX64_DEBUG on a linux named anakin by sgao Wed Mar 11 15:07:53 2015
> > [0]PETSC ERROR: Libraries linked from /tmp/PETSC33/petsc-3.3-p7/linux/lib
> > [0]PETSC ERROR: Configure run at Wed Jan 15 12:04:54 2014
> > [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local.linux64/lib64/MPI-openmpi-1.4.5/ --with-shared-libraries=0 --COPTFLAGS=-g --FOPTFLAGS=-g --with-debugging=yes
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: VecMAXPY() line 1186 in src/vec/vec/interface/rvector.c
> > [0]PETSC ERROR: KSPGMRESBuildSoln() line 345 in src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPGMRESCycle() line 206 in src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> 
> 


From bhatiamanav at gmail.com  Thu Mar 12 10:46:11 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 12 Mar 2015 10:46:11 -0500
Subject: [petsc-users] make check error
In-Reply-To: <373DD4E0-7B14-4681-8C6C-47E964ADA53E@mcs.anl.gov>
References: <6101D03D-00B4-4E2F-A494-710691CCFCF9@gmail.com>
	<D3AE9310-6087-4359-9F11-43C9C94C828A@mcs.anl.gov>
	<1A255E6A-2124-485B-99AA-C8F1C33A7169@gmail.com>
	<1C442284-88C3-425F-9A7B-311CC8C42F57@mcs.anl.gov>
	<alpine.LFD.2.11.1503111340210.27891@asterix>
	<3DDBEB71-5A5B-49F7-B86C-F8B931581AE2@gmail.com>
	<373DD4E0-7B14-4681-8C6C-47E964ADA53E@mcs.anl.gov>
Message-ID: <69F6CE2A-87E6-4BF8-8581-24EAF8584EA7@gmail.com>

I got the code to work with a consistent combination of mpi and gcc compilers. 

Thanks for your help!

-Manav

> On Mar 11, 2015, at 2:35 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Mar 11, 2015, at 1:50 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> 
>>> On Mar 11, 2015, at 1:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> Actually mumps is being used - so fortran is required [by mumps]
>>> 
>>> The reason for the 'multiple' libgfortran messages is: openmpi is
>>> compiled with an older version of gcc/gfortran
>>> 
>> 
>> You are right about openmpi being build with an older version of the compiler. I am trying to mix and match codes.
> 
>   This is not a good idea. It is really really difficult for us to help you get things working when you have mixed and matched codes.
> 
>  Barry
> 
>> 
>>> [and I suspect its harmless - as the binaries are created]
>>> 
>>> The primary issue to check: is this openmpi install functional - if so
>>> how do you run mpi binaries? Perhaps its a cluster - and have to go
>>> through a scheduler.. [i.e make check is perhaps inappropriate for
>>> this case]
>> 
>> This might be true. This is my first time working on this system, so I need to find out the details. 
>> 
>>> 
>>> Satish
>>> ----------
>>> 
>>> Python version:
>>> 2.4.2 (#1, Jan 10 2008, 17:43:47)
>>> [GCC 4.1.2 20070115 (prerelease) (SUSE Linux)]
>>> 
>>> Executing: mpicc --version
>>> stdout:
>>> gcc (GCC) 4.8.3
>>> 
>>> Executing: mpicc -show
>>> stdout: gcc -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
>>> 
>>> Driving: /usr/local/gnu/gcc-4.8.3/bin/gfortran -o /tmp/petsc-X1bXQ9/config.compilers/conftest -v -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O /tmp/petsc-X1bXQ9/config.compilers/conftest.o -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/s\
>>> erver/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -pthread -I/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -l gfortran -lm -ldl -l gfortran -l m -shared-libgcc -specs=/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64/libgfortran.spec
>>> 
>>> 
>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi-1.4.2/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>> 
>>> 
>>> 
>>> On Wed, 11 Mar 2015, Barry Smith wrote:
>>> 
>>>> 
>>>> Since you are providing mpicc and mpicxx do not provide the --with-mpi-include= --with-mpi-lib=
>>>> 
>>>> I notice you have --with-fortran=0  if you do not want fortran than use --with-fc=0 and do not provide --FC and this may solve the gfortran issue
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On Mar 11, 2015, at 1:24 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>> 
>>>>> I apologize for not sending these files along. Here they are for the two machines. 
>>>>> 
>>>>> Really appreciate your help. 
>>>>> 
>>>>> -Manav
>>>>> 
>>>>> 
>>>>>> On Mar 11, 2015, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> Lots of problems with your system but the biggest is that it appears you have two gfortran installs that interfere with each other. If you had followed the instructions and sent configure.log and make.log we could help you more.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>>> On Mar 11, 2015, at 1:12 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>> 
>>>>>>> Greetings! 
>>>>>>> 
>>>>>>> I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. 
>>>>>>> 
>>>>>>> Talon-login[114] bhatia$ make check
>>>>>>> gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>>> Running test examples to verify correct installation
>>>>>>> Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> *******************Error detected during compile or link!*******************
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19
>>>>>>> *********************************************************************************
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O      -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    `pwd`/ex19.c
>>>>>>> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19  ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /bin/rm -f ex19.o
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> [Talon-login:18281] *** Process received signal ***
>>>>>>> [Talon-login:18281] Signal: Segmentation fault (11)
>>>>>>> [Talon-login:18281] Signal code: Address not mapped (1)
>>>>>>> [Talon-login:18281] Failing at address: 0x100002990
>>>>>>> [Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]
>>>>>>> [Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]
>>>>>>> [Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]
>>>>>>> [Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]
>>>>>>> [Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]
>>>>>>> [Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]
>>>>>>> [Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]
>>>>>>> [Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]
>>>>>>> [Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]
>>>>>>> [Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]
>>>>>>> [Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]
>>>>>>> [Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>>> [Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>>> [Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]
>>>>>>> [Talon-login:18281] [14] ./ex19 [0x482639]
>>>>>>> [Talon-login:18281] *** End of error message ***
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18281 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> [Talon-login:18286] *** Process received signal ***
>>>>>>> [Talon-login:18286] Signal: Segmentation fault (11)
>>>>>>> [Talon-login:18286] Signal code: Address not mapped (1)
>>>>>>> [Talon-login:18286] Failing at address: 0x100002990
>>>>>>> [Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]
>>>>>>> [Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]
>>>>>>> [Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]
>>>>>>> [Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]
>>>>>>> [Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]
>>>>>>> [Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]
>>>>>>> [Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]
>>>>>>> [Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]
>>>>>>> [Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]
>>>>>>> [Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]
>>>>>>> [Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]
>>>>>>> [Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]
>>>>>>> [Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]
>>>>>>> [Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]
>>>>>>> [Talon-login:18286] [14] ./ex19 [0x482639]
>>>>>>> [Talon-login:18286] *** End of error message ***
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18286 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> *******************Error detected during compile or link!*******************
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> /cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f
>>>>>>> *********************************************************
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> mpif90 -c  -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include    -o ex5f.o ex5f.F
>>>>>>> mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O   -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib  -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4..8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl  
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3
>>>>>>> /bin/rm -f ex5f.o
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process
>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
>>>>>>> CMA: unable to open /dev/infiniband/rdma_cm
>>>>>>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>>>>>> This will severely limit memory registrations.
>>>>>>> libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
>>>>>>> 
>>>>>>> Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
>>>>>>> 
>>>>>>> Backtrace for this error:
>>>>>>> #0  0x2B6CD4A6F4D7
>>>>>>> #1  0x2B6CD4A6FADE
>>>>>>> #2  0x2B6CD5805E1F
>>>>>>> #3  0x2B6CD716F4CE
>>>>>>> #4  0x2B6CD716E49D
>>>>>>> #5  0x2B6CD7280159
>>>>>>> #6  0x2B6CD46E5165
>>>>>>> #7  0x2B6CD6E59820
>>>>>>> #8  0x2B6CD46E4971
>>>>>>> #9  0x2B6CD6C41C8E
>>>>>>> #10  0x2B6CD46EE742
>>>>>>> #11  0x2B6CD46B204F
>>>>>>> #12  0x2B6CD46CF99F
>>>>>>> #13  0x2B6CD494AA84
>>>>>>> #14  0x489A6D in petscinitialize_
>>>>>>> #15  0x487CEC in MAIN__ at ex5f.F:?
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpirun noticed that process rank 0 with PID 18354 on node Talon-login.HPC.MsState.Edu exited on signal 11 (Segmentation fault).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'
>>>>>>> Completed test examples
>>>>>>> gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'
>>>>>>> =========================================
>>>>>>> Now to evaluate the computer systems you plan use - do:
>>>>>>> make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use>
>>>>>>> Talon-login[115] bhatia$ 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Manav
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> <configure_shadow.log><configure_talon.log><make_shadow.log><make_talon.log>
>>>> 
>>>> 
>> 
> 


From gtheler at cites-gss.com  Thu Mar 12 12:49:36 2015
From: gtheler at cites-gss.com (Theler German Guillermo)
Date: Thu, 12 Mar 2015 17:49:36 +0000
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <94C36ABC-BFF9-4D90-9529-1567DB569DB4@mcs.anl.gov>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
	<1426007602.2659.25.camel@cites-gss.com>
	<E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>
	<1426013835.2659.37.camel@cites-gss.com>
	<94C36ABC-BFF9-4D90-9529-1567DB569DB4@mcs.anl.gov>
Message-ID: <1426182528.20542.10.camel@cites-gss.com>


On Wed, 2015-03-11 at 13:47 -0500, Barry Smith wrote:
>    Thanks for the suggestions and observations. I have made a new branch barry/improve-afterimage


Works like a charm! But note that the "reserved" line
in your definition of PetscAfterImageFormats in xinit.c:305 the
"reserved" follows mi previous e-mail, that was originally quoted from
libAfterImage's documentation. That line is a comment. I think it should
match the definitions that start at line 133 of the file import.c in
libAfterImage source
(https://github.com/sashavasko/afterstep-devel/blob/master/libAfterImage/import.c)

> that allows one to set the file format saved by passing it in the filename. On

Nice! Remember to include in the PETSc documentation the fact that
three-letter extension ".jpg" does not work and that the user should
pass ".jpeg" as extension.

> my Mac with the --download-afterimage only .Gif and .Jpeg formats are supported (by Afterimage) but that will depend on your install of Afterimage I guess (Png gives me the same error message you get).


My configure script complains about --download-afterimage:

gtheler at blinky:~/libs/petsc-afterimage$ ./configure
--download-afterimage
===============================================================================
             Configuring PETSc to compile on your
system
===============================================================================
TESTING: alternateConfigureLibrary from
config.packages.Triangle(config/BuildSystem/config/package.py:640)
*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
for details):
-------------------------------------------------------------------------------
External package afterimage does not support --download-afterimage
*******************************************************************************

gtheler at blinky:~/libs/petsc-afterimage$



The zlib error is raised in libpng12's in pngwutil.c:557 because a call
to zlib's deflateInit2 does not return Z_OK. I will investigate this
issue in more detail.


Thanks for your time
--
jeremy
________________________________
 Imprima este mensaje s?lo si es absolutamente necesario.
Para imprimir, en lo posible utilice el papel de ambos lados.
El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.



************AVISO DE CONFIDENCIALIDAD************

El Grupo Sancor Seguros comunica que:

Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.

Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.

From bsmith at mcs.anl.gov  Thu Mar 12 13:52:32 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 12 Mar 2015 13:52:32 -0500
Subject: [petsc-users] [Fwd: Re:  saving views as images]
In-Reply-To: <1426182528.20542.10.camel@cites-gss.com>
References: <1426002378.2659.21.camel@cites-gss.com>
	<78F6A43A-7BC7-4899-B20A-FAB5FBDF5E49@mcs.anl.gov>
	<1426007602.2659.25.camel@cites-gss.com>
	<E979E457-F1EE-4EBD-A172-876E6A2D1869@mcs.anl.gov>
	<1426013835.2659.37.camel@cites-gss.com>
	<94C36ABC-BFF9-4D90-9529-1567DB569DB4@mcs.anl.gov>
	<1426182528.20542.10.camel@cites-gss.com>
Message-ID: <352EBA12-DFE0-4372-9551-0011919C252A@mcs.anl.gov>


> On Mar 12, 2015, at 12:49 PM, Theler German Guillermo <gtheler at cites-gss.com> wrote:
> 
> 
> On Wed, 2015-03-11 at 13:47 -0500, Barry Smith wrote:
>>   Thanks for the suggestions and observations. I have made a new branch barry/improve-afterimage
> 
> 
> Works like a charm! But note that the "reserved" line
> in your definition of PetscAfterImageFormats in xinit.c:305 the
> "reserved" follows mi previous e-mail, that was originally quoted from
> libAfterImage's documentation. That line is a comment. I think it should
> match the definitions that start at line 133 of the file import.c in
> libAfterImage source
> (https://github.com/sashavasko/afterstep-devel/blob/master/libAfterImage/import.c)

  Fixed

> 
>> that allows one to set the file format saved by passing it in the filename. On
> 
> Nice! Remember to include in the PETSc documentation the fact that
> three-letter extension ".jpg" does not work and that the user should
> pass ".jpeg" as extension.

  Added .jpg also as a supported type. When possible better to just support what the user might try then to document that it won't work :-)

  Thanks for your feedback

Barry

> 
>> my Mac with the --download-afterimage only .Gif and .Jpeg formats are supported (by Afterimage) but that will depend on your install of Afterimage I guess (Png gives me the same error message you get).
> 
> 
> My configure script complains about --download-afterimage:
> 
> gtheler at blinky:~/libs/petsc-afterimage$ ./configure
> --download-afterimage
> ===============================================================================
>             Configuring PETSc to compile on your
> system
> ===============================================================================
> TESTING: alternateConfigureLibrary from
> config.packages.Triangle(config/BuildSystem/config/package.py:640)
> *******************************************************************************
>         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> for details):
> -------------------------------------------------------------------------------
> External package afterimage does not support --download-afterimage
> *******************************************************************************
> 
> gtheler at blinky:~/libs/petsc-afterimage$
> 
> 
> 
> The zlib error is raised in libpng12's in pngwutil.c:557 because a call
> to zlib's deflateInit2 does not return Z_OK. I will investigate this
> issue in more detail.
> 
> 
> Thanks for your time
> --
> jeremy
> ________________________________
> Imprima este mensaje s?lo si es absolutamente necesario.
> Para imprimir, en lo posible utilice el papel de ambos lados.
> El Grupo Sancor Seguros se compromete con el cuidado del medioambiente.
> 
> 
> 
> ************AVISO DE CONFIDENCIALIDAD************
> 
> El Grupo Sancor Seguros comunica que:
> 
> Este mensaje y todos los archivos adjuntos a el son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por ley. Si usted recibi? este mensaje err?neamente, por favor notif?quenos respondiendo al remitente, borre el mensaje original y destruya las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje. La publicaci?n, uso, copia o impresi?n total o parcial de este mensaje o documentos adjuntos queda prohibida.
> 
> Disposici?n DNDP 10-2008. El titular de los datos personales tiene la facultad de ejercer el derecho de acceso a los mismos en forma gratuita a intervalos no inferiores a seis meses, salvo que acredite un inter?s leg?timo al efecto conforme lo establecido en el art?culo 14, inciso 3 de la Ley 25.326. La DIRECCI?N NACIONAL DE PROTECCI?N DE DATOS PERSONALES, Organo de Control de la Ley 25.326, tiene la atribuci?n de atender las denuncias y reclamos que se interpongan con relaci?n al incumplimiento de las normas sobre la protecci?n de datos personales.


From karpeev at mcs.anl.gov  Fri Mar 13 12:04:54 2015
From: karpeev at mcs.anl.gov (Dmitry Karpeyev)
Date: Fri, 13 Mar 2015 17:04:54 +0000
Subject: [petsc-users] Postdoc position available
Message-ID: <CA+Y_9ULNnUWCht9R1=bQGJJ-=Do-HnQPGEv9X6KPNM-ashE3bA@mail.gmail.com>

Apologies in advance for multiple-posting.

There is a postdoctoral position available at the University of Chicago
that may be of interest to the members of this list.  The focus is on the
development of new and existing PETSc solvers required by a range of
applications.

Specifically, we are interested in the development of effective solvers and
preconditioners for variational inequalities with general constraints,
improvements to linear/nonlinear multigrid preconditioners, nonlinear
splitting and domain-decomposition methods.

Relevant applications include nonlinear elasticity with frictional contact,
strongly-coupled multiphysics models, problems with moving meshes, mortar
finite-element formulations, etc.

Knowledge of PETSc, MPI programming and numerical algorithms is required.
Familiarity with modern high-performance hardware as well as with FEM
libraries libMesh and MOOSE is a big plus.

Please forward this information to your colleagues that might be interested
in this opportunity. Feel free to contact me for more details.

Thanks!
Dmitry.

Dmitry Karpeyev
Computation Institute
University of Chicago
and
Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150313/30df7c20/attachment.html>

From juris.vencels at gmail.com  Mon Mar 16 12:22:04 2015
From: juris.vencels at gmail.com (Juris Vencels)
Date: Mon, 16 Mar 2015 11:22:04 -0600
Subject: [petsc-users] Complex vector / matrix to VTK
Message-ID: <5507113C.9020808@gmail.com>

Hello,

I want to print vector / matrix to VTK file.

I am running simple tutorial example ex15.c, but since I have compiled 
PETSc with complex scalars (I need them for my program) I get an error 
below. Can I force PETSc to print only real, imaginary (or both) part(s) 
of the solution?

Another question - ksp/ex42.c explicitly writes header information to 
VTK file while snes/ex15.c has only one line - to open VTK file. How 
these two examples are different in terms of writing output to VTK?
src/ksp/ksp/examples/tutorials/ex42.c 
<http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex42.c.html>
src/snes/examples/tutorials/ex15.c 
<http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex15.c.html>

Thanks!
Juris

> vencels at bigone:/scratch/petsc-git/src/snes/examples/tutorials$ *./ex15 
> -snes_monitor_short -da_grid_x 20 -da_grid_y 20 -p 1.3 -lambda 1 
> -jtype NEWTON -o ex15.vts*
>   0 SNES Function norm 0.690597
>   1 SNES Function norm 0.437399
>   2 SNES Function norm 0.309415
>   3 SNES Function norm 0.132358
>   4 SNES Function norm 0.0919276
>   5 SNES Function norm 0.0752639
>   6 SNES Function norm 0.0669195
>   7 SNES Function norm 0.0592599
>   8 SNES Function norm 0.0516658
>   9 SNES Function norm 0.0441565
>  10 SNES Function norm 0.0367695
>  11 SNES Function norm 0.0288595
>  12 SNES Function norm 0.0277417
>  13 SNES Function norm 0.0085536
>  14 SNES Function norm 0.00140906
>  15 SNES Function norm 7.35341e-05
>  16 SNES Function norm 3.63992e-07
>  17 SNES Function norm 1.034e-11
> CONVERGED_FNORM_RELATIVE Number of nonlinear iterations = 17
> [0]PETSC ERROR: --------------------- Error Message 
> --------------------------------------------------------------
> *[0]PETSC ERROR: No support for this operation for this object type**
> **[0]PETSC ERROR: Complex values not supported*
> [0]PETSC ERROR: See 
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2213-g589572e  
> GIT Date: 2015-03-07 10:56:38 -0600
> [0]PETSC ERROR: ./ex15 on a intelmkl-debug named bigone by vencels Mon 
> Mar 16 10:45:52 2015
> [0]PETSC ERROR: Configure options --with-cc=icc --with-cxx=icpc 
> --with-fc=ifort --with-blas-lapack-dir=/opt/intel/mkl/ 
> --with-fortran-kernels=generic --with-scalar-type=complex --with-mpi=0 
> --download-superlu
> [0]PETSC ERROR: #1 DMDAVTKWriteAll_VTS() line 30 in 
> /scratch/petsc-git/src/dm/impls/da/grvtk.c
> [0]PETSC ERROR: #2 DMDAVTKWriteAll() line 482 in 
> /scratch/petsc-git/src/dm/impls/da/grvtk.c
> [0]PETSC ERROR: #3 PetscViewerFlush_VTK() line 78 in 
> /scratch/petsc-git/src/sys/classes/viewer/impls/vtk/vtkv.c
> [0]PETSC ERROR: #4 PetscViewerFlush() line 30 in 
> /scratch/petsc-git/src/sys/classes/viewer/interface/flush.c
> [0]PETSC ERROR: #5 PetscViewerDestroy() line 101 in 
> /scratch/petsc-git/src/sys/classes/viewer/interface/view.c
> [0]PETSC ERROR: #6 main() line 257 in 
> /scratch/petsc-git/src/snes/examples/tutorials/ex15.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -da_grid_x 20
> [0]PETSC ERROR: -da_grid_y 20
> [0]PETSC ERROR: -jtype NEWTON
> [0]PETSC ERROR: -lambda 1
> [0]PETSC ERROR: -o ex15.vts
> [0]PETSC ERROR: -p 1.3
> [0]PETSC ERROR: -snes_monitor_short
> [0]PETSC ERROR: ----------------End of Error Message -------send 
> entire error message to petsc-maint at mcs.anl.gov----------
> Aborted (core dumped)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150316/bcc204ce/attachment.html>

From Carol.Brickley at awe.co.uk  Mon Mar 16 14:18:02 2015
From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk)
Date: Mon, 16 Mar 2015 19:18:02 +0000
Subject: [petsc-users] petsc 3.4.3
Message-ID: <201503161918.t2GJI7be031862@msw1.awe.co.uk>

All

I am using petsc 3.4.3 but finding it not working as quickly as it should. Any ideas?

Carol

Dr Carol Brickley
BSc,PhD,ARCS,DIC,MBCS

Senior Software Engineer
Applied Computer Science
DS+T,
AWE
Aldermaston
Reading
Berkshire
RG7 4PR

Direct: 0118 9855035



___________________________________________________
____________________________

The information in this email and in any attachment(s) is 
commercial in confidence. If you are not the named addressee(s) 
or 
if you receive this email in error then any distribution, copying or 
use of this communication or the information in it is strictly 
prohibited. Please notify us immediately by email at 
admin.internet(at)awe.co.uk, and then delete this message from 
your computer. While attachments are virus checked, AWE plc 
does not accept any liability in respect of any virus which is not 
detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150316/7f80c977/attachment.html>

From bsmith at mcs.anl.gov  Mon Mar 16 16:34:41 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 16 Mar 2015 16:34:41 -0500
Subject: [petsc-users] petsc 3.4.3
In-Reply-To: <201503161918.t2GJI7be031862@msw1.awe.co.uk>
References: <201503161918.t2GJI7be031862@msw1.awe.co.uk>
Message-ID: <70035F0E-FC6D-4240-9C3C-DD5F40E07A6F@mcs.anl.gov>


  We'd need more specific details on why it is slower than you think it should be. Is the it built with optimization (i.e. ./configure --with-debugging=0) ? Are the solvers converging with more iterations than expected http://www.mcs.anl.gov/petsc/documentation/faq.html#newton http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged ? Is the machine perhaps not suitable to for the simulations http://www.mcs.anl.gov/petsc/documentation/faq.html#computers? 

  Barry


> On Mar 16, 2015, at 2:18 PM, <Carol.Brickley at awe.co.uk> <Carol.Brickley at awe.co.uk> wrote:
> 
> All
>  
> I am using petsc 3.4.3 but finding it not working as quickly as it should. Any ideas?
>  
> Carol  
>  
> Dr Carol Brickley 
> BSc,PhD,ARCS,DIC,MBCS
>  
> Senior Software Engineer
> Applied Computer Science
> DS+T,
> AWE
> Aldermaston
> Reading
> Berkshire
> RG7 4PR
>  
> Direct: 0118 9855035
>  
> ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR
> 


From bsmith at mcs.anl.gov  Mon Mar 16 23:33:32 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 16 Mar 2015 23:33:32 -0500
Subject: [petsc-users] evidence that VINEWTONRSLS is broken (in next,
	but not in maint)
In-Reply-To: <CAOHboJ96=+AMnvjvKfPeu-a1yCLmHbe0-FL1S7x5BExU-Q8ugg@mail.gmail.com>
References: <CAOHboJ96=+AMnvjvKfPeu-a1yCLmHbe0-FL1S7x5BExU-Q8ugg@mail.gmail.com>
Message-ID: <4C51805F-89A0-4333-921D-9E347109A04D@mcs.anl.gov>


> On Mar 11, 2015, at 2:21 PM, Ed Bueler <elbueler at alaska.edu> wrote:
> 
> Dear PETSc --
> 
> I sought advice at the FAQ page on non-convergence.  It is a nice page, and I like the range of advice!
> 
> But both
>   -pc_type svd
>   -pc_type none
> seem to break the RSLS SNESVI in the next branch.
> 
> I tested with ex9.c (below) and ex58.c in next branch src/snes/examples/tutorials/, and get the same broken results.

  I have fixed this in master and next. The call to PCMGSetGalekin() from SNESSolve_VINEWTONRSLS was corrupting memory since it did not check if the PC type was MG. I have fixed PCMGSetGalekin() so that it only runs if the PC type is MG and the examples now work with svd and none

>  The other SNESVI-using examples (ex60.c, ex61.c, ex63.c, ex64.c, ex65.c) give compiler errors for me.

  I'll check these tomorrow.

  Barry

> 
> Note that both of the above PC types work fine with non-VI examples (e.g. ex1.c, ex3.c, ex5.c), of course.  Furthermore, in the maint branch, these pc_types DO work (very bottom) with ex9, and with ex58 (not shown).
> 
> I am suspicious of around lines 503-505 in virs.c.  That has been the failure location for a lot of my tries on various RSLS VI experiments.  I have looked at the source (i.e. virs.c) itself, and recognize roughly what is going on.  Objects are being reset because of a change in the active set; it's nontrivial.  I don't see how to fix it.
> 
> Thanks for help!
> 
> Ed
> 
> **** next branch
> 
> ~/petsc-next[next]$ export PETSC_ARCH=linux-c-dbg
> ~/petsc-next[next]$ cd src/snes/examples/tutorials/
> ~/petsc-next/src/snes/examples/tutorials[next]$ make ex9
> make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
> /home/ed/petsc-next/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0   -I/home/ed/petsc-next/include -I/home/ed/petsc-next/linux-c-dbg/include    `pwd`/ex9.c
> /home/ed/petsc-next/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -L/home/ed/petsc-next/linux-c-dbg/lib  -lpetsc -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lflapack -lfblas -lX11 -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpicxx -lstdc++ -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -L/home/ed/petsc-next/linux-c-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lmpi -lgcc_s -ldl 
> /bin/rm -f ex9.o
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type lu
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type svd
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Corrupt argument: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT Date: 2015-03-10 22:35:11 -0500
> [0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar 11 12:47:08 2015
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=1
> [0]PETSC ERROR: #1 VecDestroy() line 424 in /home/ed/petsc-next/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #2 PCReset_SVD() line 295 in /home/ed/petsc-next/src/ksp/pc/impls/svd/svd.c
> [0]PETSC ERROR: #3 PCReset() line 81 in /home/ed/petsc-next/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPReset() line 861 in /home/ed/petsc-next/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 SNESVIResetPCandKSP() line 305 in /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: #6 SNESSolve_VINEWTONRSLS() line 503 in /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: #7 SNESSolve() line 3846 in /home/ed/petsc-next/src/snes/interface/snes.c
> [0]PETSC ERROR: #8 main() line 100 in /home/ed/petsc-next/src/snes/examples/tutorials/ex9.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -pc_type svd
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type none
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PCMGSetGalerkin line 1028 /home/ed/petsc-next/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 348 /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: [0] SNESSolve line 3799 /home/ed/petsc-next/src/snes/interface/snes.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT Date: 2015-03-10 22:35:11 -0500
> [0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar 11 12:47:11 2015
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=1
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> ~/petsc-next/src/snes/examples/tutorials[next]$ 
> 
> ******* maint branch
> 
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ make ex9
> make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
> /home/ed/petsc-maint/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0   -I/home/ed/petsc-maint/include -I/home/ed/petsc-maint/linux-c-dbg/include    `pwd`/ex9.c
> /home/ed/petsc-maint/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -L/home/ed/petsc-maint/linux-c-dbg/lib  -lpetsc -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lflapack -lfblas -lX11 -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -L/home/ed/petsc-maint/linux-c-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl  
> /bin/rm -f ex9.o
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 3; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type svd
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type none
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> 
> 
> -- 
> Ed Bueler
> Dept of Math and Stat and Geophysical Institute
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 301C Chapman and 410D Elvey
> 907 474-7693 and 907 474-7199  (fax 907 474-5394)


From M.Imtiaz at victorchang.edu.au  Tue Mar 17 06:15:13 2015
From: M.Imtiaz at victorchang.edu.au (Mohammad Imtiaz)
Date: Tue, 17 Mar 2015 11:15:13 +0000
Subject: [petsc-users] Position based coupling matrix
Message-ID: <D12E5261.D4DB%m.imtiaz@victorchang.edu.au>

Hi,

I am trying to solve a 2 dimensional finite difference reaction diffusion problem where diffusional coupling is position based, i.e. fibres, discontinuities etc occur due to material properties (e.g. Heart tissue).

I have managed to import a matlab matrix representing coupling in four directions as below:


    PetscViewerBinaryOpen(PETSC_COMM_WORLD,?R",FILE_MODE_READ,&fd);

    VecCreate(PETSC_COMM_WORLD,&R);

    VecLoad(R,fd);

?..Repeated to get L, U, D, coupling.

How can I now make R,L,U,D available to my finite difference part of the code? So that I can do


v is field.


vxx = u[j][i].R*(u[j][i-1].v - u[j][i].v)  + u[j][i].L*(u[j][i+1].v ? u[j][i].v);


And similarly for up & down coupling.





Thanks
Mohammad Imtiaz




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/ed08c14d/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 17 10:10:58 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 10:10:58 -0500
Subject: [petsc-users] Position based coupling matrix
In-Reply-To: <D12E5261.D4DB%m.imtiaz@victorchang.edu.au>
References: <D12E5261.D4DB%m.imtiaz@victorchang.edu.au>
Message-ID: <675B979E-1190-4F63-B825-E8CE52093A38@mcs.anl.gov>


  You will need to associate the vector with a DMDA so you can use DMDAVecGetArray() to index into it. Use DMCreateGlobalVector (with the appropriate DMDA) to get the vector instead of VecCreate() before you do the VecLoad().

  Barry


> On Mar 17, 2015, at 6:15 AM, Mohammad Imtiaz <M.Imtiaz at victorchang.edu.au> wrote:
> 
> Hi,
> 
> I am trying to solve a 2 dimensional finite difference reaction diffusion problem where diffusional coupling is position based, i.e. fibres, discontinuities etc occur due to material properties (e.g. Heart tissue).
> 
> I have managed to import a matlab matrix representing coupling in four directions as below:
> 
>     PetscViewerBinaryOpen(PETSC_COMM_WORLD,?R",FILE_MODE_READ,&fd);
>     VecCreate(PETSC_COMM_WORLD,&R);
>     VecLoad(R,fd);
> 
> ?..Repeated to get L, U, D, coupling.
> 
> How can I now make R,L,U,D available to my finite difference part of the code? So that I can do
> 
> 
> v is field.
> 
> vxx = u[j][i].R*(u[j][i-1].v - u[j][i].v)  + u[j][i].L*(u[j][i+1].v ? u[j][i].v);
> 
> And similarly for up & down coupling.
> 
> 
> 
> 
> 
> Thanks
> Mohammad Imtiaz


From bsmith at mcs.anl.gov  Tue Mar 17 11:05:36 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 11:05:36 -0500
Subject: [petsc-users] evidence that VINEWTONRSLS is broken (in next,
	but not in maint)
In-Reply-To: <CAOHboJ96=+AMnvjvKfPeu-a1yCLmHbe0-FL1S7x5BExU-Q8ugg@mail.gmail.com>
References: <CAOHboJ96=+AMnvjvKfPeu-a1yCLmHbe0-FL1S7x5BExU-Q8ugg@mail.gmail.com>
Message-ID: <3CDB1B1E-7423-4756-92B1-6D69A608705D@mcs.anl.gov>


> On Mar 11, 2015, at 2:21 PM, Ed Bueler <elbueler at alaska.edu> wrote:
> 
> Dear PETSc --
> 
> I sought advice at the FAQ page on non-convergence.  It is a nice page, and I like the range of advice!
> 
> But both
>   -pc_type svd
>   -pc_type none
> seem to break the RSLS SNESVI in the next branch.
> 
> I tested with ex9.c (below) and ex58.c in next branch src/snes/examples/tutorials/, and get the same broken results.  The other SNESVI-using examples (ex60.c, ex61.c, ex63.c, ex64.c, ex65.c) give compiler errors for me.

   I have removed these examples since they have suffered from a great deal of bit rot.

> 
> Note that both of the above PC types work fine with non-VI examples (e.g. ex1.c, ex3.c, ex5.c), of course.  Furthermore, in the maint branch, these pc_types DO work (very bottom) with ex9, and with ex58 (not shown).
> 
> I am suspicious of around lines 503-505 in virs.c.  That has been the failure location for a lot of my tries on various RSLS VI experiments.  I have looked at the source (i.e. virs.c) itself, and recognize roughly what is going on.  Objects are being reset because of a change in the active set; it's nontrivial.  I don't see how to fix it.
> 
> Thanks for help!
> 
> Ed
> 
> **** next branch
> 
> ~/petsc-next[next]$ export PETSC_ARCH=linux-c-dbg
> ~/petsc-next[next]$ cd src/snes/examples/tutorials/
> ~/petsc-next/src/snes/examples/tutorials[next]$ make ex9
> make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
> /home/ed/petsc-next/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0   -I/home/ed/petsc-next/include -I/home/ed/petsc-next/linux-c-dbg/include    `pwd`/ex9.c
> /home/ed/petsc-next/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -L/home/ed/petsc-next/linux-c-dbg/lib  -lpetsc -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lflapack -lfblas -lX11 -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpicxx -lstdc++ -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -L/home/ed/petsc-next/linux-c-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/ed/petsc-next/linux-c-dbg/lib -lmpi -lgcc_s -ldl 
> /bin/rm -f ex9.o
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type lu
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> errors:     av |u-uexact|  = 7.156e-03    |u-uexact|_inf = 2.550e-02
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type svd
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Corrupt argument: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT Date: 2015-03-10 22:35:11 -0500
> [0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar 11 12:47:08 2015
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=1
> [0]PETSC ERROR: #1 VecDestroy() line 424 in /home/ed/petsc-next/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #2 PCReset_SVD() line 295 in /home/ed/petsc-next/src/ksp/pc/impls/svd/svd.c
> [0]PETSC ERROR: #3 PCReset() line 81 in /home/ed/petsc-next/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPReset() line 861 in /home/ed/petsc-next/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 SNESVIResetPCandKSP() line 305 in /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: #6 SNESSolve_VINEWTONRSLS() line 503 in /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: #7 SNESSolve() line 3846 in /home/ed/petsc-next/src/snes/interface/snes.c
> [0]PETSC ERROR: #8 main() line 100 in /home/ed/petsc-next/src/snes/examples/tutorials/ex9.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -pc_type svd
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 64) - process 0
> ~/petsc-next/src/snes/examples/tutorials[next]$ ./ex9 -pc_type none
> setup done: grid  Mx,My = 11,11  with spacing  dx,dy = 0.4000,0.4000
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PCMGSetGalerkin line 1028 /home/ed/petsc-next/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 348 /home/ed/petsc-next/src/snes/impls/vi/rs/virs.c
> [0]PETSC ERROR: [0] SNESSolve line 3799 /home/ed/petsc-next/src/snes/interface/snes.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2947-gc0ac3dd  GIT Date: 2015-03-10 22:35:11 -0500
> [0]PETSC ERROR: ./ex9 on a linux-c-dbg named bueler-leopard by ed Wed Mar 11 12:47:11 2015
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=1
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> ~/petsc-next/src/snes/examples/tutorials[next]$ 
> 
> ******* maint branch
> 
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ make ex9
> make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
> /home/ed/petsc-maint/linux-c-dbg/bin/mpicc -o ex9.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0   -I/home/ed/petsc-maint/include -I/home/ed/petsc-maint/linux-c-dbg/include    `pwd`/ex9.c
> /home/ed/petsc-maint/linux-c-dbg/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  -o ex9 ex9.o -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -L/home/ed/petsc-maint/linux-c-dbg/lib  -lpetsc -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lflapack -lfblas -lX11 -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -L/home/ed/petsc-maint/linux-c-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/ed/petsc-maint/linux-c-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl  
> /bin/rm -f ex9.o
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 3; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type svd
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> ~/petsc-maint/src/snes/examples/tutorials[maint]$ ./ex9 -pc_type none
> setup done: square       side length = 4.000
>             grid               Mx,My = 11,11
>             spacing            dx,dy = 0.400,0.400
> number of Newton iterations = 2; result = CONVERGED_FNORM_RELATIVE
> errors:    av |u-uexact|  = 7.156e-03
>            |u-uexact|_inf = 2.550e-02
> 
> 
> -- 
> Ed Bueler
> Dept of Math and Stat and Geophysical Institute
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 301C Chapman and 410D Elvey
> 907 474-7693 and 907 474-7199  (fax 907 474-5394)


From ckhuangf at gmail.com  Tue Mar 17 11:40:01 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 11:40:01 -0500
Subject: [petsc-users] Use MatConvert
Message-ID: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>

Hi,

I wonder what I did wrong when using MATConvert...

  MatCreate(* comm_, &J_);

  PetscInt * d_nnz = NULL;
  PetscInt * o_nnz = NULL;
  PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
  PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
  for (int i = 0; i < local_size; i++) {
    d_nnz[i] = d_nnz_v[i];
    o_nnz[i] = o_nnz_v[i];
  }
  MatSetSizes(J_,
          local_size,
          local_size,
          PETSC_DECIDE,
          PETSC_DECIDE);

  MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
  MatSetType(J_, MATAIJ);

  if (comm_->Get_size() > 1) {
    // MPI
    MatMPIAIJSetPreallocation(J_,
                  max_d_nz,
                  d_nnz,
                  max_o_nz,
                  o_nnz);

  } else {
    // Seq
    MatSeqAIJSetPreallocation(J_,
                  PETSC_DEFAULT,
                  d_nnz);
  }

  PetscFree(d_nnz);
  PetscFree(o_nnz);


  // Column oriented
  MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
  // Test code to check if J_ has right structure
  preAssembeJ();

[image: Inline image 1]
  MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
  MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);


 MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);


  MatZeroEntries(J_);
// Test code to check if J_ has right structure
  preAssembeJ();
[image: Inline image 2]
Thanks,
Kan

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/445bb81d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 1041 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/445bb81d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 1027 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/445bb81d/attachment-0001.png>

From bsmith at mcs.anl.gov  Tue Mar 17 12:43:20 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 12:43:20 -0500
Subject: [petsc-users] Use MatConvert
In-Reply-To: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
References: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
Message-ID: <A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>


> On Mar 17, 2015, at 11:40 AM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Hi,
> 
> I wonder what I did wrong when using MATConvert...
> 
>   MatCreate(* comm_, &J_);
> 
>   PetscInt * d_nnz = NULL;
>   PetscInt * o_nnz = NULL;
>   PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
>   PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
>   for (int i = 0; i < local_size; i++) {
>     d_nnz[i] = d_nnz_v[i];
>     o_nnz[i] = o_nnz_v[i];
>   }
>   MatSetSizes(J_,
>           local_size,
>           local_size,
>           PETSC_DECIDE,
>           PETSC_DECIDE);
>   
>   MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
>   MatSetType(J_, MATAIJ);
>   
>   if (comm_->Get_size() > 1) {

You don't need to have this if here; just call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation() and PETSc will use the correct one and ignore the other one automatically.

>     // MPI
>     MatMPIAIJSetPreallocation(J_,
>                   max_d_nz,
>                   d_nnz,
>                   max_o_nz,
>                   o_nnz);
>   
>   } else {
>     // Seq
>     MatSeqAIJSetPreallocation(J_,
>                   PETSC_DEFAULT, 
>                   d_nnz);
>   }
>   
>   PetscFree(d_nnz);
>   PetscFree(o_nnz);
> 
>   
>   // Column oriented
>   MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
>   // Test code to check if J_ has right structure
>   preAssembeJ(); 
> 
> <image.png>
>   MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
>   MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);
> 
> 
>  MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);

   I cannot understand what is wrong. Is it producing the wrong matrix? Please send the entire code so we can run it ourselves.

  Barry

>   
>   MatZeroEntries(J_);     
> // Test code to check if J_ has right structure  
>   preAssembeJ();
> <image.png>
> Thanks,
> Kan
> Cheers
> 


From bsmith at mcs.anl.gov  Tue Mar 17 13:29:35 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 13:29:35 -0500
Subject: [petsc-users] EXTERNAL: Re:  petsc 3.4.3
In-Reply-To: <201503171638.t2HGcZGo006017@msw2.awe.co.uk>
References: <201503161918.t2GJI7be031862@msw1.awe.co.uk>
	<70035F0E-FC6D-4240-9C3C-DD5F40E07A6F@mcs.anl.gov>
	<201503171638.t2HGcZGo006017@msw2.awe.co.uk>
Message-ID: <E1A4A524-DA26-4F76-BC36-71327AD39C49@mcs.anl.gov>


> On Mar 17, 2015, at 11:38 AM, <Carol.Brickley at awe.co.uk> <Carol.Brickley at awe.co.uk> wrote:
> 
> Barry,
> 
> Thanks - I have used the preallocation routines to allocate the non zero row length and am also looking at other ideas as discussed in these web pages. We are expecting a petsc structured matrix to solve more quickly than the corresponding CSR matrix but it is still slower!!

  In order to know why it is slower we need to understand what part is slower: assembling the matrix, building the preconditioner, solving the system etc. Also the time depends a great deal on problem size, for smaller problems one method may be much faster, while for a much larger problem a different method will be much better.
> 
> Would it be correct to also use the PetscWrapper::SparseMatrix::SparseMatrix
> method to speed things up further?

   I don't know what that is.

   If you can run the problem with -log_summary and send us the output it would give a starting point for understanding where the time is being spent.

  Barry

> 
> Thanks,
> 
> Carol
> 
> 
> 
> Dr Carol Brickley 
> BSc,PhD,ARCS,DIC,MBCS
> 
> Senior Software Engineer
> Applied Computer Science
> DS+T,
> AWE
> Aldermaston
> Reading
> Berkshire
> RG7 4PR
> 
> Direct: 0118 9855035
> 
> 
> -----Original Message-----
> From: Barry Smith [mailto:bsmith at mcs.anl.gov] 
> Sent: 16 March 2015 21:35
> To: Brickley Carol AWE
> Cc: petsc-users at mcs.anl.gov
> Subject: EXTERNAL: Re: [petsc-users] petsc 3.4.3
> 
> 
>  We'd need more specific details on why it is slower than you think it should be. Is the it built with optimization (i.e. ./configure --with-debugging=0) ? Are the solvers converging with more iterations than expected http://www.mcs.anl.gov/petsc/documentation/faq.html#newton http://www.mcs.anl.gov/petsc/documentation/faq.html#kspdiverged ? Is the machine perhaps not suitable to for the simulations http://www.mcs.anl.gov/petsc/documentation/faq.html#computers? 
> 
>  Barry
> 
> 
>> On Mar 16, 2015, at 2:18 PM, <Carol.Brickley at awe.co.uk> <Carol.Brickley at awe.co.uk> wrote:
>> 
>> All
>> 
>> I am using petsc 3.4.3 but finding it not working as quickly as it should. Any ideas?
>> 
>> Carol  
>> 
>> Dr Carol Brickley 
>> BSc,PhD,ARCS,DIC,MBCS
>> 
>> Senior Software Engineer
>> Applied Computer Science
>> DS+T,
>> AWE
>> Aldermaston
>> Reading
>> Berkshire
>> RG7 4PR
>> 
>> Direct: 0118 9855035
>> 
>> ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR
>> 
> 
> 
> ___________________________________________________
> ____________________________
> 
> The information in this email and in any attachment(s) is 
> commercial in confidence. If you are not the named addressee(s) 
> or 
> if you receive this email in error then any distribution, copying or 
> use of this communication or the information in it is strictly 
> prohibited. Please notify us immediately by email at 
> admin.internet(at)awe.co.uk, and then delete this message from 
> your computer. While attachments are virus checked, AWE plc 
> does not accept any liability in respect of any virus which is not 
> detected.
> 
> AWE Plc
> Registered in England and Wales
> Registration No 02763902
> AWE, Aldermaston, Reading, RG7 4PR


From ckhuangf at gmail.com  Tue Mar 17 13:34:54 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 13:34:54 -0500
Subject: [petsc-users] Use MatConvert
In-Reply-To: <A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>
References: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
	<A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>
Message-ID: <CABRkuZg7QhrTytTrrTzzAHp=Q=5CmTvVs+MX=WBZj3MgC7z08A@mail.gmail.com>

I created a short code to illustrate my problem..
if you run the program with
-mat_view
-mat_view draw
-draw_pause -1

the first matrix you see has the correct structure while the one after
Matconvert gives the wrong one.

I think it possibly has something to do with the way I use
MatSetValuesBlocked after the convert?

#include <iostream>
#include <string>

#include "petscksp.h"
#include "petscao.h"
#include "petsc.h"
#include "HYPRE.h"

using namespace std;

void preassem(Mat & J, double * P, const int & n);

int main (int argc, char ** argv)
{
  MPI_Init(&argc, &argv);

  PetscInitialize(PETSC_NULL,
          PETSC_NULL,
          PETSC_NULL,
           PETSC_NULL);

  PetscViewer viewer;
  Mat J;
  PetscInt bs = 4;
  PetscInt n = 3;
  PetscInt bssq = bs * bs;
  double P[bssq];
  PetscInt local_size = bs * n;
  for (int i = 0; i < bssq; i++) {
    P[i] = 1.;
  }
  MatCreate(PETSC_COMM_WORLD, &J);

  MatSetBlockSize(J, bs); // So MatSetValuesBlocked can be used

  MatSetType(J, MATAIJ);

  PetscInt d_nnz[12] = {8, 8, 8, 8, 4, 4, 4, 4, 4, 4, 4, 4};


  MatSetSizes(J,
          local_size,
          local_size,
          PETSC_DECIDE,
          PETSC_DECIDE);

  MatSeqAIJSetPreallocation(J,
                PETSC_DEFAULT,
                d_nnz);

  preassem(J, P, n); // Shows the correct structure



  MatConvert(J, MATBAIJ, MAT_REUSE_MATRIX, & J);

  MatZeroEntries(J);

  preassem(J, P, n); // Shows the incorrect structurex


  // Destroy PETSc objects.
  MatDestroy(& J);

  PetscViewerDestroy(& viewer);
  PetscFinalize();

  MPI_Finalize();

  return 0;
}

void preassem(Mat & J, double * P, const int & n)
{
  for (int GID = 0; GID < n; GID++) {
    MatSetValuesBlocked(J,
            1,
            & GID,
            1,
            & GID,
            P,
            ADD_VALUES);
  }
  int GID = 0;
  const int jGID = 2;

  MatSetValuesBlocked(J,
              1,
              & GID,
              1,
              & jGID,
              P,
              ADD_VALUES);
  MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY);
  MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY);

}


On Tue, Mar 17, 2015 at 12:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 17, 2015, at 11:40 AM, Chung-Kan Huang <ckhuangf at gmail.com>
> wrote:
> >
> > Hi,
> >
> > I wonder what I did wrong when using MATConvert...
> >
> >   MatCreate(* comm_, &J_);
> >
> >   PetscInt * d_nnz = NULL;
> >   PetscInt * o_nnz = NULL;
> >   PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
> >   PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
> >   for (int i = 0; i < local_size; i++) {
> >     d_nnz[i] = d_nnz_v[i];
> >     o_nnz[i] = o_nnz_v[i];
> >   }
> >   MatSetSizes(J_,
> >           local_size,
> >           local_size,
> >           PETSC_DECIDE,
> >           PETSC_DECIDE);
> >
> >   MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
> >   MatSetType(J_, MATAIJ);
> >
> >   if (comm_->Get_size() > 1) {
>
> You don't need to have this if here; just call both
> MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation() and PETSc will
> use the correct one and ignore the other one automatically.
>
> >     // MPI
> >     MatMPIAIJSetPreallocation(J_,
> >                   max_d_nz,
> >                   d_nnz,
> >                   max_o_nz,
> >                   o_nnz);
> >
> >   } else {
> >     // Seq
> >     MatSeqAIJSetPreallocation(J_,
> >                   PETSC_DEFAULT,
> >                   d_nnz);
> >   }
> >
> >   PetscFree(d_nnz);
> >   PetscFree(o_nnz);
> >
> >
> >   // Column oriented
> >   MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
> >   // Test code to check if J_ has right structure
> >   preAssembeJ();
> >
> > <image.png>
> >   MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
> >   MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);
> >
> >
> >  MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);
>
>    I cannot understand what is wrong. Is it producing the wrong matrix?
> Please send the entire code so we can run it ourselves.
>
>   Barry
>
> >
> >   MatZeroEntries(J_);
> > // Test code to check if J_ has right structure
> >   preAssembeJ();
> > <image.png>
> > Thanks,
> > Kan
> > Cheers
> >
>
>


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/ef6a03c4/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 17 14:09:35 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 14:09:35 -0500
Subject: [petsc-users] Use MatConvert
In-Reply-To: <CABRkuZg7QhrTytTrrTzzAHp=Q=5CmTvVs+MX=WBZj3MgC7z08A@mail.gmail.com>
References: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
	<A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>
	<CABRkuZg7QhrTytTrrTzzAHp=Q=5CmTvVs+MX=WBZj3MgC7z08A@mail.gmail.com>
Message-ID: <81249D0C-B872-4B8B-BEA8-EDFF99ED4830@mcs.anl.gov>


  I'm sorry, this is our fault. The code cannot correctly convert from AIJ to BAIJ for a block size > 1. The resulting BAIJ had a block size of 1 (which makes little sense in this case) hence the MatSetValuesBlocked() behaved in a way not expected. I have changed the code to generate an error in this case instead of being misleading.


   Barry

> On Mar 17, 2015, at 1:34 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> I created a short code to illustrate my problem..
> if you run the program with 
> -mat_view
> -mat_view draw
> -draw_pause -1
> 
> the first matrix you see has the correct structure while the one after Matconvert gives the wrong one.
> 
> I think it possibly has something to do with the way I use MatSetValuesBlocked after the convert?
> 
> #include <iostream>
> #include <string>
> 
> #include "petscksp.h"
> #include "petscao.h"
> #include "petsc.h"
> #include "HYPRE.h"
> 
> using namespace std;
> 
> void preassem(Mat & J, double * P, const int & n);
> 
> int main (int argc, char ** argv)
> {
>   MPI_Init(&argc, &argv);
> 
>   PetscInitialize(PETSC_NULL,
>           PETSC_NULL,
>           PETSC_NULL,
>            PETSC_NULL);
>   
>   PetscViewer viewer;
>   Mat J;
>   PetscInt bs = 4;
>   PetscInt n = 3;
>   PetscInt bssq = bs * bs;
>   double P[bssq];
>   PetscInt local_size = bs * n;
>   for (int i = 0; i < bssq; i++) {
>     P[i] = 1.;
>   }
>   MatCreate(PETSC_COMM_WORLD, &J);
> 
>   MatSetBlockSize(J, bs); // So MatSetValuesBlocked can be used
> 
>   MatSetType(J, MATAIJ);
> 
>   PetscInt d_nnz[12] = {8, 8, 8, 8, 4, 4, 4, 4, 4, 4, 4, 4};
>   
>   
>   MatSetSizes(J,
>           local_size,
>           local_size,
>           PETSC_DECIDE,
>           PETSC_DECIDE);
>   
>   MatSeqAIJSetPreallocation(J,
>                 PETSC_DEFAULT, 
>                 d_nnz);
> 
>   preassem(J, P, n); // Shows the correct structure
> 
>   
> 
>   MatConvert(J, MATBAIJ, MAT_REUSE_MATRIX, & J);
> 
>   MatZeroEntries(J);     
> 
>   preassem(J, P, n); // Shows the incorrect structurex
> 
> 
>   // Destroy PETSc objects.
>   MatDestroy(& J);
>   
>   PetscViewerDestroy(& viewer);
>   PetscFinalize();
> 
>   MPI_Finalize();
> 
>   return 0;
> }
> 
> void preassem(Mat & J, double * P, const int & n)
> {
>   for (int GID = 0; GID < n; GID++) {
>     MatSetValuesBlocked(J,
>             1,
>             & GID,
>             1,
>             & GID,
>             P,
>             ADD_VALUES);
>   }
>   int GID = 0;
>   const int jGID = 2;
>   
>   MatSetValuesBlocked(J,
>               1,
>               & GID,
>               1,
>               & jGID,
>               P,
>               ADD_VALUES);
>   MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY);
>   MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY);
> 
> }
> 
> 
> On Tue, Mar 17, 2015 at 12:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 17, 2015, at 11:40 AM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > Hi,
> >
> > I wonder what I did wrong when using MATConvert...
> >
> >   MatCreate(* comm_, &J_);
> >
> >   PetscInt * d_nnz = NULL;
> >   PetscInt * o_nnz = NULL;
> >   PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
> >   PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
> >   for (int i = 0; i < local_size; i++) {
> >     d_nnz[i] = d_nnz_v[i];
> >     o_nnz[i] = o_nnz_v[i];
> >   }
> >   MatSetSizes(J_,
> >           local_size,
> >           local_size,
> >           PETSC_DECIDE,
> >           PETSC_DECIDE);
> >
> >   MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
> >   MatSetType(J_, MATAIJ);
> >
> >   if (comm_->Get_size() > 1) {
> 
> You don't need to have this if here; just call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation() and PETSc will use the correct one and ignore the other one automatically.
> 
> >     // MPI
> >     MatMPIAIJSetPreallocation(J_,
> >                   max_d_nz,
> >                   d_nnz,
> >                   max_o_nz,
> >                   o_nnz);
> >
> >   } else {
> >     // Seq
> >     MatSeqAIJSetPreallocation(J_,
> >                   PETSC_DEFAULT,
> >                   d_nnz);
> >   }
> >
> >   PetscFree(d_nnz);
> >   PetscFree(o_nnz);
> >
> >
> >   // Column oriented
> >   MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
> >   // Test code to check if J_ has right structure
> >   preAssembeJ();
> >
> > <image.png>
> >   MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
> >   MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);
> >
> >
> >  MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);
> 
>    I cannot understand what is wrong. Is it producing the wrong matrix? Please send the entire code so we can run it ourselves.
> 
>   Barry
> 
> >
> >   MatZeroEntries(J_);
> > // Test code to check if J_ has right structure
> >   preAssembeJ();
> > <image.png>
> > Thanks,
> > Kan
> > Cheers
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Tue Mar 17 14:13:41 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 14:13:41 -0500
Subject: [petsc-users] Use MatConvert
In-Reply-To: <81249D0C-B872-4B8B-BEA8-EDFF99ED4830@mcs.anl.gov>
References: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
	<A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>
	<CABRkuZg7QhrTytTrrTzzAHp=Q=5CmTvVs+MX=WBZj3MgC7z08A@mail.gmail.com>
	<81249D0C-B872-4B8B-BEA8-EDFF99ED4830@mcs.anl.gov>
Message-ID: <CABRkuZgHNCNhsWimFcSJxWzSzSXJHkqP_AzEsNGkP2dF1Wj8OQ@mail.gmail.com>

Do you have plan to make AIJ to BAIJ possible in the future?

On Tue, Mar 17, 2015 at 2:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I'm sorry, this is our fault. The code cannot correctly convert from AIJ
> to BAIJ for a block size > 1. The resulting BAIJ had a block size of 1
> (which makes little sense in this case) hence the MatSetValuesBlocked()
> behaved in a way not expected. I have changed the code to generate an error
> in this case instead of being misleading.
>
>
>    Barry
>
> > On Mar 17, 2015, at 1:34 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > I created a short code to illustrate my problem..
> > if you run the program with
> > -mat_view
> > -mat_view draw
> > -draw_pause -1
> >
> > the first matrix you see has the correct structure while the one after
> Matconvert gives the wrong one.
> >
> > I think it possibly has something to do with the way I use
> MatSetValuesBlocked after the convert?
> >
> > #include <iostream>
> > #include <string>
> >
> > #include "petscksp.h"
> > #include "petscao.h"
> > #include "petsc.h"
> > #include "HYPRE.h"
> >
> > using namespace std;
> >
> > void preassem(Mat & J, double * P, const int & n);
> >
> > int main (int argc, char ** argv)
> > {
> >   MPI_Init(&argc, &argv);
> >
> >   PetscInitialize(PETSC_NULL,
> >           PETSC_NULL,
> >           PETSC_NULL,
> >            PETSC_NULL);
> >
> >   PetscViewer viewer;
> >   Mat J;
> >   PetscInt bs = 4;
> >   PetscInt n = 3;
> >   PetscInt bssq = bs * bs;
> >   double P[bssq];
> >   PetscInt local_size = bs * n;
> >   for (int i = 0; i < bssq; i++) {
> >     P[i] = 1.;
> >   }
> >   MatCreate(PETSC_COMM_WORLD, &J);
> >
> >   MatSetBlockSize(J, bs); // So MatSetValuesBlocked can be used
> >
> >   MatSetType(J, MATAIJ);
> >
> >   PetscInt d_nnz[12] = {8, 8, 8, 8, 4, 4, 4, 4, 4, 4, 4, 4};
> >
> >
> >   MatSetSizes(J,
> >           local_size,
> >           local_size,
> >           PETSC_DECIDE,
> >           PETSC_DECIDE);
> >
> >   MatSeqAIJSetPreallocation(J,
> >                 PETSC_DEFAULT,
> >                 d_nnz);
> >
> >   preassem(J, P, n); // Shows the correct structure
> >
> >
> >
> >   MatConvert(J, MATBAIJ, MAT_REUSE_MATRIX, & J);
> >
> >   MatZeroEntries(J);
> >
> >   preassem(J, P, n); // Shows the incorrect structurex
> >
> >
> >   // Destroy PETSc objects.
> >   MatDestroy(& J);
> >
> >   PetscViewerDestroy(& viewer);
> >   PetscFinalize();
> >
> >   MPI_Finalize();
> >
> >   return 0;
> > }
> >
> > void preassem(Mat & J, double * P, const int & n)
> > {
> >   for (int GID = 0; GID < n; GID++) {
> >     MatSetValuesBlocked(J,
> >             1,
> >             & GID,
> >             1,
> >             & GID,
> >             P,
> >             ADD_VALUES);
> >   }
> >   int GID = 0;
> >   const int jGID = 2;
> >
> >   MatSetValuesBlocked(J,
> >               1,
> >               & GID,
> >               1,
> >               & jGID,
> >               P,
> >               ADD_VALUES);
> >   MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY);
> >   MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY);
> >
> > }
> >
> >
> > On Tue, Mar 17, 2015 at 12:43 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Mar 17, 2015, at 11:40 AM, Chung-Kan Huang <ckhuangf at gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > I wonder what I did wrong when using MATConvert...
> > >
> > >   MatCreate(* comm_, &J_);
> > >
> > >   PetscInt * d_nnz = NULL;
> > >   PetscInt * o_nnz = NULL;
> > >   PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
> > >   PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
> > >   for (int i = 0; i < local_size; i++) {
> > >     d_nnz[i] = d_nnz_v[i];
> > >     o_nnz[i] = o_nnz_v[i];
> > >   }
> > >   MatSetSizes(J_,
> > >           local_size,
> > >           local_size,
> > >           PETSC_DECIDE,
> > >           PETSC_DECIDE);
> > >
> > >   MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
> > >   MatSetType(J_, MATAIJ);
> > >
> > >   if (comm_->Get_size() > 1) {
> >
> > You don't need to have this if here; just call both
> MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation() and PETSc will
> use the correct one and ignore the other one automatically.
> >
> > >     // MPI
> > >     MatMPIAIJSetPreallocation(J_,
> > >                   max_d_nz,
> > >                   d_nnz,
> > >                   max_o_nz,
> > >                   o_nnz);
> > >
> > >   } else {
> > >     // Seq
> > >     MatSeqAIJSetPreallocation(J_,
> > >                   PETSC_DEFAULT,
> > >                   d_nnz);
> > >   }
> > >
> > >   PetscFree(d_nnz);
> > >   PetscFree(o_nnz);
> > >
> > >
> > >   // Column oriented
> > >   MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
> > >   // Test code to check if J_ has right structure
> > >   preAssembeJ();
> > >
> > > <image.png>
> > >   MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
> > >   MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);
> > >
> > >
> > >  MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);
> >
> >    I cannot understand what is wrong. Is it producing the wrong matrix?
> Please send the entire code so we can run it ourselves.
> >
> >   Barry
> >
> > >
> > >   MatZeroEntries(J_);
> > > // Test code to check if J_ has right structure
> > >   preAssembeJ();
> > > <image.png>
> > > Thanks,
> > > Kan
> > > Cheers
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
>
>


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/700f90e3/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Mar 17 14:25:03 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 14:25:03 -0500
Subject: [petsc-users] Use MatConvert
In-Reply-To: <CABRkuZgHNCNhsWimFcSJxWzSzSXJHkqP_AzEsNGkP2dF1Wj8OQ@mail.gmail.com>
References: <CABRkuZgL8eyEJxGL_DbkqkosG1bAqZAvTZa4ZiBgWWopWNc9Kw@mail.gmail.com>
	<A026FCAC-CEFC-477C-9CCC-7848D2D99426@mcs.anl.gov>
	<CABRkuZg7QhrTytTrrTzzAHp=Q=5CmTvVs+MX=WBZj3MgC7z08A@mail.gmail.com>
	<81249D0C-B872-4B8B-BEA8-EDFF99ED4830@mcs.anl.gov>
	<CABRkuZgHNCNhsWimFcSJxWzSzSXJHkqP_AzEsNGkP2dF1Wj8OQ@mail.gmail.com>
Message-ID: <B7F69185-0DF4-4231-BE5A-CAFE5A52B369@mcs.anl.gov>


   We could do it but it would not be efficient. 

   It would be better if you create the BAIJ and then convert to AIJ, that can be efficient or to just use BAIJ.

   Barry

> On Mar 17, 2015, at 2:13 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Do you have plan to make AIJ to BAIJ possible in the future?
> 
> On Tue, Mar 17, 2015 at 2:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   I'm sorry, this is our fault. The code cannot correctly convert from AIJ to BAIJ for a block size > 1. The resulting BAIJ had a block size of 1 (which makes little sense in this case) hence the MatSetValuesBlocked() behaved in a way not expected. I have changed the code to generate an error in this case instead of being misleading.
> 
> 
>    Barry
> 
> > On Mar 17, 2015, at 1:34 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > I created a short code to illustrate my problem..
> > if you run the program with
> > -mat_view
> > -mat_view draw
> > -draw_pause -1
> >
> > the first matrix you see has the correct structure while the one after Matconvert gives the wrong one.
> >
> > I think it possibly has something to do with the way I use MatSetValuesBlocked after the convert?
> >
> > #include <iostream>
> > #include <string>
> >
> > #include "petscksp.h"
> > #include "petscao.h"
> > #include "petsc.h"
> > #include "HYPRE.h"
> >
> > using namespace std;
> >
> > void preassem(Mat & J, double * P, const int & n);
> >
> > int main (int argc, char ** argv)
> > {
> >   MPI_Init(&argc, &argv);
> >
> >   PetscInitialize(PETSC_NULL,
> >           PETSC_NULL,
> >           PETSC_NULL,
> >            PETSC_NULL);
> >
> >   PetscViewer viewer;
> >   Mat J;
> >   PetscInt bs = 4;
> >   PetscInt n = 3;
> >   PetscInt bssq = bs * bs;
> >   double P[bssq];
> >   PetscInt local_size = bs * n;
> >   for (int i = 0; i < bssq; i++) {
> >     P[i] = 1.;
> >   }
> >   MatCreate(PETSC_COMM_WORLD, &J);
> >
> >   MatSetBlockSize(J, bs); // So MatSetValuesBlocked can be used
> >
> >   MatSetType(J, MATAIJ);
> >
> >   PetscInt d_nnz[12] = {8, 8, 8, 8, 4, 4, 4, 4, 4, 4, 4, 4};
> >
> >
> >   MatSetSizes(J,
> >           local_size,
> >           local_size,
> >           PETSC_DECIDE,
> >           PETSC_DECIDE);
> >
> >   MatSeqAIJSetPreallocation(J,
> >                 PETSC_DEFAULT,
> >                 d_nnz);
> >
> >   preassem(J, P, n); // Shows the correct structure
> >
> >
> >
> >   MatConvert(J, MATBAIJ, MAT_REUSE_MATRIX, & J);
> >
> >   MatZeroEntries(J);
> >
> >   preassem(J, P, n); // Shows the incorrect structurex
> >
> >
> >   // Destroy PETSc objects.
> >   MatDestroy(& J);
> >
> >   PetscViewerDestroy(& viewer);
> >   PetscFinalize();
> >
> >   MPI_Finalize();
> >
> >   return 0;
> > }
> >
> > void preassem(Mat & J, double * P, const int & n)
> > {
> >   for (int GID = 0; GID < n; GID++) {
> >     MatSetValuesBlocked(J,
> >             1,
> >             & GID,
> >             1,
> >             & GID,
> >             P,
> >             ADD_VALUES);
> >   }
> >   int GID = 0;
> >   const int jGID = 2;
> >
> >   MatSetValuesBlocked(J,
> >               1,
> >               & GID,
> >               1,
> >               & jGID,
> >               P,
> >               ADD_VALUES);
> >   MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY);
> >   MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY);
> >
> > }
> >
> >
> > On Tue, Mar 17, 2015 at 12:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Mar 17, 2015, at 11:40 AM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I wonder what I did wrong when using MATConvert...
> > >
> > >   MatCreate(* comm_, &J_);
> > >
> > >   PetscInt * d_nnz = NULL;
> > >   PetscInt * o_nnz = NULL;
> > >   PetscMalloc(local_size * sizeof(PetscInt), & d_nnz);
> > >   PetscMalloc(local_size * sizeof(PetscInt), & o_nnz);
> > >   for (int i = 0; i < local_size; i++) {
> > >     d_nnz[i] = d_nnz_v[i];
> > >     o_nnz[i] = o_nnz_v[i];
> > >   }
> > >   MatSetSizes(J_,
> > >           local_size,
> > >           local_size,
> > >           PETSC_DECIDE,
> > >           PETSC_DECIDE);
> > >
> > >   MatSetBlockSize(J_, M_SIZE); // So MatSetValuesBlocked can be used
> > >   MatSetType(J_, MATAIJ);
> > >
> > >   if (comm_->Get_size() > 1) {
> >
> > You don't need to have this if here; just call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation() and PETSc will use the correct one and ignore the other one automatically.
> >
> > >     // MPI
> > >     MatMPIAIJSetPreallocation(J_,
> > >                   max_d_nz,
> > >                   d_nnz,
> > >                   max_o_nz,
> > >                   o_nnz);
> > >
> > >   } else {
> > >     // Seq
> > >     MatSeqAIJSetPreallocation(J_,
> > >                   PETSC_DEFAULT,
> > >                   d_nnz);
> > >   }
> > >
> > >   PetscFree(d_nnz);
> > >   PetscFree(o_nnz);
> > >
> > >
> > >   // Column oriented
> > >   MatSetOption(J_, MAT_ROW_ORIENTED, PETSC_FALSE);
> > >   // Test code to check if J_ has right structure
> > >   preAssembeJ();
> > >
> > > <image.png>
> > >   MatAssemblyBegin(J_, MAT_FINAL_ASSEMBLY);
> > >   MatAssemblyEnd(J_, MAT_FINAL_ASSEMBLY);
> > >
> > >
> > >  MatConvert(J_, MATBAIJ, MAT_REUSE_MATRIX, & J_);
> >
> >    I cannot understand what is wrong. Is it producing the wrong matrix? Please send the entire code so we can run it ourselves.
> >
> >   Barry
> >
> > >
> > >   MatZeroEntries(J_);
> > > // Test code to check if J_ has right structure
> > >   preAssembeJ();
> > > <image.png>
> > > Thanks,
> > > Kan
> > > Cheers
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Tue Mar 17 14:32:02 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 14:32:02 -0500
Subject: [petsc-users] BILU for AIJ matrix
Message-ID: <CABRkuZjSWBeY-7Y-v1H-9u=oiT4KkTpv-_GwA-FG8rO1+Li4mQ@mail.gmail.com>

Hi,

It is true that PCILU constructs only pointed ILU when AIJ is used.

Will it matter if I use MATSetBlockSize to a AIJ with block size > 1?

Is it true that the only benefit I will get from using MATSetBlockSize is
to be able to use MatSetValuesBlocked?


Kan


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/dedd0eae/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 17 14:45:52 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 14:45:52 -0500
Subject: [petsc-users] BILU for AIJ matrix
In-Reply-To: <CABRkuZjSWBeY-7Y-v1H-9u=oiT4KkTpv-_GwA-FG8rO1+Li4mQ@mail.gmail.com>
References: <CABRkuZjSWBeY-7Y-v1H-9u=oiT4KkTpv-_GwA-FG8rO1+Li4mQ@mail.gmail.com>
Message-ID: <BEF2B43C-A182-4E1F-A400-CAC4105930B2@mcs.anl.gov>


> On Mar 17, 2015, at 2:32 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Hi,
> 
> It is true that PCILU constructs only pointed ILU when AIJ is used.
> 
> Will it matter if I use MATSetBlockSize to a AIJ with block size > 1?
> 
> Is it true that the only benefit I will get from using MATSetBlockSize is to be able to use MatSetValuesBlocked?

   Yes, for AIJ that is pretty much the only benefit. But if you just start with a BAIJ matrix you get 1) the point block ILU, 2) less memory usage, 3) faster matrix operations so all around BAIJ wins when you have a natural block size > 1. 

  Barry

> 
> 
> Kan
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Tue Mar 17 16:26:56 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 16:26:56 -0500
Subject: [petsc-users] Use of KSPSetOperators
Message-ID: <CABRkuZhaLvK0CMQ007YHcW6WfJjFhhEbXM9tAhtFh6GHhAZ=6g@mail.gmail.com>

Hi,

I am a little confused about the statement in KSPSateOperators.


        KSP <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSP.html#KSP>/PCGetOperators
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGetOperators.html#PCGetOperators>(ksp/pc,&mat,&pmat);
is equivalent to

          set size, type, etc of mat and pmat

         MatCreate
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreate.html#MatCreate>(comm,&mat);

        MatCreate
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreate.html#MatCreate>(comm,&pmat);

        KSP <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSP.html#KSP>/PCSetOperators
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCSetOperators.html#PCSetOperators>(ksp/pc,mat,pmat);

        PetscObjectDereference
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectDereference.html#PetscObjectDereference>((PetscObject
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObject.html#PetscObject>)mat);

        PetscObjectDereference
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectDereference.html#PetscObjectDereference>((PetscObject
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObject.html#PetscObject>)pmat);

          set size, type, etc of mat and pmat



In my case I want to construct preconditioner from P which will be
assembled differently from A.

So I only need to do
KSPSetOperators(ksp, A, P);
after A & P are created.

or do I need to also do
PCSetOperators(pc, A, P);
as well?

and what are those PetscObjectDereference
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectDereference.html#PetscObjectDereference>
about?

I have tried doing only
KSPSetOperators(ksp, A, P);

and I saw the performance of linear solver changed and I got right answer
but I am not sure is if it was doing the thing I think it was doing.

And is it anyway to know if it is doing what I am asking?

Thanks,

Kan
-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/e0e1e56b/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 17 16:46:54 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 16:46:54 -0500
Subject: [petsc-users] Use of KSPSetOperators
In-Reply-To: <CABRkuZhaLvK0CMQ007YHcW6WfJjFhhEbXM9tAhtFh6GHhAZ=6g@mail.gmail.com>
References: <CABRkuZhaLvK0CMQ007YHcW6WfJjFhhEbXM9tAhtFh6GHhAZ=6g@mail.gmail.com>
Message-ID: <972111FD-E2E3-412E-AAFB-0D83D348F299@mcs.anl.gov>


> On Mar 17, 2015, at 4:26 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Hi,
> 
> I am a little confused about the statement in KSPSateOperators.
> 
> 
>         KSP/PCGetOperators
> (ksp/pc,&mat,&pmat); is equivalent to
> 
>           set size, type, etc of mat and pmat
> 
> 
>         MatCreate
> (comm,&mat);
> 
>         MatCreate
> (comm,&pmat);
> 
>         KSP/PCSetOperators
> (ksp/pc,mat,pmat);
> 
>         PetscObjectDereference((PetscObject
> )mat);
> 
>         PetscObjectDereference((PetscObject
> )pmat);
> 
>           set size, type, etc of mat and pmat
> 
> 
> 
> In my case I want to construct preconditioner from P which will be assembled differently from A.
> 
> So I only need to do 
> KSPSetOperators(ksp, A, P);
> after A & P are created.
> 
> or do I need to also do 
> PCSetOperators(pc, A, P);
> as well?
> 
> and what are those PetscObjectDereference about?
> 
> I have tried doing only 
> KSPSetOperators(ksp, A, P);
> 
> and I saw the performance of linear solver changed and I got right answer but I am not sure is if it was doing the thing I think it was doing.

   This is what you should be doing so it is fine. You can ignore all that other stuff.

 Barry

> 
> And is it anyway to know if it is doing what I am asking?
> 
> Thanks,
> 
> Kan
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Tue Mar 17 17:22:03 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Tue, 17 Mar 2015 17:22:03 -0500
Subject: [petsc-users] Does VecSetBlockSize works if Vec is created by
	VecCreateGhost
Message-ID: <CABRkuZicm_0z4fJ-Q4FdBoiku_W+CKFcwvLTUg9OhLZxZxcvWg@mail.gmail.com>

-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150317/85fb3019/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 17 19:05:50 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 17 Mar 2015 19:05:50 -0500
Subject: [petsc-users] Does VecSetBlockSize works if Vec is created by
	VecCreateGhost
In-Reply-To: <CABRkuZicm_0z4fJ-Q4FdBoiku_W+CKFcwvLTUg9OhLZxZxcvWg@mail.gmail.com>
References: <CABRkuZicm_0z4fJ-Q4FdBoiku_W+CKFcwvLTUg9OhLZxZxcvWg@mail.gmail.com>
Message-ID: <EDD33A7B-F720-4B3B-88BB-EF4A716512BE@mcs.anl.gov>


 Using VecCreateGhostBlock() is more efficient. Setting a block size with VecSetBlockSize() on a vector created with VecCreateGhost() likely won't do much.

  Barry


> On Mar 17, 2015, at 5:22 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Wed Mar 18 12:12:22 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 12:12:22 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
Message-ID: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>

Hi,

I created matrix with MATBAIJ and I use
    MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
to convert it to MATAIJ.

I ran some tests
MATBAIJ: create & use MATBAIJ for later assembly
MATBAIJ2MATAIJ: create as MATBAIJ then convert to MATAIJ & use MATAIJ for
later assembly
MATAIJ: create & use MATAIJ for later assembly

and I found following results


MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
as I was expected to see
MATBAIJ != MATBAIJ2MATAIJ == MATAIJ

Is it a potential of bug or it is what I should actually expect?

Thanks,
Kan
-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/19e89072/attachment.html>

From jed at jedbrown.org  Wed Mar 18 12:17:03 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 18 Mar 2015 11:17:03 -0600
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
Message-ID: <87d246p6nk.fsf@jedbrown.org>

Chung-Kan Huang <ckhuangf at gmail.com> writes:

> Hi,
>
> I created matrix with MATBAIJ and I use
>     MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> to convert it to MATAIJ.
>
> I ran some tests
> MATBAIJ: create & use MATBAIJ for later assembly
> MATBAIJ2MATAIJ: create as MATBAIJ then convert to MATAIJ & use MATAIJ for
> later assembly
> MATAIJ: create & use MATAIJ for later assembly
>
> and I found following results
>
>
> MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> as I was expected to see
> MATBAIJ != MATBAIJ2MATAIJ == MATAIJ

How are you defining equality?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/d041ed3c/attachment.pgp>

From amesga1 at tigers.lsu.edu  Wed Mar 18 12:18:09 2015
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Wed, 18 Mar 2015 12:18:09 -0500
Subject: [petsc-users] combining two PetscSFs
Message-ID: <CAC+VmGed5A=ZKxjh=zbLGykh3LMQkE-4BNUSGT9fxaCFqjcnqg@mail.gmail.com>

Dear all,

I was wondering if there is a way to combine two PetscSFs into one SF?
e.g., I'm doing two back to back  PetscSFBCasts:

                  Vec1--(SF1)-->Vec2--(SF2)-->Vec3

is there a way to combine SF3=SF1+SF2 so I can do one PetscSFBcast:

                    Vec1--(SF3)-->Vec3

Even if you can point me to somewhere in the source that something like
this is done?

Thanks as always,
Ata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/72329939/attachment.html>

From jed at jedbrown.org  Wed Mar 18 12:35:02 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 18 Mar 2015 11:35:02 -0600
Subject: [petsc-users] combining two PetscSFs
In-Reply-To: <CAC+VmGed5A=ZKxjh=zbLGykh3LMQkE-4BNUSGT9fxaCFqjcnqg@mail.gmail.com>
References: <CAC+VmGed5A=ZKxjh=zbLGykh3LMQkE-4BNUSGT9fxaCFqjcnqg@mail.gmail.com>
Message-ID: <87a8zap5tl.fsf@jedbrown.org>

Ataollah Mesgarnejad <amesga1 at tigers.lsu.edu> writes:

> Dear all,
>
> I was wondering if there is a way to combine two PetscSFs into one SF?
> e.g., I'm doing two back to back  PetscSFBCasts:
>
>                   Vec1--(SF1)-->Vec2--(SF2)-->Vec3
>
> is there a way to combine SF3=SF1+SF2 so I can do one PetscSFBcast:
>
>                     Vec1--(SF3)-->Vec3

Let's suppose your roots are on the left.  Get the SF1 graph
(PetscSFGetGraph) and PetscSFBcast it over SF2.  Now you have the Vec1
roots in the space of Vec3, so you can create SF3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/fbcb3ed6/attachment.pgp>

From ckhuangf at gmail.com  Wed Mar 18 12:36:23 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 12:36:23 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <87d246p6nk.fsf@jedbrown.org>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<87d246p6nk.fsf@jedbrown.org>
Message-ID: <CABRkuZin2AP82_UFx5BYPbZPufOSbLLVL5T_UbWU7BKvnwkGdQ@mail.gmail.com>

In this case I expect the result should be byte-wise identical.  Should I
expect some roundoff? Why?

Thanks,

On Wed, Mar 18, 2015 at 12:17 PM, Jed Brown <jed at jedbrown.org> wrote:

> Chung-Kan Huang <ckhuangf at gmail.com> writes:
>
> > Hi,
> >
> > I created matrix with MATBAIJ and I use
> >     MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> > to convert it to MATAIJ.
> >
> > I ran some tests
> > MATBAIJ: create & use MATBAIJ for later assembly
> > MATBAIJ2MATAIJ: create as MATBAIJ then convert to MATAIJ & use MATAIJ for
> > later assembly
> > MATAIJ: create & use MATAIJ for later assembly
> >
> > and I found following results
> >
> >
> > MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> > as I was expected to see
> > MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
>
> How are you defining equality?
>



-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/e524f15f/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 18 12:50:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 12:50:44 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
Message-ID: <9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>


  Do you mean that some entries in the matrices are different? That the nonzero structure is different? Something else?

  In general if you are putting the values in with MatSetValuesBlocked() then I would expect the resulting matrices to have the same entries (though once you start solvers such as ILU you will have different convergence results).

  Barry

> On Mar 18, 2015, at 12:12 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Hi,
> 
> I created matrix with MATBAIJ and I use  
>     MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> to convert it to MATAIJ.
> 
> I ran some tests 
> MATBAIJ: create & use MATBAIJ for later assembly
> MATBAIJ2MATAIJ: create as MATBAIJ then convert to MATAIJ & use MATAIJ for later assembly
> MATAIJ: create & use MATAIJ for later assembly
> 
> and I found following results
> 
> 
> MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> as I was expected to see
> MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
> 
> Is it a potential of bug or it is what I should actually expect?
> 
> Thanks,
> Kan
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Wed Mar 18 13:23:39 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 13:23:39 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
Message-ID: <CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>

Sorry I didn't state my problem more accurate.  The entries in the input
matrixes and vectors? are correct -- they have right values and correct
nonzero structure.
The KSP was setup so I can solve the lienar systems for an unsteady
state convective problem.
I use KSPBCGS + PCILU and so far I only try sequential on a very
small problem.
I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/52f1b66e/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 18 13:33:33 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 13:33:33 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
Message-ID: <87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>


   As I think I said earlier: ILU will behave differently with BAIJ and AIJ format. With BAIJ it uses a point block factorization (doing an ILU treating each block in the matrix as a single point (and doing a dense factorization for the inverse) while AIJ ignores the block size and just uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU will be different; usually BAIJ will converge better but it need not.

   Barry

> On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Sorry I didn't state my problem more accurate.  The entries in the input matrixes and vectors? are correct -- they have right values and correct nonzero structure.
> The KSP was setup so I can solve the lienar systems for an unsteady state convective problem.
> I use KSPBCGS + PCILU and so far I only try sequential on a very small problem.
> I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
>  
> Thanks,
>  
>  


From lb2653 at columbia.edu  Wed Mar 18 13:41:15 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Wed, 18 Mar 2015 14:41:15 -0400
Subject: [petsc-users] Fast LU solver for block diagonal matrix
Message-ID: <5509C6CB.7030703@columbi.edu>

Hi all,
I am solving multi-physics problem that leads to a jacobian of the form:

[ Jee  Jeo ]
[ Joe  Joo ]

where Jee is 5by5 block diagonal. This feature makes it a very good 
candidate for a Schur complement.
Indeed, Jee could be inverted in parallel with no inter-nodes communication.
My only issue is the fact that the Schur complement is not accessible 
directly with PETSC, only an approximation is available, for direct 
solvers (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).

Any advice on how I could efficiently compute Jee^-1 for the given 
structure?
I am currently thinking about hard coding the formula for the inverse of 
a 5by5 and sweeping through Jee (with some threading) and storing the 
inverse in-place. Instead of hard coding the formula for a 5by5 I could 
also do a MatLUFactorSym on a 5by5 matrix but it would not give me an 
inverse, only a factorization...

Thanks in advance for your suggestions!

-- 
Best,
Luc



From bsmith at mcs.anl.gov  Wed Mar 18 14:22:16 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 14:22:16 -0500
Subject: [petsc-users] Fast LU solver for block diagonal matrix
In-Reply-To: <5509C6CB.7030703@columbi.edu>
References: <5509C6CB.7030703@columbi.edu>
Message-ID: <DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>


  Do you want to explicitly compute (as a matrix) S = Joo - joe * inv(Jee) jeo   or do you want to just have an efficient computation of 
S y  for any y vector? 

  Here is some possibly useful information. If you create Jee as a BAIJ matrix of block size 5 and use MatILUFactor() it will efficiently factor this matrix (each 5 by 5 block factorization is done with custom code) then you can use MatSolve() efficiently with the result (note that internally when factoring a BAIJ matrix PETSc actually stores the inverse of the diagonal blocks so in your case the MatSolve() actually ends up doing little matrix-vector products (and there are no triangular solves). 

  To use this with the MatCreateSchurComplement() object you can do

   MatCreateSchurComplement(...,&S)
   MatSchurComplementGetKSP(S,&ksp)
   KSPSetType(ksp,KSPPREONLY);

   now MatMult(S,y,z) will be efficient. 

   Of course you still have the question, how do you plan to solve S? This depends on its structure and if you have a good way of preconditioning it.

   If you want to explicitly form S you can use MatMatSolve( fact,jeo) but this requires making jeo dense which seems to defeat the purpose.

  Barry



> On Mar 18, 2015, at 1:41 PM, Luc Berger-Vergiat <lb2653 at columbia.edu> wrote:
> 
> Hi all,
> I am solving multi-physics problem that leads to a jacobian of the form:
> 
> [ Jee  Jeo ]
> [ Joe  Joo ]
> 
> where Jee is 5by5 block diagonal. This feature makes it a very good candidate for a Schur complement.
> Indeed, Jee could be inverted in parallel with no inter-nodes communication.
> My only issue is the fact that the Schur complement is not accessible directly with PETSC, only an approximation is available, for direct solvers (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).
> 
> Any advice on how I could efficiently compute Jee^-1 for the given structure?
> I am currently thinking about hard coding the formula for the inverse of a 5by5 and sweeping through Jee (with some threading) and storing the inverse in-place. Instead of hard coding the formula for a 5by5 I could also do a MatLUFactorSym on a 5by5 matrix but it would not give me an inverse, only a factorization...
> 
> Thanks in advance for your suggestions!
> 
> -- 
> Best,
> Luc
> 
> 


From bsmith at mcs.anl.gov  Wed Mar 18 14:24:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 14:24:40 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
Message-ID: <0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>


  Are you always using MatSetValuesBlocked()? 

  Did you look at the three matrices (before passing to the solve), do they all have the exact same nonzero structure and entries?

  Barry

> On Mar 18, 2015, at 1:45 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> So go back to my original question,
>  
> I found the resuls look like
> MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> as I was expected to see
> MATBAIJ != MATBAIJ2MATAIJ == MATAIJ 
>  
> because I thought MATBAIJ2MATAIJ & MATAIJ should use the same ILU factorization.
> 
> On Wed, Mar 18, 2015 at 1:33 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    As I think I said earlier: ILU will behave differently with BAIJ and AIJ format. With BAIJ it uses a point block factorization (doing an ILU treating each block in the matrix as a single point (and doing a dense factorization for the inverse) while AIJ ignores the block size and just uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU will be different; usually BAIJ will converge better but it need not.
> 
>    Barry
> 
> > On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > Sorry I didn't state my problem more accurate.  The entries in the input matrixes and vectors? are correct -- they have right values and correct nonzero structure.
> > The KSP was setup so I can solve the lienar systems for an unsteady state convective problem.
> > I use KSPBCGS + PCILU and so far I only try sequential on a very small problem.
> > I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
> >
> > Thanks,
> >
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Wed Mar 18 14:46:20 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 14:46:20 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
Message-ID: <CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>

Yes, I was suprised that I can still use MatSetValuesBlocked after I did
MatConvert MATBAIJ to MATAIJ.

Then I just found the mat type is still seqbaij which means the MatConvert
didn't work.

This is how I did MatConvert

MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);

where J is my Jacobian matrix.

Thanks,


On Wed, Mar 18, 2015 at 2:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Are you always using MatSetValuesBlocked()?
>
>   Did you look at the three matrices (before passing to the solve), do
> they all have the exact same nonzero structure and entries?
>
>   Barry
>
> > On Mar 18, 2015, at 1:45 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > So go back to my original question,
> >
> > I found the resuls look like
> > MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> > as I was expected to see
> > MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
> >
> > because I thought MATBAIJ2MATAIJ & MATAIJ should use the same ILU
> factorization.
> >
> > On Wed, Mar 18, 2015 at 1:33 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    As I think I said earlier: ILU will behave differently with BAIJ and
> AIJ format. With BAIJ it uses a point block factorization (doing an ILU
> treating each block in the matrix as a single point (and doing a dense
> factorization for the inverse) while AIJ ignores the block size and just
> uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU
> will be different; usually BAIJ will converge better but it need not.
> >
> >    Barry
> >
> > > On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com>
> wrote:
> > >
> > > Sorry I didn't state my problem more accurate.  The entries in the
> input matrixes and vectors? are correct -- they have right values and
> correct nonzero structure.
> > > The KSP was setup so I can solve the lienar systems for an unsteady
> state convective problem.
> > > I use KSPBCGS + PCILU and so far I only try sequential on a very small
> problem.
> > > I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
> > >
> > > Thanks,
> > >
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
>
>


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/c62b6a72/attachment.html>

From jed at jedbrown.org  Wed Mar 18 14:48:54 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 18 Mar 2015 13:48:54 -0600
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
Message-ID: <87vbhynl21.fsf@jedbrown.org>

Chung-Kan Huang <ckhuangf at gmail.com> writes:

> Yes, I was suprised that I can still use MatSetValuesBlocked after I did
> MatConvert MATBAIJ to MATAIJ.
>
> Then I just found the mat type is still seqbaij which means the MatConvert
> didn't work.
>
> This is how I did MatConvert
>
> MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
>
> where J is my Jacobian matrix.

Seems to me this should fail because you didn't MatConvert this matrix
yet.  Anyway, you should use MAT_INITIAL_MATRIX and put the result in a
new Mat.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/0f39d45a/attachment.pgp>

From bsmith at mcs.anl.gov  Wed Mar 18 14:54:43 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 14:54:43 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
Message-ID: <D0C32243-704B-4322-874D-7A174E9D1D15@mcs.anl.gov>


   You can use MatSetValuesBlocked() for AIJ matrices, obviously the matrix needs to have the block size

  Barry

> On Mar 18, 2015, at 2:46 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Yes, I was suprised that I can still use MatSetValuesBlocked after I did MatConvert MATBAIJ to MATAIJ.
>  
> Then I just found the mat type is still seqbaij which means the MatConvert didn't work.
>  
> This is how I did MatConvert
>  
> MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
>  
> where J is my Jacobian matrix.
>  
> Thanks,
>  
> 
> On Wed, Mar 18, 2015 at 2:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Are you always using MatSetValuesBlocked()?
> 
>   Did you look at the three matrices (before passing to the solve), do they all have the exact same nonzero structure and entries?
> 
>   Barry
> 
> > On Mar 18, 2015, at 1:45 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > So go back to my original question,
> >
> > I found the resuls look like
> > MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> > as I was expected to see
> > MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
> >
> > because I thought MATBAIJ2MATAIJ & MATAIJ should use the same ILU factorization.
> >
> > On Wed, Mar 18, 2015 at 1:33 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    As I think I said earlier: ILU will behave differently with BAIJ and AIJ format. With BAIJ it uses a point block factorization (doing an ILU treating each block in the matrix as a single point (and doing a dense factorization for the inverse) while AIJ ignores the block size and just uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU will be different; usually BAIJ will converge better but it need not.
> >
> >    Barry
> >
> > > On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> > >
> > > Sorry I didn't state my problem more accurate.  The entries in the input matrixes and vectors? are correct -- they have right values and correct nonzero structure.
> > > The KSP was setup so I can solve the lienar systems for an unsteady state convective problem.
> > > I use KSPBCGS + PCILU and so far I only try sequential on a very small problem.
> > > I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
> > >
> > > Thanks,
> > >
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From lb2653 at columbia.edu  Wed Mar 18 14:54:47 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Wed, 18 Mar 2015 15:54:47 -0400
Subject: [petsc-users] Fast LU solver for block diagonal matrix
In-Reply-To: <DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>
References: <5509C6CB.7030703@columbi.edu>
	<DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>
Message-ID: <5509D807.2000809@columbi.edu>

Hi Barry,
I would like to compute S explicitly, I have a few good option to 
precondition S but they are based on S not an approximation of S (i.e. I 
don't want to compute my preconditioner using Sp different from S).

Also Jee is obtained using MatGetSubmatrix(J,isrow_e,iscol_e) and J is 
an AIJ matrix so I assume that Jee is AIJ too.
I can convert Jee to BAIJ to take advantage of the block structure but 
from your conversation with Chung-Kan it might require to reassemble?
In that case what about the following:

    MatCreateBAIJ
    <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateBAIJ.html#MatCreateBAIJ>(comm,
    5, Jee_inv)
    for(i=0,i<nblocks,i++)
    {
    MatGetValues
    <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetValues.html#MatGetValues>(Jee,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],
    block_values)
         some_package_inverts( block_values )
    MatSetValuesBlocked
    <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesBlocked.html#MatSetValuesBlocked>(Jee_inv,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],
    block_values)
    }
    MatAssemblyBegin
    <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatAssemblyBegin.html#MatAssemblyBegin>()
    MatAssemblyEnd
    <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatAssemblyBegin.html#MatAssemblyBegin>()

With this I could then just go on and do matrix multiplications to 
finish my Schur complement calculations.
Would this be decently efficient?

Best,
Luc

On 03/18/2015 03:22 PM, Barry Smith wrote:
>    Do you want to explicitly compute (as a matrix) S = Joo - joe * inv(Jee) jeo   or do you want to just have an efficient computation of
> S y  for any y vector?
>
>    Here is some possibly useful information. If you create Jee as a BAIJ matrix of block size 5 and use MatILUFactor() it will efficiently factor this matrix (each 5 by 5 block factorization is done with custom code) then you can use MatSolve() efficiently with the result (note that internally when factoring a BAIJ matrix PETSc actually stores the inverse of the diagonal blocks so in your case the MatSolve() actually ends up doing little matrix-vector products (and there are no triangular solves).
>
>    To use this with the MatCreateSchurComplement() object you can do
>
>     MatCreateSchurComplement(...,&S)
>     MatSchurComplementGetKSP(S,&ksp)
>     KSPSetType(ksp,KSPPREONLY);
>
>     now MatMult(S,y,z) will be efficient.
>
>     Of course you still have the question, how do you plan to solve S? This depends on its structure and if you have a good way of preconditioning it.
>
>     If you want to explicitly form S you can use MatMatSolve( fact,jeo) but this requires making jeo dense which seems to defeat the purpose.
>
>    Barry
>
>
>
>> On Mar 18, 2015, at 1:41 PM, Luc Berger-Vergiat <lb2653 at columbia.edu> wrote:
>>
>> Hi all,
>> I am solving multi-physics problem that leads to a jacobian of the form:
>>
>> [ Jee  Jeo ]
>> [ Joe  Joo ]
>>
>> where Jee is 5by5 block diagonal. This feature makes it a very good candidate for a Schur complement.
>> Indeed, Jee could be inverted in parallel with no inter-nodes communication.
>> My only issue is the fact that the Schur complement is not accessible directly with PETSC, only an approximation is available, for direct solvers (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).
>>
>> Any advice on how I could efficiently compute Jee^-1 for the given structure?
>> I am currently thinking about hard coding the formula for the inverse of a 5by5 and sweeping through Jee (with some threading) and storing the inverse in-place. Instead of hard coding the formula for a 5by5 I could also do a MatLUFactorSym on a 5by5 matrix but it would not give me an inverse, only a factorization...
>>
>> Thanks in advance for your suggestions!
>>
>> -- 
>> Best,
>> Luc
>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/6b281253/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Mar 18 15:04:10 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 15:04:10 -0500
Subject: [petsc-users] Fast LU solver for block diagonal matrix
In-Reply-To: <5509D807.2000809@columbi.edu>
References: <5509C6CB.7030703@columbi.edu>
	<DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>
	<5509D807.2000809@columbi.edu>
Message-ID: <19406F78-9CBC-42D8-B5AC-47635D833F42@mcs.anl.gov>


> On Mar 18, 2015, at 2:54 PM, Luc Berger-Vergiat <lb2653 at columbia.edu> wrote:
> 
> Hi Barry,
> I would like to compute S explicitly, I have a few good option to precondition S but they are based on S not an approximation of S (i.e. I don't want to compute my preconditioner using Sp different from S).
> 
> Also Jee is obtained using MatGetSubmatrix(J,isrow_e,iscol_e) and J is an AIJ matrix so I assume that Jee is AIJ too.
> I can convert Jee to BAIJ to take advantage of the block structure but from your conversation with Chung-Kan it might require to reassemble?
> In that case what about the following:
> MatCreateBAIJ(comm, 5, Jee_inv)
> for(i=0,i<nblocks,i++)
> {
>     MatGetValues(Jee,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4], block_values)
>     some_package_inverts( block_values )
>     MatSetValuesBlocked(Jee_inv,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4], block_values)
> }
> MatAssemblyBegin()
> MatAssemblyEnd()
> With this I could then just go on and do matrix multiplications to finish my Schur complement calculations.
> Would this be decently efficient?

   This would be fine. Then you can use MatPtAP or MatRARt to compute the triple matrix product.

  Barry

> Best,
> Luc
> 
> On 03/18/2015 03:22 PM, Barry Smith wrote:
>>   Do you want to explicitly compute (as a matrix) S = Joo - joe * inv(Jee) jeo   or do you want to just have an efficient computation of 
>> S y  for any y vector? 
>> 
>>   Here is some possibly useful information. If you create Jee as a BAIJ matrix of block size 5 and use MatILUFactor() it will efficiently factor this matrix (each 5 by 5 block factorization is done with custom code) then you can use MatSolve() efficiently with the result (note that internally when factoring a BAIJ matrix PETSc actually stores the inverse of the diagonal blocks so in your case the MatSolve() actually ends up doing little matrix-vector products (and there are no triangular solves). 
>> 
>>   To use this with the MatCreateSchurComplement() object you can do
>> 
>>    MatCreateSchurComplement(...,&S)
>>    MatSchurComplementGetKSP(S,&ksp)
>>    KSPSetType(ksp,KSPPREONLY);
>> 
>>    now MatMult(S,y,z) will be efficient. 
>> 
>>    Of course you still have the question, how do you plan to solve S? This depends on its structure and if you have a good way of preconditioning it.
>> 
>>    If you want to explicitly form S you can use MatMatSolve( fact,jeo) but this requires making jeo dense which seems to defeat the purpose.
>> 
>>   Barry
>> 
>> 
>> 
>> 
>>> On Mar 18, 2015, at 1:41 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
>>>  wrote:
>>> 
>>> Hi all,
>>> I am solving multi-physics problem that leads to a jacobian of the form:
>>> 
>>> [ Jee  Jeo ]
>>> [ Joe  Joo ]
>>> 
>>> where Jee is 5by5 block diagonal. This feature makes it a very good candidate for a Schur complement.
>>> Indeed, Jee could be inverted in parallel with no inter-nodes communication.
>>> My only issue is the fact that the Schur complement is not accessible directly with PETSC, only an approximation is available, for direct solvers (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).
>>> 
>>> Any advice on how I could efficiently compute Jee^-1 for the given structure?
>>> I am currently thinking about hard coding the formula for the inverse of a 5by5 and sweeping through Jee (with some threading) and storing the inverse in-place. Instead of hard coding the formula for a 5by5 I could also do a MatLUFactorSym on a 5by5 matrix but it would not give me an inverse, only a factorization...
>>> 
>>> Thanks in advance for your suggestions!
>>> 
>>> -- 
>>> Best,
>>> Luc
>>> 
>>> 
>>> 
> 


From ckhuangf at gmail.com  Wed Mar 18 15:07:46 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 15:07:46 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <87vbhynl21.fsf@jedbrown.org>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<87vbhynl21.fsf@jedbrown.org>
Message-ID: <CABRkuZhBjO7eHMGnGsK=JCxqtAHkvrixSMr4WOuny27WQCRCZw@mail.gmail.com>

So I guess I misunderstood the meaning of " inplace conversion".
and what do MAT_INITIAL_MATRIX and MAT_REUSE_MATRIX do anyway?




On Wed, Mar 18, 2015 at 2:48 PM, Jed Brown <jed at jedbrown.org> wrote:

> Chung-Kan Huang <ckhuangf at gmail.com> writes:
>
> > Yes, I was suprised that I can still use MatSetValuesBlocked after I did
> > MatConvert MATBAIJ to MATAIJ.
> >
> > Then I just found the mat type is still seqbaij which means the
> MatConvert
> > didn't work.
> >
> > This is how I did MatConvert
> >
> > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> >
> > where J is my Jacobian matrix.
>
> Seems to me this should fail because you didn't MatConvert this matrix
> yet.  Anyway, you should use MAT_INITIAL_MATRIX and put the result in a
> new Mat.
>



-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/4272db46/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 18 15:09:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 15:09:40 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZhBjO7eHMGnGsK=JCxqtAHkvrixSMr4WOuny27WQCRCZw@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<87vbhynl21.fsf@jedbrown.org>
	<CABRkuZhBjO7eHMGnGsK=JCxqtAHkvrixSMr4WOuny27WQCRCZw@mail.gmail.com>
Message-ID: <6A5DFFAE-D20E-45D3-BACA-2635F16C25A4@mcs.anl.gov>


> On Mar 18, 2015, at 3:07 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> So I guess I misunderstood the meaning of " inplace conversion".
> and what do MAT_INITIAL_MATRIX and MAT_REUSE_MATRIX do anyway?

  No you have not misunderstood it.

Currently
   MAT_REUSE_MATRIX is only supported for inplace conversion, otherwise use
   MAT_INITIAL_MATRIX.

  For testing purposes you can use MAT_INITIAL_MATRIX. 


> 
> 
> 
> 
> On Wed, Mar 18, 2015 at 2:48 PM, Jed Brown <jed at jedbrown.org> wrote:
> Chung-Kan Huang <ckhuangf at gmail.com> writes:
> 
> > Yes, I was suprised that I can still use MatSetValuesBlocked after I did
> > MatConvert MATBAIJ to MATAIJ.
> >
> > Then I just found the mat type is still seqbaij which means the MatConvert
> > didn't work.
> >
> > This is how I did MatConvert
> >
> > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> >
> > where J is my Jacobian matrix.
> 
> Seems to me this should fail because you didn't MatConvert this matrix
> yet.  Anyway, you should use MAT_INITIAL_MATRIX and put the result in a
> new Mat.
> 
> 
> 
> -- 
> Cheers
> 


From ckhuangf at gmail.com  Wed Mar 18 15:10:03 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 15:10:03 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <D0C32243-704B-4322-874D-7A174E9D1D15@mcs.anl.gov>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<D0C32243-704B-4322-874D-7A174E9D1D15@mcs.anl.gov>
Message-ID: <CABRkuZi-vpwa81_VtdaaZ1Xy-bhiihFWFEiJRCqnBNVHbZuPMg@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    You can use MatSetValuesBlocked() for AIJ matrices, obviously the
> matrix needs to have the block size
>

Barry,
I don't think that is the problem.  The problem should be like Jed said,
the MatConvert didn't actually work.
Now we can focus on how to make it work in my case.

>
>   Barry
>
> > On Mar 18, 2015, at 2:46 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > Yes, I was suprised that I can still use MatSetValuesBlocked after I did
> MatConvert MATBAIJ to MATAIJ.
> >
> > Then I just found the mat type is still seqbaij which means the
> MatConvert didn't work.
> >
> > This is how I did MatConvert
> >
> > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> >
> > where J is my Jacobian matrix.
> >
> > Thanks,
> >
> >
> > On Wed, Mar 18, 2015 at 2:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Are you always using MatSetValuesBlocked()?
> >
> >   Did you look at the three matrices (before passing to the solve), do
> they all have the exact same nonzero structure and entries?
> >
> >   Barry
> >
> > > On Mar 18, 2015, at 1:45 PM, Chung-Kan Huang <ckhuangf at gmail.com>
> wrote:
> > >
> > > So go back to my original question,
> > >
> > > I found the resuls look like
> > > MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> > > as I was expected to see
> > > MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
> > >
> > > because I thought MATBAIJ2MATAIJ & MATAIJ should use the same ILU
> factorization.
> > >
> > > On Wed, Mar 18, 2015 at 1:33 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >    As I think I said earlier: ILU will behave differently with BAIJ
> and AIJ format. With BAIJ it uses a point block factorization (doing an ILU
> treating each block in the matrix as a single point (and doing a dense
> factorization for the inverse) while AIJ ignores the block size and just
> uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU
> will be different; usually BAIJ will converge better but it need not.
> > >
> > >    Barry
> > >
> > > > On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com>
> wrote:
> > > >
> > > > Sorry I didn't state my problem more accurate.  The entries in the
> input matrixes and vectors? are correct -- they have right values and
> correct nonzero structure.
> > > > The KSP was setup so I can solve the lienar systems for an unsteady
> state convective problem.
> > > > I use KSPBCGS + PCILU and so far I only try sequential on a very
> small problem.
> > > > I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
> > > >
> > > > Thanks,
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
>
>


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/7cbff69f/attachment.html>

From lb2653 at columbia.edu  Wed Mar 18 15:10:25 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Wed, 18 Mar 2015 16:10:25 -0400
Subject: [petsc-users] Fast LU solver for block diagonal matrix
In-Reply-To: <19406F78-9CBC-42D8-B5AC-47635D833F42@mcs.anl.gov>
References: <5509C6CB.7030703@columbi.edu>
	<DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>
	<5509D807.2000809@columbi.edu>
	<19406F78-9CBC-42D8-B5AC-47635D833F42@mcs.anl.gov>
Message-ID: <5509DBB1.4020108@columbi.edu>

I will have to do a MatMatMatMult or two MatMatMult since J is 
unsymmetric (Jeo != Joe^T).
I think two MatMatMult could be more efficient if I use the 
MatMatMultSymbolic/MatMatMultNumeric correctly.

Best,
Luc

On 03/18/2015 04:04 PM, Barry Smith wrote:
>> On Mar 18, 2015, at 2:54 PM, Luc Berger-Vergiat <lb2653 at columbia.edu> wrote:
>>
>> Hi Barry,
>> I would like to compute S explicitly, I have a few good option to precondition S but they are based on S not an approximation of S (i.e. I don't want to compute my preconditioner using Sp different from S).
>>
>> Also Jee is obtained using MatGetSubmatrix(J,isrow_e,iscol_e) and J is an AIJ matrix so I assume that Jee is AIJ too.
>> I can convert Jee to BAIJ to take advantage of the block structure but from your conversation with Chung-Kan it might require to reassemble?
>> In that case what about the following:
>> MatCreateBAIJ(comm, 5, Jee_inv)
>> for(i=0,i<nblocks,i++)
>> {
>>      MatGetValues(Jee,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4], block_values)
>>      some_package_inverts( block_values )
>>      MatSetValuesBlocked(Jee_inv,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4], block_values)
>> }
>> MatAssemblyBegin()
>> MatAssemblyEnd()
>> With this I could then just go on and do matrix multiplications to finish my Schur complement calculations.
>> Would this be decently efficient?
>     This would be fine. Then you can use MatPtAP or MatRARt to compute the triple matrix product.
>
>    Barry
>
>> Best,
>> Luc
>>
>> On 03/18/2015 03:22 PM, Barry Smith wrote:
>>>    Do you want to explicitly compute (as a matrix) S = Joo - joe * inv(Jee) jeo   or do you want to just have an efficient computation of
>>> S y  for any y vector?
>>>
>>>    Here is some possibly useful information. If you create Jee as a BAIJ matrix of block size 5 and use MatILUFactor() it will efficiently factor this matrix (each 5 by 5 block factorization is done with custom code) then you can use MatSolve() efficiently with the result (note that internally when factoring a BAIJ matrix PETSc actually stores the inverse of the diagonal blocks so in your case the MatSolve() actually ends up doing little matrix-vector products (and there are no triangular solves).
>>>
>>>    To use this with the MatCreateSchurComplement() object you can do
>>>
>>>     MatCreateSchurComplement(...,&S)
>>>     MatSchurComplementGetKSP(S,&ksp)
>>>     KSPSetType(ksp,KSPPREONLY);
>>>
>>>     now MatMult(S,y,z) will be efficient.
>>>
>>>     Of course you still have the question, how do you plan to solve S? This depends on its structure and if you have a good way of preconditioning it.
>>>
>>>     If you want to explicitly form S you can use MatMatSolve( fact,jeo) but this requires making jeo dense which seems to defeat the purpose.
>>>
>>>    Barry
>>>
>>>
>>>
>>>
>>>> On Mar 18, 2015, at 1:41 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
>>>>   wrote:
>>>>
>>>> Hi all,
>>>> I am solving multi-physics problem that leads to a jacobian of the form:
>>>>
>>>> [ Jee  Jeo ]
>>>> [ Joe  Joo ]
>>>>
>>>> where Jee is 5by5 block diagonal. This feature makes it a very good candidate for a Schur complement.
>>>> Indeed, Jee could be inverted in parallel with no inter-nodes communication.
>>>> My only issue is the fact that the Schur complement is not accessible directly with PETSC, only an approximation is available, for direct solvers (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).
>>>>
>>>> Any advice on how I could efficiently compute Jee^-1 for the given structure?
>>>> I am currently thinking about hard coding the formula for the inverse of a 5by5 and sweeping through Jee (with some threading) and storing the inverse in-place. Instead of hard coding the formula for a 5by5 I could also do a MatLUFactorSym on a 5by5 matrix but it would not give me an inverse, only a factorization...
>>>>
>>>> Thanks in advance for your suggestions!
>>>>
>>>> -- 
>>>> Best,
>>>> Luc
>>>>
>>>>
>>>>



From ckhuangf at gmail.com  Wed Mar 18 15:14:01 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 18 Mar 2015 15:14:01 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <6A5DFFAE-D20E-45D3-BACA-2635F16C25A4@mcs.anl.gov>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<87vbhynl21.fsf@jedbrown.org>
	<CABRkuZhBjO7eHMGnGsK=JCxqtAHkvrixSMr4WOuny27WQCRCZw@mail.gmail.com>
	<6A5DFFAE-D20E-45D3-BACA-2635F16C25A4@mcs.anl.gov>
Message-ID: <CABRkuZjBxK2dhooAWzzbwK9aTqwJg=JOwHGZoPUo+EJ57SpoVg@mail.gmail.com>

On Wed, Mar 18, 2015 at 3:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 18, 2015, at 3:07 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > So I guess I misunderstood the meaning of " inplace conversion".
> > and what do MAT_INITIAL_MATRIX and MAT_REUSE_MATRIX do anyway?
>
>   No you have not misunderstood it.
>
> Currently
>    MAT_REUSE_MATRIX is only supported for inplace conversion, otherwise use
>    MAT_INITIAL_MATRIX.
>
>   For testing purposes you can use MAT_INITIAL_MATRIX.
>
>
I will try to convert the matrix in a new matrix using MAT_INITIAL_MATRIX.
while
MatConvert(J, MATAIJ, MAT_INITIAL_MATRIX, & A);
will convert the matrix and port the results into a new
what does
MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
actually do?

Thanks,


>
> >
> >
> >
> >
> > On Wed, Mar 18, 2015 at 2:48 PM, Jed Brown <jed at jedbrown.org> wrote:
> > Chung-Kan Huang <ckhuangf at gmail.com> writes:
> >
> > > Yes, I was suprised that I can still use MatSetValuesBlocked after I
> did
> > > MatConvert MATBAIJ to MATAIJ.
> > >
> > > Then I just found the mat type is still seqbaij which means the
> MatConvert
> > > didn't work.
> > >
> > > This is how I did MatConvert
> > >
> > > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> > >
> > > where J is my Jacobian matrix.
> >
> > Seems to me this should fail because you didn't MatConvert this matrix
> > yet.  Anyway, you should use MAT_INITIAL_MATRIX and put the result in a
> > new Mat.
> >
> >
> >
> > --
> > Cheers
> >
>
>


-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/892e9798/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Mar 18 15:25:37 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 15:25:37 -0500
Subject: [petsc-users] Complex vector / matrix to VTK
In-Reply-To: <5507113C.9020808@gmail.com>
References: <5507113C.9020808@gmail.com>
Message-ID: <9EFBCCC0-8D99-49B9-999C-83AC41E9B130@mcs.anl.gov>


> On Mar 16, 2015, at 12:22 PM, Juris Vencels <juris.vencels at gmail.com> wrote:
> 
> Hello,
> 
> I want to print vector / matrix to VTK file.
> 
> I am running simple tutorial example ex15.c, but since I have compiled PETSc with complex scalars (I need them for my program) I get an error below. Can I force PETSc to print only real, imaginary (or both) part(s) of the solution?
> 
> Another question - ksp/ex42.c explicitly writes header information to VTK file while snes/ex15.c has only one line - to open VTK file. How these two examples are different in terms of writing output to VTK?
> src/ksp/ksp/examples/tutorials/ex42.c
> src/snes/examples/tutorials/ex15.c

  Ex42 shows how you can provide all the information yourself for VTK while ex15 uses the PETSc library code to provide it automatically (of course if you do it all yourself you have complete control over what is saved but it is more work on your part). 

  I recommend trying to use the library version. We don't currently have code in the library to save just the real part, imaginary parts. If VTK has a concept of complex numbers you should read up on it and see how it works. 

  Depending on the vectors/DMs you are using (Plex, DMDA? something else?) you will need to go through where the code actually writes out the numerical values and adjust to handle what portions of the complex numbers you wish to save. In the case you ran below it looks like the routine
DMDAVTKWriteAll_VTS() in the file src/dm/impls/da/grvtk.c


  Barry

> 
> Thanks!
> Juris
> 
>> vencels at bigone:/scratch/petsc-git/src/snes/examples/tutorials$ ./ex15 -snes_monitor_short -da_grid_x 20 -da_grid_y 20 -p 1.3 -lambda 1 -jtype NEWTON -o ex15.vts
>>   0 SNES Function norm 0.690597 
>>   1 SNES Function norm 0.437399 
>>   2 SNES Function norm 0.309415 
>>   3 SNES Function norm 0.132358 
>>   4 SNES Function norm 0.0919276 
>>   5 SNES Function norm 0.0752639 
>>   6 SNES Function norm 0.0669195 
>>   7 SNES Function norm 0.0592599 
>>   8 SNES Function norm 0.0516658 
>>   9 SNES Function norm 0.0441565 
>>  10 SNES Function norm 0.0367695 
>>  11 SNES Function norm 0.0288595 
>>  12 SNES Function norm 0.0277417 
>>  13 SNES Function norm 0.0085536 
>>  14 SNES Function norm 0.00140906 
>>  15 SNES Function norm 7.35341e-05 
>>  16 SNES Function norm 3.63992e-07 
>>  17 SNES Function norm 1.034e-11 
>> CONVERGED_FNORM_RELATIVE Number of nonlinear iterations = 17
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Complex values not supported
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2213-g589572e  GIT Date: 2015-03-07 10:56:38 -0600
>> [0]PETSC ERROR: ./ex15 on a intelmkl-debug named bigone by vencels Mon Mar 16 10:45:52 2015
>> [0]PETSC ERROR: Configure options --with-cc=icc --with-cxx=icpc --with-fc=ifort --with-blas-lapack-dir=/opt/intel/mkl/ --with-fortran-kernels=generic --with-scalar-type=complex --with-mpi=0 --download-superlu
>> [0]PETSC ERROR: #1 DMDAVTKWriteAll_VTS() line 30 in /scratch/petsc-git/src/dm/impls/da/grvtk.c
>> [0]PETSC ERROR: #2 DMDAVTKWriteAll() line 482 in /scratch/petsc-git/src/dm/impls/da/grvtk.c
>> [0]PETSC ERROR: #3 PetscViewerFlush_VTK() line 78 in /scratch/petsc-git/src/sys/classes/viewer/impls/vtk/vtkv.c
>> [0]PETSC ERROR: #4 PetscViewerFlush() line 30 in /scratch/petsc-git/src/sys/classes/viewer/interface/flush.c
>> [0]PETSC ERROR: #5 PetscViewerDestroy() line 101 in /scratch/petsc-git/src/sys/classes/viewer/interface/view.c
>> [0]PETSC ERROR: #6 main() line 257 in /scratch/petsc-git/src/snes/examples/tutorials/ex15.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -da_grid_x 20
>> [0]PETSC ERROR: -da_grid_y 20
>> [0]PETSC ERROR: -jtype NEWTON
>> [0]PETSC ERROR: -lambda 1
>> [0]PETSC ERROR: -o ex15.vts
>> [0]PETSC ERROR: -p 1.3
>> [0]PETSC ERROR: -snes_monitor_short
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> Aborted (core dumped)
> 
> 


From bsmith at mcs.anl.gov  Wed Mar 18 15:30:22 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 15:30:22 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZjBxK2dhooAWzzbwK9aTqwJg=JOwHGZoPUo+EJ57SpoVg@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<87vbhynl21.fsf@jedbrown.org>
	<CABRkuZhBjO7eHMGnGsK=JCxqtAHkvrixSMr4WOuny27WQCRCZw@mail.gmail.com>
	<6A5DFFAE-D20E-45D3-BACA-2635F16C25A4@mcs.anl.gov>
	<CABRkuZjBxK2dhooAWzzbwK9aTqwJg=JOwHGZoPUo+EJ57SpoVg@mail.gmail.com>
Message-ID: <F5B6F22D-1539-437E-B863-D4B0C22E6A33@mcs.anl.gov>


> On Mar 18, 2015, at 3:14 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> 
> 
> On Wed, Mar 18, 2015 at 3:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 18, 2015, at 3:07 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > So I guess I misunderstood the meaning of " inplace conversion".
> > and what do MAT_INITIAL_MATRIX and MAT_REUSE_MATRIX do anyway?
> 
>   No you have not misunderstood it.
> 
> Currently
>    MAT_REUSE_MATRIX is only supported for inplace conversion, otherwise use
>    MAT_INITIAL_MATRIX.
> 
>   For testing purposes you can use MAT_INITIAL_MATRIX.
> 
> 
> I will try to convert the matrix in a new matrix using MAT_INITIAL_MATRIX. 
> while 
> MatConvert(J, MATAIJ, MAT_INITIAL_MATRIX, & A); 
> will convert the matrix and port the results into a new
> what does 
> MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> actually do?

   It puts the result into the J matrix you passed in, "replacing" the matrix that was there.

  Barry

> 
> Thanks,
>  
> 
> >
> >
> >
> >
> > On Wed, Mar 18, 2015 at 2:48 PM, Jed Brown <jed at jedbrown.org> wrote:
> > Chung-Kan Huang <ckhuangf at gmail.com> writes:
> >
> > > Yes, I was suprised that I can still use MatSetValuesBlocked after I did
> > > MatConvert MATBAIJ to MATAIJ.
> > >
> > > Then I just found the mat type is still seqbaij which means the MatConvert
> > > didn't work.
> > >
> > > This is how I did MatConvert
> > >
> > > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> > >
> > > where J is my Jacobian matrix.
> >
> > Seems to me this should fail because you didn't MatConvert this matrix
> > yet.  Anyway, you should use MAT_INITIAL_MATRIX and put the result in a
> > new Mat.
> >
> >
> >
> > --
> > Cheers
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From bsmith at mcs.anl.gov  Wed Mar 18 15:30:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 15:30:46 -0500
Subject: [petsc-users] Questions about MatConvert BAIJ to AIJ
In-Reply-To: <CABRkuZi-vpwa81_VtdaaZ1Xy-bhiihFWFEiJRCqnBNVHbZuPMg@mail.gmail.com>
References: <CABRkuZjLfiYCBsv5SVhQ8prKFUOt2Kk_zLQnwuAAPYya63Scog@mail.gmail.com>
	<9DBD06B1-E72A-435B-AF2F-7406A681A887@mcs.anl.gov>
	<CABRkuZi3-zBvD1Ts8TA4mzpNofLdJidoKUxr_BWSPy4uoY8m9Q@mail.gmail.com>
	<87861EB9-2B65-446D-B867-3EC26E7C04B9@mcs.anl.gov>
	<CABRkuZjtBfeFZGoBki08Aq-a97K8tbRnFGqFcUxUBJg67YznFw@mail.gmail.com>
	<0EC9DD75-0289-45D2-BB5D-3BF7CC0759C2@mcs.anl.gov>
	<CABRkuZiyp6FBio2SjYQuQnLU2Le08_dKHeYiEeK4FP_VKuagsw@mail.gmail.com>
	<D0C32243-704B-4322-874D-7A174E9D1D15@mcs.anl.gov>
	<CABRkuZi-vpwa81_VtdaaZ1Xy-bhiihFWFEiJRCqnBNVHbZuPMg@mail.gmail.com>
Message-ID: <C262E49C-3846-4320-B6CA-128ABB449B24@mcs.anl.gov>


> On Mar 18, 2015, at 3:10 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> 
> 
> On Wed, Mar 18, 2015 at 2:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    You can use MatSetValuesBlocked() for AIJ matrices, obviously the matrix needs to have the block size
> 
> Barry,
> I don't think that is the problem.  The problem should be like Jed said, the MatConvert didn't actually work.  
> Now we can focus on how to make it work in my case.

   Just print the three matrices and confirm if they are different or the same.

> 
>   Barry
> 
> > On Mar 18, 2015, at 2:46 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> >
> > Yes, I was suprised that I can still use MatSetValuesBlocked after I did MatConvert MATBAIJ to MATAIJ.
> >
> > Then I just found the mat type is still seqbaij which means the MatConvert didn't work.
> >
> > This is how I did MatConvert
> >
> > MatConvert(J, MATAIJ, MAT_REUSE_MATRIX, & J);
> >
> > where J is my Jacobian matrix.
> >
> > Thanks,
> >
> >
> > On Wed, Mar 18, 2015 at 2:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Are you always using MatSetValuesBlocked()?
> >
> >   Did you look at the three matrices (before passing to the solve), do they all have the exact same nonzero structure and entries?
> >
> >   Barry
> >
> > > On Mar 18, 2015, at 1:45 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> > >
> > > So go back to my original question,
> > >
> > > I found the resuls look like
> > > MATBAIJ == MATBAIJ2MATAIJ != MATAIJ
> > > as I was expected to see
> > > MATBAIJ != MATBAIJ2MATAIJ == MATAIJ
> > >
> > > because I thought MATBAIJ2MATAIJ & MATAIJ should use the same ILU factorization.
> > >
> > > On Wed, Mar 18, 2015 at 1:33 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > >    As I think I said earlier: ILU will behave differently with BAIJ and AIJ format. With BAIJ it uses a point block factorization (doing an ILU treating each block in the matrix as a single point (and doing a dense factorization for the inverse) while AIJ ignores the block size and just uses a point ILU. So in general the convergence with BAIJ vs AIJ and ILU will be different; usually BAIJ will converge better but it need not.
> > >
> > >    Barry
> > >
> > > > On Mar 18, 2015, at 1:23 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> > > >
> > > > Sorry I didn't state my problem more accurate.  The entries in the input matrixes and vectors? are correct -- they have right values and correct nonzero structure.
> > > > The KSP was setup so I can solve the lienar systems for an unsteady state convective problem.
> > > > I use KSPBCGS + PCILU and so far I only try sequential on a very small problem.
> > > > I wonder if blocked ilu is actually used in  MATBAIJ2MATAIJ case.
> > > >
> > > > Thanks,
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > >
> >
> >
> >
> >
> > --
> > Cheers
> >
> 
> 
> 
> 
> -- 
> Cheers
> 


From ansp6066 at colorado.edu  Wed Mar 18 15:41:56 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Wed, 18 Mar 2015 13:41:56 -0700 (PDT)
Subject: [petsc-users] HermitianTranspose version of MatCreateTranspose.
In-Reply-To: <F38B56C7-7CB7-4301-A231-FD556FB16A6A@mcs.anl.gov>
References: <F38B56C7-7CB7-4301-A231-FD556FB16A6A@mcs.anl.gov>
Message-ID: <1426711316683.8dd3ac42@Nodemailer>

So, I?ve got a MatCreateHermitianTranspose function that has close to the same functionality as the MatCreatTranspose version. ?So I?m getting ready to send a pull request.




A few questions:




What branch should I add my changes to?




Should I create tests for this? ?Where should I put them?




On Monday, Feb 23, 2015 at 9:02 PM, Barry Smith <bsmith at mcs.anl.gov>, wrote:


   We've had a small amount of debate over the years on how to handle the Hermitian transpose and non-Hermitian transpose that never got fully resolved.


Approach 1) Each (complex) matrix has a full set of transpose and Hermitian transpose operations (MatTranspose(), MatHermitianTranspose(), MatMultTranspose()), MatMultHermitianTranspose(), MatSolveTranspose(), MatSolveHermitianTranspose(), MatMatMultTranspose(), MatMatMultHermitianTranspose(), MatTranposeMatMult(), MatHermitianTransposeMatMult().......)  plus there are two vector "inner" products; VecDot() and VecTDot(). 


Approach 2) Consider a (complex) vector (and hence the associated matrix operators on it) to live in the usual Hermitian inner product space or the non-Hermitian "inner product space". Then one only needs a single VecDot() and MatTranspose(), MatMultTranspose() ... that just "does the right thing" based on what space the user has declared the vectors/matrices to be in. 


Approach 2) seems nicer since it only requires 1/2 the functions :-) and so long as the two vector "spaces" never interact directly (for example what would be the meaning of the "inner" product of a vector in the usual Hermitian inner product space with a vector from the non-Hermitian "inner product space"?) certain seems simpler.  Approach 1) might be simpler for some people who like to always see exactly what they are doing. 


I personally wish I had started with Approach 2 (but I did not), but there could be some flaw with it I am not seeing.


  Barry








> On Feb 23, 2015, at 6:50 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:

> 

> I?m definitely willing to submit it as a pull request.

> 

> Also, while I?m at it, I?m going to write a ?duplicate? function for transpose and hermitian_transpose.  Just because this seems 1) easy ( MatHermitianTranspose can return a new copy, as well as MatTranspose), and 2) necessary to use these for EPS.

> 

> Also, is ?transpose? a good enough MatType?  Or does a new one need to be written?

> 

> -Andrew

> 

> 

> 

> On Mon, Feb 23, 2015 at 3:12 PM, Jed Brown <jed at jedbrown.org> wrote:

> 

> <signature.asc>

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/c57d2e04/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 18 16:10:47 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 16:10:47 -0500
Subject: [petsc-users] HermitianTranspose version of MatCreateTranspose.
In-Reply-To: <1426711316683.8dd3ac42@Nodemailer>
References: <F38B56C7-7CB7-4301-A231-FD556FB16A6A@mcs.anl.gov>
	<1426711316683.8dd3ac42@Nodemailer>
Message-ID: <1653360C-A2CA-40EA-B48C-763D9BFA3DF9@mcs.anl.gov>


> On Mar 18, 2015, at 3:41 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
> 
> So, I?ve got a MatCreateHermitianTranspose function that has close to the same functionality as the MatCreatTranspose version.  So I?m getting ready to send a pull request.
> 
  Great

> A few questions:
> 
> What branch should I add my changes to?

master

> 
> Should I create tests for this?

most definitely

>  Where should I put them?

src/mat/examples/tests

> 
> On Monday, Feb 23, 2015 at 9:02 PM, Barry Smith <bsmith at mcs.anl.gov>, wrote:
> 
> We've had a small amount of debate over the years on how to handle the Hermitian transpose and non-Hermitian transpose that never got fully resolved. 
> 
> Approach 1) Each (complex) matrix has a full set of transpose and Hermitian transpose operations (MatTranspose(), MatHermitianTranspose(), MatMultTranspose()), MatMultHermitianTranspose(), MatSolveTranspose(), MatSolveHermitianTranspose(), MatMatMultTranspose(), MatMatMultHermitianTranspose(), MatTranposeMatMult(), MatHermitianTransposeMatMult().......) plus there are two vector "inner" products; VecDot() and VecTDot(). 
> 
> Approach 2) Consider a (complex) vector (and hence the associated matrix operators on it) to live in the usual Hermitian inner product space or the non-Hermitian "inner product space". Then one only needs a single VecDot() and MatTranspose(), MatMultTranspose() ... that just "does the right thing" based on what space the user has declared the vectors/matrices to be in. 
> 
> Approach 2) seems nicer since it only requires 1/2 the functions :-) and so long as the two vector "spaces" never interact directly (for example what would be the meaning of the "inner" product of a vector in the usual Hermitian inner product space with a vector from the non-Hermitian "inner product space"?) certain seems simpler. Approach 1) might be simpler for some people who like to always see exactly what they are doing. 
> 
> I personally wish I had started with Approach 2 (but I did not), but there could be some flaw with it I am not seeing. 
> 
> Barry 
> 
> 
> 
> 
> 
> 
> 
> > On Feb 23, 2015, at 6:50 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > 
> > I?m definitely willing to submit it as a pull request. 
> > 
> > Also, while I?m at it, I?m going to write a ?duplicate? function for transpose and hermitian_transpose. Just because this seems 1) easy ( MatHermitianTranspose can return a new copy, as well as MatTranspose), and 2) necessary to use these for EPS. 
> > 
> > Also, is ?transpose? a good enough MatType? Or does a new one need to be written? 
> > 
> > -Andrew 
> > 
> > 
> > 
> > On Mon, Feb 23, 2015 at 3:12 PM, Jed Brown <jed at jedbrown.org> wrote: 
> > 
> > <signature.asc> 
> > 
> 


From knepley at gmail.com  Wed Mar 18 16:16:59 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 18 Mar 2015 15:16:59 -0600
Subject: [petsc-users] Fast LU solver for block diagonal matrix
In-Reply-To: <5509DBB1.4020108@columbi.edu>
References: <5509C6CB.7030703@columbi.edu>
	<DFF02462-0F9C-4E54-8876-719C86E8545A@mcs.anl.gov>
	<5509D807.2000809@columbi.edu>
	<19406F78-9CBC-42D8-B5AC-47635D833F42@mcs.anl.gov>
	<5509DBB1.4020108@columbi.edu>
Message-ID: <CAMYG4Gmqnzp1xgMZpKjHU0=t7bSaL-JRCxwtFTzHF3rX0uYBXw@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:10 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
wrote:

> I will have to do a MatMatMatMult or two MatMatMult since J is unsymmetric
> (Jeo != Joe^T).
> I think two MatMatMult could be more efficient if I use the
> MatMatMultSymbolic/MatMatMultNumeric correctly.
>

Test it first using

  -pc_fieldsplit_schur_precondition full

which does what you want, although maybe not in an optimal way:


https://bitbucket.org/petsc/petsc/src/f412d3e7cd17c738b1d11aea53d16ae27a4a5abb/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-777

  Thanks,

    Matt

Best,
> Luc
>
>
> On 03/18/2015 04:04 PM, Barry Smith wrote:
>
>> On Mar 18, 2015, at 2:54 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
>>> wrote:
>>>
>>> Hi Barry,
>>> I would like to compute S explicitly, I have a few good option to
>>> precondition S but they are based on S not an approximation of S (i.e. I
>>> don't want to compute my preconditioner using Sp different from S).
>>>
>>> Also Jee is obtained using MatGetSubmatrix(J,isrow_e,iscol_e) and J is
>>> an AIJ matrix so I assume that Jee is AIJ too.
>>> I can convert Jee to BAIJ to take advantage of the block structure but
>>> from your conversation with Chung-Kan it might require to reassemble?
>>> In that case what about the following:
>>> MatCreateBAIJ(comm, 5, Jee_inv)
>>> for(i=0,i<nblocks,i++)
>>> {
>>>      MatGetValues(Jee,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4],
>>> block_values)
>>>      some_package_inverts( block_values )
>>>      MatSetValuesBlocked(Jee_inv,5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+
>>> 4],5,[5*i+0,5*i+1,5*i+2,5*i+3,5*i+4], block_values)
>>> }
>>> MatAssemblyBegin()
>>> MatAssemblyEnd()
>>> With this I could then just go on and do matrix multiplications to
>>> finish my Schur complement calculations.
>>> Would this be decently efficient?
>>>
>>     This would be fine. Then you can use MatPtAP or MatRARt to compute
>> the triple matrix product.
>>
>>    Barry
>>
>>  Best,
>>> Luc
>>>
>>> On 03/18/2015 03:22 PM, Barry Smith wrote:
>>>
>>>>    Do you want to explicitly compute (as a matrix) S = Joo - joe *
>>>> inv(Jee) jeo   or do you want to just have an efficient computation of
>>>> S y  for any y vector?
>>>>
>>>>    Here is some possibly useful information. If you create Jee as a
>>>> BAIJ matrix of block size 5 and use MatILUFactor() it will efficiently
>>>> factor this matrix (each 5 by 5 block factorization is done with custom
>>>> code) then you can use MatSolve() efficiently with the result (note that
>>>> internally when factoring a BAIJ matrix PETSc actually stores the inverse
>>>> of the diagonal blocks so in your case the MatSolve() actually ends up
>>>> doing little matrix-vector products (and there are no triangular solves).
>>>>
>>>>    To use this with the MatCreateSchurComplement() object you can do
>>>>
>>>>     MatCreateSchurComplement(...,&S)
>>>>     MatSchurComplementGetKSP(S,&ksp)
>>>>     KSPSetType(ksp,KSPPREONLY);
>>>>
>>>>     now MatMult(S,y,z) will be efficient.
>>>>
>>>>     Of course you still have the question, how do you plan to solve S?
>>>> This depends on its structure and if you have a good way of preconditioning
>>>> it.
>>>>
>>>>     If you want to explicitly form S you can use MatMatSolve( fact,jeo)
>>>> but this requires making jeo dense which seems to defeat the purpose.
>>>>
>>>>    Barry
>>>>
>>>>
>>>>
>>>>
>>>>  On Mar 18, 2015, at 1:41 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
>>>>>   wrote:
>>>>>
>>>>> Hi all,
>>>>> I am solving multi-physics problem that leads to a jacobian of the
>>>>> form:
>>>>>
>>>>> [ Jee  Jeo ]
>>>>> [ Joe  Joo ]
>>>>>
>>>>> where Jee is 5by5 block diagonal. This feature makes it a very good
>>>>> candidate for a Schur complement.
>>>>> Indeed, Jee could be inverted in parallel with no inter-nodes
>>>>> communication.
>>>>> My only issue is the fact that the Schur complement is not accessible
>>>>> directly with PETSC, only an approximation is available, for direct solvers
>>>>> (usually S~Joo or S~Joo-Joe* diag(Jee)^-1 *Jeo).
>>>>>
>>>>> Any advice on how I could efficiently compute Jee^-1 for the given
>>>>> structure?
>>>>> I am currently thinking about hard coding the formula for the inverse
>>>>> of a 5by5 and sweeping through Jee (with some threading) and storing the
>>>>> inverse in-place. Instead of hard coding the formula for a 5by5 I could
>>>>> also do a MatLUFactorSym on a 5by5 matrix but it would not give me an
>>>>> inverse, only a factorization...
>>>>>
>>>>> Thanks in advance for your suggestions!
>>>>>
>>>>> --
>>>>> Best,
>>>>> Luc
>>>>>
>>>>>
>>>>>
>>>>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150318/60f33fb1/attachment-0001.html>

From a.croucher at auckland.ac.nz  Wed Mar 18 21:06:12 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Thu, 19 Mar 2015 15:06:12 +1300
Subject: [petsc-users] FD Dirichlet boundary conditions with ghosted DMDA
Message-ID: <550A2F14.3010007@auckland.ac.nz>

hi,

I'm setting up a simple 1-D finite difference test problem with 
Dirichlet boundary conditions at each end, using DMDA.

I used DMDACreate1d() with the DM_BOUNDARY_GHOSTED option, and set the 
global array dimension to be the total number of nodes (including 
boundary nodes) minus 2- so the boundary nodes aren't included in the mesh.

When I'm evaluating the differential equation on this mesh, I use 
DMGetLocalVector() and DMGlobalToLocalBegin/End() to scatter to my 
ghosted vector, then DMDAVecGetArray() to get an array on the local vector.

When I want to apply the BCs at each end, I call DMDAGetGhostCorners() 
to find the indices of the ghost points.  I can now plug the required 
boundary values into my array.

At present, if I'm running on multiple processors, I'm just checking if 
rank=0 or rank=size-1 to make sure the BCs only get applied at the 
actual ends of the mesh, and not at the ends of the internal partitions 
as well.

This works, but is there a better way to do it? It seems potentially 
unreliable- as it assumes rank 0 is always the left hand end and rank 
size-1 always the right-hand end. Also it won't really work in 2D or 3D.

Thanks!

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611


From bsmith at mcs.anl.gov  Wed Mar 18 21:48:11 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 18 Mar 2015 21:48:11 -0500
Subject: [petsc-users] FD Dirichlet boundary conditions with ghosted DMDA
In-Reply-To: <550A2F14.3010007@auckland.ac.nz>
References: <550A2F14.3010007@auckland.ac.nz>
Message-ID: <D6FB1611-BE67-4732-9E19-BC9B2A15D359@mcs.anl.gov>


  Use either the results for DMDAGetGhostCorners() or DMDAGetCorners() to determine if the first or last node is local. This extends trivially to 2 and 3d. See for example src/ksp/ksp/examples/tutorials/ex34.c ComputeMatrix() if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1) { etc

  Barry

> On Mar 18, 2015, at 9:06 PM, Adrian Croucher <a.croucher at auckland.ac.nz> wrote:
> 
> hi,
> 
> I'm setting up a simple 1-D finite difference test problem with Dirichlet boundary conditions at each end, using DMDA.
> 
> I used DMDACreate1d() with the DM_BOUNDARY_GHOSTED option, and set the global array dimension to be the total number of nodes (including boundary nodes) minus 2- so the boundary nodes aren't included in the mesh.
> 
> When I'm evaluating the differential equation on this mesh, I use DMGetLocalVector() and DMGlobalToLocalBegin/End() to scatter to my ghosted vector, then DMDAVecGetArray() to get an array on the local vector.
> 
> When I want to apply the BCs at each end, I call DMDAGetGhostCorners() to find the indices of the ghost points.  I can now plug the required boundary values into my array.
> 
> At present, if I'm running on multiple processors, I'm just checking if rank=0 or rank=size-1 to make sure the BCs only get applied at the actual ends of the mesh, and not at the ends of the internal partitions as well.
> 
> This works, but is there a better way to do it? It seems potentially unreliable- as it assumes rank 0 is always the left hand end and rank size-1 always the right-hand end. Also it won't really work in 2D or 3D.
> 
> Thanks!
> 
> - Adrian
> 
> -- 
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
> 


From a.croucher at auckland.ac.nz  Wed Mar 18 22:28:24 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Thu, 19 Mar 2015 16:28:24 +1300
Subject: [petsc-users] FD Dirichlet boundary conditions with ghosted DMDA
In-Reply-To: <D6FB1611-BE67-4732-9E19-BC9B2A15D359@mcs.anl.gov>
References: <550A2F14.3010007@auckland.ac.nz>
	<D6FB1611-BE67-4732-9E19-BC9B2A15D359@mcs.anl.gov>
Message-ID: <550A4258.7080307@auckland.ac.nz>


>    Use either the results for DMDAGetGhostCorners() or DMDAGetCorners() to determine if the first or last node is local. This extends trivially to 2 and 3d. See for example src/ksp/ksp/examples/tutorials/ex34.c ComputeMatrix() if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1) { etc

Aha, of course. Thanks, that works great.

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611


From haakon at hakostra.net  Thu Mar 19 04:19:53 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Thu, 19 Mar 2015 10:19:53 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset
Message-ID: <550A94B9.90407@hakostra.net>

Hi,

I am again in trouble with PETSc and its HDF5-related functions.

I have a code in which I want to read a 1D dataset from a HDF5 file into 
a PETSc Vec. The array are "small", and each MPI process should read the 
entire dataset and store it in its own sequential Vec.

Currently my code is as follows (somewhat simplified):

VecCreate(PETSC_COMM_SELF, &v);
VecSetType(v, VECSEQ);
PetscObjectSetName((PetscObject)v, "x");
PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
VecLoad(v, viewer);

This works, but only if the dataset in the HDF5 file is a 2D dataset, of 
size Nx1, where N is the number of elements I read. If the dataset in 
the HDF5 file is of size N (i.e. a 1D dataset) this does not work.

The error message I get is the following:
[0]PETSC ERROR: Unexpected data in file
[0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
etc. and it points back to the line 296 in file vecio.c, function 
VecLoad_HDF5().

Technically this is not a problem. I could of course just go ahead and 
"reshape" all my datasets into 2D datasets. I do however find this 
behaviour very un-logical. Why can't I read a 1D dataset into a Vec?

So my question are:
1) Do I do anything wrong when creating my Vec and reading my datasets?
2) Is there anything I can do to read my 1D datasets into Vecs?
3) Could PETSc perhaps be a little less restrictive in these cases?

Thanks in advance.

Regards,
H?kon Strandenes

From M.Imtiaz at victorchang.edu.au  Thu Mar 19 04:42:54 2015
From: M.Imtiaz at victorchang.edu.au (Mohammad Imtiaz)
Date: Thu, 19 Mar 2015 09:42:54 +0000
Subject: [petsc-users] Position based coupling matrix
In-Reply-To: <675B979E-1190-4F63-B825-E8CE52093A38@mcs.anl.gov>
References: <D12E5261.D4DB%m.imtiaz@victorchang.edu.au>
	<675B979E-1190-4F63-B825-E8CE52093A38@mcs.anl.gov>
Message-ID: <D130E46D.D620%m.imtiaz@victorchang.edu.au>

Thank you, I am trying this now.

One more question:

I am my assigning my coupling matrix R as one of the state variable in the
DMDA passed to RHS function even though R does not change once
initialized.  Is this the most efficient way to do this?

Thanks
Mohammad






On 18/03/2015 2:10 am, "Barry Smith" <bsmith at mcs.anl.gov> wrote:

>
>  You will need to associate the vector with a DMDA so you can use
>DMDAVecGetArray() to index into it. Use DMCreateGlobalVector (with the
>appropriate DMDA) to get the vector instead of VecCreate() before you do
>the VecLoad().
>
>  Barry
>
>
>> On Mar 17, 2015, at 6:15 AM, Mohammad Imtiaz
>><M.Imtiaz at victorchang.edu.au> wrote:
>> 
>> Hi,
>> 
>> I am trying to solve a 2 dimensional finite difference reaction
>>diffusion problem where diffusional coupling is position based, i.e.
>>fibres, discontinuities etc occur due to material properties (e.g. Heart
>>tissue).
>> 
>> I have managed to import a matlab matrix representing coupling in four
>>directions as below:
>> 
>>     PetscViewerBinaryOpen(PETSC_COMM_WORLD,?R",FILE_MODE_READ,&fd);
>>     VecCreate(PETSC_COMM_WORLD,&R);
>>     VecLoad(R,fd);
>> 
>> ?..Repeated to get L, U, D, coupling.
>> 
>> How can I now make R,L,U,D available to my finite difference part of
>>the code? So that I can do
>> 
>> 
>> v is field.
>> 
>> vxx = u[j][i].R*(u[j][i-1].v - u[j][i].v)  + u[j][i].L*(u[j][i+1].v ?
>>u[j][i].v);
>> 
>> And similarly for up & down coupling.
>> 
>> 
>> 
>> 
>> 
>> Thanks
>> Mohammad Imtiaz
>
>Scanned by Messagelabs ***


From bsmith at mcs.anl.gov  Thu Mar 19 12:00:41 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 19 Mar 2015 12:00:41 -0500
Subject: [petsc-users] Position based coupling matrix
In-Reply-To: <D130E46D.D620%m.imtiaz@victorchang.edu.au>
References: <D12E5261.D4DB%m.imtiaz@victorchang.edu.au>
	<675B979E-1190-4F63-B825-E8CE52093A38@mcs.anl.gov>
	<D130E46D.D620%m.imtiaz@victorchang.edu.au>
Message-ID: <75F820C4-59E0-4961-8E41-079938CF67B0@mcs.anl.gov>


> On Mar 19, 2015, at 4:42 AM, Mohammad Imtiaz <M.Imtiaz at victorchang.edu.au> wrote:
> 
> Thank you, I am trying this now.
> 
> One more question:
> 
> I am my assigning my coupling matrix R as one of the state variable in the
> DMDA passed to RHS function even though R does not change once
> initialized.

   Not sure what you mean by this.
>  Is this the most efficient way to do this?

  So long as you are just passing around the Mat R then it is efficient, since R is just a pointer to the data structure. If you are calling MatDuplicate() on R or  creating a new R repeatedly then no it would not be efficient.

   Barry

> 
> Thanks
> Mohammad
> 
> 
> 
> 
> 
> 
> On 18/03/2015 2:10 am, "Barry Smith" <bsmith at mcs.anl.gov> wrote:
> 
>> 
>> You will need to associate the vector with a DMDA so you can use
>> DMDAVecGetArray() to index into it. Use DMCreateGlobalVector (with the
>> appropriate DMDA) to get the vector instead of VecCreate() before you do
>> the VecLoad().
>> 
>> Barry
>> 
>> 
>>> On Mar 17, 2015, at 6:15 AM, Mohammad Imtiaz
>>> <M.Imtiaz at victorchang.edu.au> wrote:
>>> 
>>> Hi,
>>> 
>>> I am trying to solve a 2 dimensional finite difference reaction
>>> diffusion problem where diffusional coupling is position based, i.e.
>>> fibres, discontinuities etc occur due to material properties (e.g. Heart
>>> tissue).
>>> 
>>> I have managed to import a matlab matrix representing coupling in four
>>> directions as below:
>>> 
>>>    PetscViewerBinaryOpen(PETSC_COMM_WORLD,?R",FILE_MODE_READ,&fd);
>>>    VecCreate(PETSC_COMM_WORLD,&R);
>>>    VecLoad(R,fd);
>>> 
>>> ?..Repeated to get L, U, D, coupling.
>>> 
>>> How can I now make R,L,U,D available to my finite difference part of
>>> the code? So that I can do
>>> 
>>> 
>>> v is field.
>>> 
>>> vxx = u[j][i].R*(u[j][i-1].v - u[j][i].v)  + u[j][i].L*(u[j][i+1].v ?
>>> u[j][i].v);
>>> 
>>> And similarly for up & down coupling.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Thanks
>>> Mohammad Imtiaz
>> 
>> Scanned by Messagelabs ***


From bsmith at mcs.anl.gov  Thu Mar 19 12:17:06 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 19 Mar 2015 12:17:06 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset
In-Reply-To: <550A94B9.90407@hakostra.net>
References: <550A94B9.90407@hakostra.net>
Message-ID: <90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>


  Please email the file in.h5 so we can try to reproduce the problem and debug it. Are you using petsc version 3.5.? 

  Barry

> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> 
> Hi,
> 
> I am again in trouble with PETSc and its HDF5-related functions.
> 
> I have a code in which I want to read a 1D dataset from a HDF5 file into a PETSc Vec. The array are "small", and each MPI process should read the entire dataset and store it in its own sequential Vec.
> 
> Currently my code is as follows (somewhat simplified):
> 
> VecCreate(PETSC_COMM_SELF, &v);
> VecSetType(v, VECSEQ);
> PetscObjectSetName((PetscObject)v, "x");
> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
> VecLoad(v, viewer);
> 
> This works, but only if the dataset in the HDF5 file is a 2D dataset, of size Nx1, where N is the number of elements I read. If the dataset in the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
> 
> The error message I get is the following:
> [0]PETSC ERROR: Unexpected data in file
> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
> etc. and it points back to the line 296 in file vecio.c, function VecLoad_HDF5().
> 
> Technically this is not a problem. I could of course just go ahead and "reshape" all my datasets into 2D datasets. I do however find this behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
> 
> So my question are:
> 1) Do I do anything wrong when creating my Vec and reading my datasets?
> 2) Is there anything I can do to read my 1D datasets into Vecs?
> 3) Could PETSc perhaps be a little less restrictive in these cases?
> 
> Thanks in advance.
> 
> Regards,
> H?kon Strandenes


From haakon at hakostra.net  Thu Mar 19 12:31:04 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Thu, 19 Mar 2015 18:31:04 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset
In-Reply-To: <90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
Message-ID: <550B07D8.9060903@hakostra.net>

I'm sorry I didn't include this in the first email. Here is an example 
HDF5-file, together with a sample program that demonstrate the problem.

I have been trying to dig into VecLoad_HDF5() in vecio.c in order to 
suggest a solution myself, however, I do not think I fully understand 
the concept of the Vec block size and how that affects how the Vec is 
stored in memory and on disk (HDF5 file).

Regards,
H?kon Strandenes


On 19. mars 2015 18:17, Barry Smith wrote:
>
>    Please email the file in.h5 so we can try to reproduce the problem and debug it. Are you using petsc version 3.5.?
>
>    Barry
>
>> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
>>
>> Hi,
>>
>> I am again in trouble with PETSc and its HDF5-related functions.
>>
>> I have a code in which I want to read a 1D dataset from a HDF5 file into a PETSc Vec. The array are "small", and each MPI process should read the entire dataset and store it in its own sequential Vec.
>>
>> Currently my code is as follows (somewhat simplified):
>>
>> VecCreate(PETSC_COMM_SELF, &v);
>> VecSetType(v, VECSEQ);
>> PetscObjectSetName((PetscObject)v, "x");
>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
>> VecLoad(v, viewer);
>>
>> This works, but only if the dataset in the HDF5 file is a 2D dataset, of size Nx1, where N is the number of elements I read. If the dataset in the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
>>
>> The error message I get is the following:
>> [0]PETSC ERROR: Unexpected data in file
>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
>> etc. and it points back to the line 296 in file vecio.c, function VecLoad_HDF5().
>>
>> Technically this is not a problem. I could of course just go ahead and "reshape" all my datasets into 2D datasets. I do however find this behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
>>
>> So my question are:
>> 1) Do I do anything wrong when creating my Vec and reading my datasets?
>> 2) Is there anything I can do to read my 1D datasets into Vecs?
>> 3) Could PETSc perhaps be a little less restrictive in these cases?
>>
>> Thanks in advance.
>>
>> Regards,
>> H?kon Strandenes
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VecLoadTest.tar.gz
Type: application/gzip
Size: 2183 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150319/39fdaf0f/attachment.bin>

From ckhuangf at gmail.com  Thu Mar 19 14:17:43 2015
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Thu, 19 Mar 2015 14:17:43 -0500
Subject: [petsc-users] USE_HASH_TABLE
Message-ID: <CABRkuZhUUG+DfV2kfwbqmkd6WBFrpgDF6Rikv6UXXaFUNM3Dyw@mail.gmail.com>

Hi,

I was looking for some performance enhancements around the linear solver
area and I was curious about this USE_HASH_TABLE option.

I wonder what scale of improvement I should expect and I got follow error
during my first attempt.

[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatSetValuesBlocked_MPIBAIJ_HT line 470
/d/dev01/chungh/lib2/petsc-3.5.2/src/mat/impls/baij/mpi/mpibaij.c
[0]PETSC ERROR: [0] MatSetValuesBlocked line 1636
/d/dev01/chungh/lib2/petsc-3.5.2/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------


I wonder anything I didn't do right..

Thanks,

Kan

-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150319/6ea24134/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar 19 14:42:53 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 19 Mar 2015 14:42:53 -0500
Subject: [petsc-users] USE_HASH_TABLE
In-Reply-To: <CABRkuZhUUG+DfV2kfwbqmkd6WBFrpgDF6Rikv6UXXaFUNM3Dyw@mail.gmail.com>
References: <CABRkuZhUUG+DfV2kfwbqmkd6WBFrpgDF6Rikv6UXXaFUNM3Dyw@mail.gmail.com>
Message-ID: <77D267EE-0D1C-44A5-963F-8EABDCB0A6AD@mcs.anl.gov>


  That code was just an experiment and doesn't work. 

  Barry

> On Mar 19, 2015, at 2:17 PM, Chung-Kan Huang <ckhuangf at gmail.com> wrote:
> 
> Hi,
> 
> I was looking for some performance enhancements around the linear solver area and I was curious about this USE_HASH_TABLE option.
> 
> I wonder what scale of improvement I should expect and I got follow error during my first attempt.
> 
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatSetValuesBlocked_MPIBAIJ_HT line 470 /d/dev01/chungh/lib2/petsc-3.5.2/src/mat/impls/baij/mpi/mpibaij.c
> [0]PETSC ERROR: [0] MatSetValuesBlocked line 1636 /d/dev01/chungh/lib2/petsc-3.5.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> 
> 
> I wonder anything I didn't do right..
> 
> Thanks,
> 
> Kan
> 
> -- 
> Cheers
> 


From bsmith at mcs.anl.gov  Thu Mar 19 21:00:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 19 Mar 2015 21:00:46 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <550B07D8.9060903@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
Message-ID: <6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>


  MATT READ THIS EMAIL!

  Ok here is the scoop.  In  VecView_MPI_HDF5() is the code (use meta x vc-annotate in emacs)

df09a399 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2011-01-20 16:27:23 -0600  682)   if (bs >= 1) {
272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  683)     dims[dim]      = bs;
272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  684)     maxDims[dim]   = dims[dim];
61ea93a1 src/vec/vec/impls/mpi/pdvec.c (Brad Aagaard         2010-11-06 15:31:18 -0700  685)     chunkDims[dim] = dims[dim];
43c8710a src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-11-05 19:21:58 -0500  686)     ++dim;
5464dd29 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-10-31 19:04:52 -0500  687)   }  if (bs >= 1) {

  then in PetscErrorCode VecLoad_HDF5(Vec xin, PetscViewer viewer)

  f6e5521d src/vec/vec/utils/vecio.c     (Karl Rupp            2013-01-28 13:32:07 -0600 277) if (bs >= 1) ++dim;

I check the commit message for df09a399 at bitbucket.com/petsc/petsc/ and it is 

Small change to HDF5 output (now Vec always has blocksize dim)

So what is happening is even the simplest vector (with a block size of 1) is treated as a 2d HDF5 object 

Matt, what was the rational for making vectors always have a dimension of block size even when it is one? Seems bad to me. Maybe makes HDF5 readers simpler since they don't need to handle bs 1 differently?

 Barry




> On Mar 19, 2015, at 12:31 PM, H?kon Strandenes <haakon at hakostra.net> wrote:
> 
> I'm sorry I didn't include this in the first email. Here is an example HDF5-file, together with a sample program that demonstrate the problem.
> 
> I have been trying to dig into VecLoad_HDF5() in vecio.c in order to suggest a solution myself, however, I do not think I fully understand the concept of the Vec block size and how that affects how the Vec is stored in memory and on disk (HDF5 file).
> 
> Regards,
> H?kon Strandenes
> 
> 
> On 19. mars 2015 18:17, Barry Smith wrote:
>> 
>>   Please email the file in.h5 so we can try to reproduce the problem and debug it. Are you using petsc version 3.5.?
>> 
>>   Barry
>> 
>>> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
>>> 
>>> Hi,
>>> 
>>> I am again in trouble with PETSc and its HDF5-related functions.
>>> 
>>> I have a code in which I want to read a 1D dataset from a HDF5 file into a PETSc Vec. The array are "small", and each MPI process should read the entire dataset and store it in its own sequential Vec.
>>> 
>>> Currently my code is as follows (somewhat simplified):
>>> 
>>> VecCreate(PETSC_COMM_SELF, &v);
>>> VecSetType(v, VECSEQ);
>>> PetscObjectSetName((PetscObject)v, "x");
>>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
>>> VecLoad(v, viewer);
>>> 
>>> This works, but only if the dataset in the HDF5 file is a 2D dataset, of size Nx1, where N is the number of elements I read. If the dataset in the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
>>> 
>>> The error message I get is the following:
>>> [0]PETSC ERROR: Unexpected data in file
>>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
>>> etc. and it points back to the line 296 in file vecio.c, function VecLoad_HDF5().
>>> 
>>> Technically this is not a problem. I could of course just go ahead and "reshape" all my datasets into 2D datasets. I do however find this behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
>>> 
>>> So my question are:
>>> 1) Do I do anything wrong when creating my Vec and reading my datasets?
>>> 2) Is there anything I can do to read my 1D datasets into Vecs?
>>> 3) Could PETSc perhaps be a little less restrictive in these cases?
>>> 
>>> Thanks in advance.
>>> 
>>> Regards,
>>> H?kon Strandenes
>> 
>> 
> <VecLoadTest.tar.gz>


From knepley at gmail.com  Thu Mar 19 21:05:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Mar 2015 20:05:35 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
Message-ID: <CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>

On Thu, Mar 19, 2015 at 8:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   MATT READ THIS EMAIL!
>
>   Ok here is the scoop.  In  VecView_MPI_HDF5() is the code (use meta x
> vc-annotate in emacs)
>
> df09a399 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2011-01-20
> 16:27:23 -0600  682)   if (bs >= 1) {
> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26
> 20:27:50 -0600  683)     dims[dim]      = bs;
> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26
> 20:27:50 -0600  684)     maxDims[dim]   = dims[dim];
> 61ea93a1 src/vec/vec/impls/mpi/pdvec.c (Brad Aagaard         2010-11-06
> 15:31:18 -0700  685)     chunkDims[dim] = dims[dim];
> 43c8710a src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-11-05
> 19:21:58 -0500  686)     ++dim;
> 5464dd29 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-10-31
> 19:04:52 -0500  687)   }  if (bs >= 1) {
>
>   then in PetscErrorCode VecLoad_HDF5(Vec xin, PetscViewer viewer)
>
>   f6e5521d src/vec/vec/utils/vecio.c     (Karl Rupp            2013-01-28
> 13:32:07 -0600 277) if (bs >= 1) ++dim;
>
> I check the commit message for df09a399 at bitbucket.com/petsc/petsc/ and
> it is
>
> Small change to HDF5 output (now Vec always has blocksize dim)
>
> So what is happening is even the simplest vector (with a block size of 1)
> is treated as a 2d HDF5 object
>
> Matt, what was the rational for making vectors always have a dimension of
> block size even when it is one? Seems bad to me. Maybe makes HDF5 readers
> simpler since they don't need to handle bs 1 differently?
>

My rationale was to simplify our code path, and have one way to handle
fields with components. I did not
want to special case a field with one component.

  Thanks,

     Matt


>  Barry
>
>
>
>
> > On Mar 19, 2015, at 12:31 PM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> >
> > I'm sorry I didn't include this in the first email. Here is an example
> HDF5-file, together with a sample program that demonstrate the problem.
> >
> > I have been trying to dig into VecLoad_HDF5() in vecio.c in order to
> suggest a solution myself, however, I do not think I fully understand the
> concept of the Vec block size and how that affects how the Vec is stored in
> memory and on disk (HDF5 file).
> >
> > Regards,
> > H?kon Strandenes
> >
> >
> > On 19. mars 2015 18:17, Barry Smith wrote:
> >>
> >>   Please email the file in.h5 so we can try to reproduce the problem
> and debug it. Are you using petsc version 3.5.?
> >>
> >>   Barry
> >>
> >>> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am again in trouble with PETSc and its HDF5-related functions.
> >>>
> >>> I have a code in which I want to read a 1D dataset from a HDF5 file
> into a PETSc Vec. The array are "small", and each MPI process should read
> the entire dataset and store it in its own sequential Vec.
> >>>
> >>> Currently my code is as follows (somewhat simplified):
> >>>
> >>> VecCreate(PETSC_COMM_SELF, &v);
> >>> VecSetType(v, VECSEQ);
> >>> PetscObjectSetName((PetscObject)v, "x");
> >>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ,
> &viewer);
> >>> VecLoad(v, viewer);
> >>>
> >>> This works, but only if the dataset in the HDF5 file is a 2D dataset,
> of size Nx1, where N is the number of elements I read. If the dataset in
> the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
> >>>
> >>> The error message I get is the following:
> >>> [0]PETSC ERROR: Unexpected data in file
> >>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
> >>> etc. and it points back to the line 296 in file vecio.c, function
> VecLoad_HDF5().
> >>>
> >>> Technically this is not a problem. I could of course just go ahead and
> "reshape" all my datasets into 2D datasets. I do however find this
> behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
> >>>
> >>> So my question are:
> >>> 1) Do I do anything wrong when creating my Vec and reading my datasets?
> >>> 2) Is there anything I can do to read my 1D datasets into Vecs?
> >>> 3) Could PETSc perhaps be a little less restrictive in these cases?
> >>>
> >>> Thanks in advance.
> >>>
> >>> Regards,
> >>> H?kon Strandenes
> >>
> >>
> > <VecLoadTest.tar.gz>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150319/5d634821/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Mar 19 21:22:01 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 19 Mar 2015 21:22:01 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
Message-ID: <FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>


> On Mar 19, 2015, at 9:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Mar 19, 2015 at 8:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   MATT READ THIS EMAIL!
> 
>   Ok here is the scoop.  In  VecView_MPI_HDF5() is the code (use meta x vc-annotate in emacs)
> 
> df09a399 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2011-01-20 16:27:23 -0600  682)   if (bs >= 1) {
> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  683)     dims[dim]      = bs;
> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  684)     maxDims[dim]   = dims[dim];
> 61ea93a1 src/vec/vec/impls/mpi/pdvec.c (Brad Aagaard         2010-11-06 15:31:18 -0700  685)     chunkDims[dim] = dims[dim];
> 43c8710a src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-11-05 19:21:58 -0500  686)     ++dim;
> 5464dd29 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-10-31 19:04:52 -0500  687)   }  if (bs >= 1) {
> 
>   then in PetscErrorCode VecLoad_HDF5(Vec xin, PetscViewer viewer)
> 
>   f6e5521d src/vec/vec/utils/vecio.c     (Karl Rupp            2013-01-28 13:32:07 -0600 277) if (bs >= 1) ++dim;
> 
> I check the commit message for df09a399 at bitbucket.com/petsc/petsc/ and it is
> 
> Small change to HDF5 output (now Vec always has blocksize dim)
> 
> So what is happening is even the simplest vector (with a block size of 1) is treated as a 2d HDF5 object
> 
> Matt, what was the rational for making vectors always have a dimension of block size even when it is one? Seems bad to me. Maybe makes HDF5 readers simpler since they don't need to handle bs 1 differently?
> 
> My rationale was to simplify our code path, and have one way to handle fields with components. I did not
> want to special case a field with one component.

  Why have the meaningless glop of   if (bs >= 1) { in the code? If you are always adding one to the dimension then always add it. Having this if test in the source is strange and confusing. It should be removed in several places.

  This is not documented anywhere. How is anyone to know that plain old PETSc vectors always have at least two dimensions in HDF5? That is certainly not intuitive. At a minimum the error message which is currently

> [0]PETSC ERROR: Unexpected data in file
>>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected

 could point (when the HDF5 file has a dimension of 1)  to a PETSc FAQ or manual page that explains that PETSc vectors always have a dimension of at least 2 and at that complex PETSC_SCALAR makes it a dimension of at least 3 currently users have to search rather obscure code to figure this out.

Barry

> 
>   Thanks,
> 
>      Matt
>  
>  Barry
> 
> 
> 
> 
> > On Mar 19, 2015, at 12:31 PM, H?kon Strandenes <haakon at hakostra.net> wrote:
> >
> > I'm sorry I didn't include this in the first email. Here is an example HDF5-file, together with a sample program that demonstrate the problem.
> >
> > I have been trying to dig into VecLoad_HDF5() in vecio.c in order to suggest a solution myself, however, I do not think I fully understand the concept of the Vec block size and how that affects how the Vec is stored in memory and on disk (HDF5 file).
> >
> > Regards,
> > H?kon Strandenes
> >
> >
> > On 19. mars 2015 18:17, Barry Smith wrote:
> >>
> >>   Please email the file in.h5 so we can try to reproduce the problem and debug it. Are you using petsc version 3.5.?
> >>
> >>   Barry
> >>
> >>> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am again in trouble with PETSc and its HDF5-related functions.
> >>>
> >>> I have a code in which I want to read a 1D dataset from a HDF5 file into a PETSc Vec. The array are "small", and each MPI process should read the entire dataset and store it in its own sequential Vec.
> >>>
> >>> Currently my code is as follows (somewhat simplified):
> >>>
> >>> VecCreate(PETSC_COMM_SELF, &v);
> >>> VecSetType(v, VECSEQ);
> >>> PetscObjectSetName((PetscObject)v, "x");
> >>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
> >>> VecLoad(v, viewer);
> >>>
> >>> This works, but only if the dataset in the HDF5 file is a 2D dataset, of size Nx1, where N is the number of elements I read. If the dataset in the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
> >>>
> >>> The error message I get is the following:
> >>> [0]PETSC ERROR: Unexpected data in file
> >>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
> >>> etc. and it points back to the line 296 in file vecio.c, function VecLoad_HDF5().
> >>>
> >>> Technically this is not a problem. I could of course just go ahead and "reshape" all my datasets into 2D datasets. I do however find this behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
> >>>
> >>> So my question are:
> >>> 1) Do I do anything wrong when creating my Vec and reading my datasets?
> >>> 2) Is there anything I can do to read my 1D datasets into Vecs?
> >>> 3) Could PETSc perhaps be a little less restrictive in these cases?
> >>>
> >>> Thanks in advance.
> >>>
> >>> Regards,
> >>> H?kon Strandenes
> >>
> >>
> > <VecLoadTest.tar.gz>
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From haakon at hakostra.net  Fri Mar 20 02:27:21 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Fri, 20 Mar 2015 08:27:21 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>	<550B07D8.9060903@hakostra.net>	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
Message-ID: <550BCBD9.80608@hakostra.net>

In my opinion, one should handle a 1D Vec as a 1D dataset and vice 
versa. Why? Because that is the most logical behaviour for the users.

In the code I am working on, I use Python, with h5py, to generate grids, 
which is used to create a DMDA, and the initial condition on this DMDA 
is again loaded from the same HDF5 file created by the Python script. 
This works GREAT, expect reading these 1D datasets (grid point locations).

When dealing with DMDA's this is simple. If I have a 3D DMDA with dof=1, 
then I create a numpy array with three dimensions, fill it with data and 
write it to a HDF5 file, like this:

   p = np.zeros([nz, ny, nx]) # Note only 3 dims!
   # Fill p with data here
   f = h5py.File('grid.h5', 'w')
   f.create_dataset('/FIELDS/p', data=p)

If my DMDA has dof=3, I do the following:

   U = np.zeros([nz, ny, nx, 3]) # Note 4 dims!
   # Fill U with data here
   f = h5py.File('grid.h5', 'w')
   f.create_dataset('/FIELDS/U', data=U)

So there is obviously a case of special-treatment in these cases when 
dof=1, since that leads to one lower dimension.

I think this "special treatment" of DMDA Vecs with dof=1 is a consistent 
behaviour.

I think that the cases of plain, non-DMDA Vecs as well should be handled 
the same way, i.e. a Vec with bs=1, no timestepping and no complex 
numbers should be 1D. This would be the same "logic" as used in DMDA 
Vecs, where dof=1 means one lower dimension.

I also think that one should accept reading of a 2D dataset with size 
Nx1 into a Vec, in order to give maximum flexibility for the user. The 
possibility of manipulating "heavy data" in HDF5 files in simple Python 
scripts (for pre- and postprocessing) is simply a too sweet fruit to 
leave behind, and then you need as much flexibility as possible for the 
users.

Regards,
H?kon


On 20. mars 2015 03:22, Barry Smith wrote:
>
>> On Mar 19, 2015, at 9:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Thu, Mar 19, 2015 at 8:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>    MATT READ THIS EMAIL!
>>
>>    Ok here is the scoop.  In  VecView_MPI_HDF5() is the code (use meta x vc-annotate in emacs)
>>
>> df09a399 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2011-01-20 16:27:23 -0600  682)   if (bs >= 1) {
>> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  683)     dims[dim]      = bs;
>> 272345b0 src/vec/vec/impls/mpi/pdvec.c (Karl Rupp            2013-01-26 20:27:50 -0600  684)     maxDims[dim]   = dims[dim];
>> 61ea93a1 src/vec/vec/impls/mpi/pdvec.c (Brad Aagaard         2010-11-06 15:31:18 -0700  685)     chunkDims[dim] = dims[dim];
>> 43c8710a src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-11-05 19:21:58 -0500  686)     ++dim;
>> 5464dd29 src/vec/vec/impls/mpi/pdvec.c (Matthew Knepley      2010-10-31 19:04:52 -0500  687)   }  if (bs >= 1) {
>>
>>    then in PetscErrorCode VecLoad_HDF5(Vec xin, PetscViewer viewer)
>>
>>    f6e5521d src/vec/vec/utils/vecio.c     (Karl Rupp            2013-01-28 13:32:07 -0600 277) if (bs >= 1) ++dim;
>>
>> I check the commit message for df09a399 at bitbucket.com/petsc/petsc/ and it is
>>
>> Small change to HDF5 output (now Vec always has blocksize dim)
>>
>> So what is happening is even the simplest vector (with a block size of 1) is treated as a 2d HDF5 object
>>
>> Matt, what was the rational for making vectors always have a dimension of block size even when it is one? Seems bad to me. Maybe makes HDF5 readers simpler since they don't need to handle bs 1 differently?
>>
>> My rationale was to simplify our code path, and have one way to handle fields with components. I did not
>> want to special case a field with one component.
>
>    Why have the meaningless glop of   if (bs >= 1) { in the code? If you are always adding one to the dimension then always add it. Having this if test in the source is strange and confusing. It should be removed in several places.
>
>    This is not documented anywhere. How is anyone to know that plain old PETSc vectors always have at least two dimensions in HDF5? That is certainly not intuitive. At a minimum the error message which is currently
>
>> [0]PETSC ERROR: Unexpected data in file
>>>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
>
>   could point (when the HDF5 file has a dimension of 1)  to a PETSc FAQ or manual page that explains that PETSc vectors always have a dimension of at least 2 and at that complex PETSC_SCALAR makes it a dimension of at least 3 currently users have to search rather obscure code to figure this out.
>
> Barry
>
>>
>>    Thanks,
>>
>>       Matt
>>
>>   Barry
>>
>>
>>
>>
>>> On Mar 19, 2015, at 12:31 PM, H?kon Strandenes <haakon at hakostra.net> wrote:
>>>
>>> I'm sorry I didn't include this in the first email. Here is an example HDF5-file, together with a sample program that demonstrate the problem.
>>>
>>> I have been trying to dig into VecLoad_HDF5() in vecio.c in order to suggest a solution myself, however, I do not think I fully understand the concept of the Vec block size and how that affects how the Vec is stored in memory and on disk (HDF5 file).
>>>
>>> Regards,
>>> H?kon Strandenes
>>>
>>>
>>> On 19. mars 2015 18:17, Barry Smith wrote:
>>>>
>>>>    Please email the file in.h5 so we can try to reproduce the problem and debug it. Are you using petsc version 3.5.?
>>>>
>>>>    Barry
>>>>
>>>>> On Mar 19, 2015, at 4:19 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am again in trouble with PETSc and its HDF5-related functions.
>>>>>
>>>>> I have a code in which I want to read a 1D dataset from a HDF5 file into a PETSc Vec. The array are "small", and each MPI process should read the entire dataset and store it in its own sequential Vec.
>>>>>
>>>>> Currently my code is as follows (somewhat simplified):
>>>>>
>>>>> VecCreate(PETSC_COMM_SELF, &v);
>>>>> VecSetType(v, VECSEQ);
>>>>> PetscObjectSetName((PetscObject)v, "x");
>>>>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "in.h5", FILE_MODE_READ, &viewer);
>>>>> VecLoad(v, viewer);
>>>>>
>>>>> This works, but only if the dataset in the HDF5 file is a 2D dataset, of size Nx1, where N is the number of elements I read. If the dataset in the HDF5 file is of size N (i.e. a 1D dataset) this does not work.
>>>>>
>>>>> The error message I get is the following:
>>>>> [0]PETSC ERROR: Unexpected data in file
>>>>> [0]PETSC ERROR: Dimension of array in file 1 not 2 as expected
>>>>> etc. and it points back to the line 296 in file vecio.c, function VecLoad_HDF5().
>>>>>
>>>>> Technically this is not a problem. I could of course just go ahead and "reshape" all my datasets into 2D datasets. I do however find this behaviour very un-logical. Why can't I read a 1D dataset into a Vec?
>>>>>
>>>>> So my question are:
>>>>> 1) Do I do anything wrong when creating my Vec and reading my datasets?
>>>>> 2) Is there anything I can do to read my 1D datasets into Vecs?
>>>>> 3) Could PETSc perhaps be a little less restrictive in these cases?
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> Regards,
>>>>> H?kon Strandenes
>>>>
>>>>
>>> <VecLoadTest.tar.gz>
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>
>

From jed at jedbrown.org  Fri Mar 20 09:29:11 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 20 Mar 2015 08:29:11 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <550BCBD9.80608@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net>
Message-ID: <87zj77ivyg.fsf@jedbrown.org>

H?kon Strandenes <haakon at hakostra.net> writes:
> I think this "special treatment" of DMDA Vecs with dof=1 is a consistent 
> behaviour.

Why is it more consistent?  There are many places in the Python standard
libraries where tuples of size 1 are returned for the analogue of dof=1,
instead of implicitly unwrapping that case and leaving it to the caller
to handle each case separately.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/7dd42c47/attachment.pgp>

From haakon at hakostra.net  Fri Mar 20 10:01:22 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Fri, 20 Mar 2015 16:01:22 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <87zj77ivyg.fsf@jedbrown.org>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
Message-ID: <550C3642.5000208@hakostra.net>

That formulation is obviously not one of my best. I did not mean that it 
was consistent with respect to what is done or not done in Python.

My point was that when using DMDA's in PETSc, you consistently 
"disregard" the dof when dof=1, i.e. create 3D arrays instead of 4D (for 
a 3D DMDA). A dump from an HDF5 file (written by PETSc) where I have 
saved two fields (on 20x20x1 grid cells), one with dof=1 and one with 
dof=3 looks like:

U                        Dataset {1, 20, 20, 3}
p                        Dataset {1, 20, 20}

I think that in case you choose to stick to the policy of adding another 
dimension for bs=1, you should also add that dimension for a DMDA Vec 
with dof=1.

H?kon


On 20. mars 2015 15:29, Jed Brown wrote:
> H?kon Strandenes <haakon at hakostra.net> writes:
>> I think this "special treatment" of DMDA Vecs with dof=1 is a consistent
>> behaviour.
>
> Why is it more consistent?  There are many places in the Python standard
> libraries where tuples of size 1 are returned for the analogue of dof=1,
> instead of implicitly unwrapping that case and leaving it to the caller
> to handle each case separately.
>

From jed at jedbrown.org  Fri Mar 20 10:15:58 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 20 Mar 2015 09:15:58 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <550C3642.5000208@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net>
Message-ID: <87h9tfaedt.fsf@jedbrown.org>

H?kon Strandenes <haakon at hakostra.net> writes:

> That formulation is obviously not one of my best. I did not mean that it 
> was consistent with respect to what is done or not done in Python.
>
> My point was that when using DMDA's in PETSc, you consistently 
> "disregard" the dof when dof=1, i.e. create 3D arrays instead of 4D (for 
> a 3D DMDA). 

If I'm writing extensible code, I even solve scalar problems by writing

typedef struct {
  PetscScalar phi; // only one variable now
} Field;

If I add another field later, I don't have to revisit code that only
references phi.  Having this dimension also leaves a place for metadata.

> I think that in case you choose to stick to the policy of adding another 
> dimension for bs=1, you should also add that dimension for a DMDA Vec 
> with dof=1.

What do you mean?  You can use DMDAVecGetArrayDOF when dof=1 if you want.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/5b60b494/attachment.pgp>

From haakon at hakostra.net  Fri Mar 20 10:45:43 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Fri, 20 Mar 2015 16:45:43 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <87h9tfaedt.fsf@jedbrown.org>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
Message-ID: <550C40A7.5090303@hakostra.net>


On 20. mars 2015 16:15, Jed Brown wrote:
> H?kon Strandenes <haakon at hakostra.net> writes:
>
>> That formulation is obviously not one of my best. I did not mean that it
>> was consistent with respect to what is done or not done in Python.
>>
>> My point was that when using DMDA's in PETSc, you consistently
>> "disregard" the dof when dof=1, i.e. create 3D arrays instead of 4D (for
>> a 3D DMDA).
>
> If I'm writing extensible code, I even solve scalar problems by writing
>
> typedef struct {
>    PetscScalar phi; // only one variable now
> } Field;
>
> If I add another field later, I don't have to revisit code that only
> references phi.  Having this dimension also leaves a place for metadata.
>
>> I think that in case you choose to stick to the policy of adding another
>> dimension for bs=1, you should also add that dimension for a DMDA Vec
>> with dof=1.
>
> What do you mean?  You can use DMDAVecGetArrayDOF when dof=1 if you want.
>

I mean that if you decide to stick with the current way of 
reading/writing a "plain 1D" Vec with N elements as a N x 1 
array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1 
array/dataset.

H?kon

From jed at jedbrown.org  Fri Mar 20 10:57:02 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 20 Mar 2015 09:57:02 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <550C40A7.5090303@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net>
Message-ID: <878uerachd.fsf@jedbrown.org>

H?kon Strandenes <haakon at hakostra.net> writes:
> I mean that if you decide to stick with the current way of 
> reading/writing a "plain 1D" Vec with N elements as a N x 1 
> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1 
> array/dataset.

In the HDF5 file?  I agree.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/cd13aa56/attachment.pgp>

From haakon at hakostra.net  Fri Mar 20 11:31:03 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Fri, 20 Mar 2015 17:31:03 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <878uerachd.fsf@jedbrown.org>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
Message-ID: <550C4B47.60903@hakostra.net>

Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x NY 
x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5 
file.

In my humble opinion, it is best to leave the DMDA's as it is, and fix 
up the code related to the plain Vecs. That code is already full of "if 
(bs >= 1) dim++" etc. which is useless (as Barry points out).

My final point is that one should handle this in a CONSISTENT manner for 
both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to always 
add one dimension for bs and dof or leave that dimension out for the 
special case with bs=1/dof=1.

H?kon


On 20. mars 2015 16:57, Jed Brown wrote:
> H?kon Strandenes <haakon at hakostra.net> writes:
>> I mean that if you decide to stick with the current way of
>> reading/writing a "plain 1D" Vec with N elements as a N x 1
>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1
>> array/dataset.
>
> In the HDF5 file?  I agree.
>

From may at bu.edu  Fri Mar 20 13:02:40 2015
From: may at bu.edu (Young, Matthew, Adam)
Date: Fri, 20 Mar 2015 18:02:40 +0000
Subject: [petsc-users] PETSc within a larger simulation
Message-ID: <17A35C213185A84BB8ED54C88FBFD712B4CA356A@IST-EX10MBX-4.ad.bu.edu>

I have a PETSc routine that I am trying to use as an electric-field solver within a particle-in-cell code. The existing PIC code is parallelized with MPI. The pure PIC code (written many years ago by my advisor) is well-tested and I am trying to add an option for it to operate as a hybrid fluid/PIC code. This requires solving Ax=b at every time step but I don't think I understand how to manage communication between the existing code and my PETSc solver. Since we may not always run the hybrid version, the code calls MPI_Init() before PetscInitialize(). Section 14.1 of the manual suggests that I use PetscSetCommWorld between MPI_Init() and PetscInitialize(), but that routine appears to not be supported anymore. Is that correct? Can you point me to examples of simulations that use PETSc within a larger code?

--Matt

--------------------------------------------------------------
Matthew Young
Graduate Student
Boston University Dept. of Astronomy
--------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/8be5b3a3/attachment.html>

From bsmith at mcs.anl.gov  Fri Mar 20 13:08:26 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 13:08:26 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ
	THIS	EMAIL!
In-Reply-To: <550C4B47.60903@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
Message-ID: <01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>


> On Mar 20, 2015, at 11:31 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> 
> Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x NY x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5 file.

  Actually a 1d DMDA is stored as a NX dataset, a 2d DMDA is stored as a NY x NX data set and a 3d DMDA is stored as a NZ x NY x NX data set and an extra dimension is added if dof > 1.   And I think this is great and intuitively obvious for users.

   I agree with Hakon and advocate NOT sticking the extra dimension in when bs is 1. Here is Matt's entire commit


   maxDims[dim] = dims[dim];
   chunkDims[dim] = dims[dim];
   ++dim;
-  if (bs > 1) {
+  if (bs >= 1) {
     dims[dim]    = bs;
     maxDims[dim] = dims[dim];
     chunkDims[dim] = dims[dim];
   }
   count[dim] = PetscHDF5IntCast(xin->map->n)/bs;
   ++dim;
-  if (bs > 1) {
+  if (bs >= 1) {
     count[dim] = bs;
     ++dim;
   }
   }
   offset[dim] = PetscHDF5IntCast(low/bs);
   ++dim;
-  if (bs > 1) {
+  if (bs >= 1) {
     offset[dim] = 0;
     ++dim;
   }

  (Then there is Karl's one line fix for reading HDF5 files.)

  Arguing that handling bs = 1 as a special case greatly (or even modestly) increases the complexity of the PETSc source code is nonsense.

  So I want a stronger case presented then

>> My rationale was to simplify our code path, and have one way to handle fields with components. I did not
>> want to special case a field with one component.

  otherwise I think we should undo Matt's change.

  Barry


  
> 
> In my humble opinion, it is best to leave the DMDA's as it is, and fix up the code related to the plain Vecs. That code is already full of "if (bs >= 1) dim++" etc. which is useless (as Barry points out).
> 
> My final point is that one should handle this in a CONSISTENT manner for both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to always add one dimension for bs and dof or leave that dimension out for the special case with bs=1/dof=1.
> 
> H?kon
> 
> 
> On 20. mars 2015 16:57, Jed Brown wrote:
>> H?kon Strandenes <haakon at hakostra.net> writes:
>>> I mean that if you decide to stick with the current way of
>>> reading/writing a "plain 1D" Vec with N elements as a N x 1
>>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1
>>> array/dataset.
>> 
>> In the HDF5 file?  I agree.
>> 


From bsmith at mcs.anl.gov  Fri Mar 20 13:11:37 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 13:11:37 -0500
Subject: [petsc-users] PETSc within a larger simulation
In-Reply-To: <17A35C213185A84BB8ED54C88FBFD712B4CA356A@IST-EX10MBX-4.ad.bu.edu>
References: <17A35C213185A84BB8ED54C88FBFD712B4CA356A@IST-EX10MBX-4.ad.bu.edu>
Message-ID: <B677DA97-8451-437A-A8A3-F2A20A5BF433@mcs.anl.gov>


> On Mar 20, 2015, at 1:02 PM, Young, Matthew, Adam <may at bu.edu> wrote:
> 
> I have a PETSc routine that I am trying to use as an electric-field solver within a particle-in-cell code. The existing PIC code is parallelized with MPI. The pure PIC code (written many years ago by my advisor) is well-tested and I am trying to add an option for it to operate as a hybrid fluid/PIC code. This requires solving Ax=b at every time step but I don't think I understand how to manage communication between the existing code and my PETSc solver. Since we may not always run the hybrid version, the code calls MPI_Init() before PetscInitialize(). Section 14.1 of the manual suggests that I use PetscSetCommWorld between MPI_Init() and PetscInitialize(), but that routine appears to not be supported anymore. Is that correct? Can you point me to examples of simulations that use PETSc within a larger code?

   Matt,

    In the standard case where the PIC and fluid code are both running on the same communicator (MPI_COMM_WORLD) you don't need to do anything special, just call the MPI_Init() and then the PetscInitialize(). If the PETSc code runs on a smaller communicator then BEFORE the call to PetscInitialize() you just need to set PETSC_COMM_WORLD = comm (where comm is that smaller communicator). (This replaces the use of the function call PetscSetCommWorld() that once existed.)

  Barry

> 
> --Matt
> 
> --------------------------------------------------------------
> Matthew Young
> Graduate Student
> Boston University Dept. of Astronomy
> --------------------------------------------------------------


From knepley at gmail.com  Fri Mar 20 13:17:13 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 20 Mar 2015 12:17:13 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
Message-ID: <CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>

On Fri, Mar 20, 2015 at 12:08 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 20, 2015, at 11:31 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> >
> > Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x NY
> x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5
> file.
>
>   Actually a 1d DMDA is stored as a NX dataset, a 2d DMDA is stored as a
> NY x NX data set and a 3d DMDA is stored as a NZ x NY x NX data set and an
> extra dimension is added if dof > 1.   And I think this is great and
> intuitively obvious for users.
>
>    I agree with Hakon and advocate NOT sticking the extra dimension in
> when bs is 1. Here is Matt's entire commit
>
>
>    maxDims[dim] = dims[dim];
>    chunkDims[dim] = dims[dim];
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      dims[dim]    = bs;
>      maxDims[dim] = dims[dim];
>      chunkDims[dim] = dims[dim];
>    }
>    count[dim] = PetscHDF5IntCast(xin->map->n)/bs;
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      count[dim] = bs;
>      ++dim;
>    }
>    }
>    offset[dim] = PetscHDF5IntCast(low/bs);
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      offset[dim] = 0;
>      ++dim;
>    }
>
>   (Then there is Karl's one line fix for reading HDF5 files.)
>
>   Arguing that handling bs = 1 as a special case greatly (or even
> modestly) increases the complexity of the PETSc source code is nonsense.
>
>   So I want a stronger case presented then
>
> >> My rationale was to simplify our code path, and have one way to handle
> fields with components. I did not
> >> want to special case a field with one component.
>

If we special case here, we must special case in everything we do, such as
automatic generation of Xdmf.
I would rather say that the PETSc format always has the bs in it, which
makes it easier to process and
consistent. However, we allowing reading of of a set without blocksize.

We would have the ability to write without blocksize by setting the
blocksize to -1.

   Matt


>   otherwise I think we should undo Matt's change.
>
>   Barry
>
>
>
> >
> > In my humble opinion, it is best to leave the DMDA's as it is, and fix
> up the code related to the plain Vecs. That code is already full of "if (bs
> >= 1) dim++" etc. which is useless (as Barry points out).
> >
> > My final point is that one should handle this in a CONSISTENT manner for
> both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to always
> add one dimension for bs and dof or leave that dimension out for the
> special case with bs=1/dof=1.
> >
> > H?kon
> >
> >
> > On 20. mars 2015 16:57, Jed Brown wrote:
> >> H?kon Strandenes <haakon at hakostra.net> writes:
> >>> I mean that if you decide to stick with the current way of
> >>> reading/writing a "plain 1D" Vec with N elements as a N x 1
> >>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1
> >>> array/dataset.
> >>
> >> In the HDF5 file?  I agree.
> >>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/7dc250d5/attachment.html>

From bsmith at mcs.anl.gov  Fri Mar 20 13:32:49 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 13:32:49 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
Message-ID: <1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>


> On Mar 20, 2015, at 1:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Mar 20, 2015 at 12:08 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 20, 2015, at 11:31 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> >
> > Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x NY x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5 file.
> 
>   Actually a 1d DMDA is stored as a NX dataset, a 2d DMDA is stored as a NY x NX data set and a 3d DMDA is stored as a NZ x NY x NX data set and an extra dimension is added if dof > 1.   And I think this is great and intuitively obvious for users.
> 
>    I agree with Hakon and advocate NOT sticking the extra dimension in when bs is 1. Here is Matt's entire commit
> 
> 
>    maxDims[dim] = dims[dim];
>    chunkDims[dim] = dims[dim];
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      dims[dim]    = bs;
>      maxDims[dim] = dims[dim];
>      chunkDims[dim] = dims[dim];
>    }
>    count[dim] = PetscHDF5IntCast(xin->map->n)/bs;
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      count[dim] = bs;
>      ++dim;
>    }
>    }
>    offset[dim] = PetscHDF5IntCast(low/bs);
>    ++dim;
> -  if (bs > 1) {
> +  if (bs >= 1) {
>      offset[dim] = 0;
>      ++dim;
>    }
> 
>   (Then there is Karl's one line fix for reading HDF5 files.)
> 
>   Arguing that handling bs = 1 as a special case greatly (or even modestly) increases the complexity of the PETSc source code is nonsense.
> 
>   So I want a stronger case presented then
> 
> >> My rationale was to simplify our code path, and have one way to handle fields with components. I did not
> >> want to special case a field with one component.
> 
> If we special case here, we must special case in everything we do, such as automatic generation of Xdmf.
> I would rather say that the PETSc format always has the bs in it, which makes it easier to process and
> consistent.

   From what I see the "easier" is just trivially easier; having a tiny bit more complicated code that is intuitively clear for users seems a better choice.

> However, we allowing reading of of a set without block size.

  Hmm, according to the bug report which I confirmed, VecLoad() refuses to load in a 1d dataset
> 
> We would have the ability to write without blocksize by setting the blocksize to -1.

  Ok, so now you are still supporting that "special case" but with some totally non-intuitive value for a block size that will crash if used with the vector in any other usage of the vector. How is this either simpler code or more intuitive code.

 As Hakon notes also, VecView from a DMDA is handled differently with regard to the bs dimension then VecView from a simple "vector". Which is terrible.

Barry

> 
>    Matt
>  
>   otherwise I think we should undo Matt's change.
> 
>   Barry
> 
> 
> 
> >
> > In my humble opinion, it is best to leave the DMDA's as it is, and fix up the code related to the plain Vecs. That code is already full of "if (bs >= 1) dim++" etc. which is useless (as Barry points out).
> >
> > My final point is that one should handle this in a CONSISTENT manner for both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to always add one dimension for bs and dof or leave that dimension out for the special case with bs=1/dof=1.
> >
> > H?kon
> >
> >
> > On 20. mars 2015 16:57, Jed Brown wrote:
> >> H?kon Strandenes <haakon at hakostra.net> writes:
> >>> I mean that if you decide to stick with the current way of
> >>> reading/writing a "plain 1D" Vec with N elements as a N x 1
> >>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1
> >>> array/dataset.
> >>
> >> In the HDF5 file?  I agree.
> >>
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From knepley at gmail.com  Fri Mar 20 13:37:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 20 Mar 2015 12:37:04 -0600
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
Message-ID: <CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>

On Fri, Mar 20, 2015 at 12:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 20, 2015, at 1:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Fri, Mar 20, 2015 at 12:08 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Mar 20, 2015, at 11:31 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> > >
> > > Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x
> NY x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5
> file.
> >
> >   Actually a 1d DMDA is stored as a NX dataset, a 2d DMDA is stored as a
> NY x NX data set and a 3d DMDA is stored as a NZ x NY x NX data set and an
> extra dimension is added if dof > 1.   And I think this is great and
> intuitively obvious for users.
> >
> >    I agree with Hakon and advocate NOT sticking the extra dimension in
> when bs is 1. Here is Matt's entire commit
> >
> >
> >    maxDims[dim] = dims[dim];
> >    chunkDims[dim] = dims[dim];
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      dims[dim]    = bs;
> >      maxDims[dim] = dims[dim];
> >      chunkDims[dim] = dims[dim];
> >    }
> >    count[dim] = PetscHDF5IntCast(xin->map->n)/bs;
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      count[dim] = bs;
> >      ++dim;
> >    }
> >    }
> >    offset[dim] = PetscHDF5IntCast(low/bs);
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      offset[dim] = 0;
> >      ++dim;
> >    }
> >
> >   (Then there is Karl's one line fix for reading HDF5 files.)
> >
> >   Arguing that handling bs = 1 as a special case greatly (or even
> modestly) increases the complexity of the PETSc source code is nonsense.
> >
> >   So I want a stronger case presented then
> >
> > >> My rationale was to simplify our code path, and have one way to
> handle fields with components. I did not
> > >> want to special case a field with one component.
> >
> > If we special case here, we must special case in everything we do, such
> as automatic generation of Xdmf.
> > I would rather say that the PETSc format always has the bs in it, which
> makes it easier to process and
> > consistent.
>
>    From what I see the "easier" is just trivially easier; having a tiny
> bit more complicated code that is intuitively clear for users seems a
> better choice.
>
> > However, we allowing reading of of a set without block size.
>
>   Hmm, according to the bug report which I confirmed, VecLoad() refuses to
> load in a 1d dataset
>

Yes, I was saying this is the part we change.


> >
> > We would have the ability to write without blocksize by setting the
> blocksize to -1.
>
>   Ok, so now you are still supporting that "special case" but with some
> totally non-intuitive value for a block size that will crash if used with
> the vector in any other usage of the vector. How is this either simpler
> code or more intuitive code.
>

I am supporting a special case in this function for output, but not for our
accepted format. I want
to avoid a special case in the format, not in the function itself. Then I
can use the format from
other tools without making them complicated too.


>  As Hakon notes also, VecView from a DMDA is handled differently with
> regard to the bs dimension then VecView from a simple "vector". Which is
> terrible.


Yes, I think we should change it so that all vectors come with the
blocksize.

  Matt


>
> Barry
>
> >
> >    Matt
> >
> >   otherwise I think we should undo Matt's change.
> >
> >   Barry
> >
> >
> >
> > >
> > > In my humble opinion, it is best to leave the DMDA's as it is, and fix
> up the code related to the plain Vecs. That code is already full of "if (bs
> >= 1) dim++" etc. which is useless (as Barry points out).
> > >
> > > My final point is that one should handle this in a CONSISTENT manner
> for both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to
> always add one dimension for bs and dof or leave that dimension out for the
> special case with bs=1/dof=1.
> > >
> > > H?kon
> > >
> > >
> > > On 20. mars 2015 16:57, Jed Brown wrote:
> > >> H?kon Strandenes <haakon at hakostra.net> writes:
> > >>> I mean that if you decide to stick with the current way of
> > >>> reading/writing a "plain 1D" Vec with N elements as a N x 1
> > >>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX
> x 1
> > >>> array/dataset.
> > >>
> > >> In the HDF5 file?  I agree.
> > >>
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/ae46aacd/attachment-0001.html>

From ansp6066 at colorado.edu  Fri Mar 20 13:52:04 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Fri, 20 Mar 2015 11:52:04 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems.
Message-ID: <1426877523913.4294d19f@Nodemailer>

I have a fairly simple problem that I?m trying to timestep:


u? = A(t) u


I?m using the crank-nicholson method, which I understand (for this problem) to be:


u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
or
[1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)


When I attempt to timestep using PETSc, the norm of `u` blows up. ?When I do it directly (using the above), the norm of `u` doesn?t blow up.


It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. ?The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.


I?m not sure what I?m doing wrong.


PETSc code is taken out of the manual and is pretty simple:


? ? ? ? TSCreate( comm, &ts );
? ? ? ? TSSetProblemType( ts, TS_LINEAR);
? ? ? ? TSSetType( ts, TSCN );
? ? ? ? TSSetInitialTimeStep( ts, 0, 0.01 );
? ? ? ? TSSetDuration( ts, 5, 0.03 );
? ? ? ? TSSetFromOptions( ts );
? ? ? ? TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
? ? ? ? TSSetRHSJacobian( ts, A, A, func, &cntx );
? ? ? ? TSSolve( ts, psi0 );


`func` just constructs A(t) at the time given. ?The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.


Let me know what other information is needed. ?I?m not sure what could be the problem. ?`func` doesn?t touch U at all (should it?).


-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/c1ab10c3/attachment.html>

From bsmith at mcs.anl.gov  Fri Mar 20 14:48:39 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 14:48:39 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
Message-ID: <6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>


> On Mar 20, 2015, at 1:37 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Mar 20, 2015 at 12:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 20, 2015, at 1:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Fri, Mar 20, 2015 at 12:08 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Mar 20, 2015, at 11:31 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> > >
> > > Yes, in the HDF5 file. Currently a dof=1 DMDA Vec is saved as a NZ x NY x NX dataset, wile a dof > 1 is saved as NZ x NY x NX x dof in the HDF5 file.
> >
> >   Actually a 1d DMDA is stored as a NX dataset, a 2d DMDA is stored as a NY x NX data set and a 3d DMDA is stored as a NZ x NY x NX data set and an extra dimension is added if dof > 1.   And I think this is great and intuitively obvious for users.
> >
> >    I agree with Hakon and advocate NOT sticking the extra dimension in when bs is 1. Here is Matt's entire commit
> >
> >
> >    maxDims[dim] = dims[dim];
> >    chunkDims[dim] = dims[dim];
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      dims[dim]    = bs;
> >      maxDims[dim] = dims[dim];
> >      chunkDims[dim] = dims[dim];
> >    }
> >    count[dim] = PetscHDF5IntCast(xin->map->n)/bs;
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      count[dim] = bs;
> >      ++dim;
> >    }
> >    }
> >    offset[dim] = PetscHDF5IntCast(low/bs);
> >    ++dim;
> > -  if (bs > 1) {
> > +  if (bs >= 1) {
> >      offset[dim] = 0;
> >      ++dim;
> >    }
> >
> >   (Then there is Karl's one line fix for reading HDF5 files.)
> >
> >   Arguing that handling bs = 1 as a special case greatly (or even modestly) increases the complexity of the PETSc source code is nonsense.
> >
> >   So I want a stronger case presented then
> >
> > >> My rationale was to simplify our code path, and have one way to handle fields with components. I did not
> > >> want to special case a field with one component.
> >
> > If we special case here, we must special case in everything we do, such as automatic generation of Xdmf.
> > I would rather say that the PETSc format always has the bs in it, which makes it easier to process and
> > consistent.
> 
>    From what I see the "easier" is just trivially easier; having a tiny bit more complicated code that is intuitively clear for users seems a better choice.
> 
> > However, we allowing reading of of a set without block size.
> 
>   Hmm, according to the bug report which I confirmed, VecLoad() refuses to load in a 1d dataset
> 
> Yes, I was saying this is the part we change.
>  
> >
> > We would have the ability to write without blocksize by setting the blocksize to -1.
> 
>   Ok, so now you are still supporting that "special case" but with some totally non-intuitive value for a block size that will crash if used with the vector in any other usage of the vector. How is this either simpler code or more intuitive code.
> 
> I am supporting a special case in this function for output, but not for our accepted format. I want
> to avoid a special case in the format, not in the function itself.


   Why is 1 dimension a special case that is not worthy of its own format? The same thing would hold for 2d and 3d. One could then argue that we should have a single six dimensional format for the files for all vectors that PETSc produces. Then a 1d problem has five of the dimensions being 1. 

   

> Then I can use the format from
> other tools without making them complicated too.
>  
>  As Hakon notes also, VecView from a DMDA is handled differently with regard to the bs dimension then VecView from a simple "vector". Which is terrible.
> 
> Yes, I think we should change it so that all vectors come with the blocksize.
> 
>   Matt
>  
> 
> Barry
> 
> >
> >    Matt
> >
> >   otherwise I think we should undo Matt's change.
> >
> >   Barry
> >
> >
> >
> > >
> > > In my humble opinion, it is best to leave the DMDA's as it is, and fix up the code related to the plain Vecs. That code is already full of "if (bs >= 1) dim++" etc. which is useless (as Barry points out).
> > >
> > > My final point is that one should handle this in a CONSISTENT manner for both "plain Vecs" and "DMDA Vecs", i.e. decide wither one want to always add one dimension for bs and dof or leave that dimension out for the special case with bs=1/dof=1.
> > >
> > > H?kon
> > >
> > >
> > > On 20. mars 2015 16:57, Jed Brown wrote:
> > >> H?kon Strandenes <haakon at hakostra.net> writes:
> > >>> I mean that if you decide to stick with the current way of
> > >>> reading/writing a "plain 1D" Vec with N elements as a N x 1
> > >>> array/dataset, you should treat a dof=1 DMDA Vec as an NZ x NY x NX x 1
> > >>> array/dataset.
> > >>
> > >> In the HDF5 file?  I agree.
> > >>
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From knepley at gmail.com  Fri Mar 20 15:17:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 20 Mar 2015 14:17:47 -0600
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <1426877523913.4294d19f@Nodemailer>
References: <1426877523913.4294d19f@Nodemailer>
Message-ID: <CAMYG4GnPFjc5pv-sPsm-x74zA8dyiqfBhhGmJZ0Y+f0Q3f28kg@mail.gmail.com>

This sounds like a problem in your calculation function where a Vec or Mat
does not get reset to 0, but it does in your by hand code.

   Matt
On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:

>  I have a fairly simple problem that I?m trying to timestep:
>
> u? = A(t) u
>
> I?m using the crank-nicholson method, which I understand (for this
> problem) to be:
>
> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
> or
> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>
> When I attempt to timestep using PETSc, the norm of `u` blows up.  When I
> do it directly (using the above), the norm of `u` doesn?t blow up.
>
> It is important to note that the solution generated after the first step
> is identical for both, but the second step for Petsc has a norm of ~2,
> while for the directly calculated version it is ~1.  The third step for
> petsc has a norm of ~4, while the directly calculated version it is still
> ~1.
>
> I?m not sure what I?m doing wrong.
>
> PETSc code is taken out of the manual and is pretty simple:
>
>          TSCreate( comm, &ts );
>         TSSetProblemType( ts, TS_LINEAR);
>         TSSetType( ts, TSCN );
>         TSSetInitialTimeStep( ts, 0, 0.01 );
>         TSSetDuration( ts, 5, 0.03 );
>         TSSetFromOptions( ts );
>         TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>         TSSetRHSJacobian( ts, A, A, func, &cntx );
>         TSSolve( ts, psi0 );
>
> `func` just constructs A(t) at the time given.  The same code for
> calculating A(t) is used in both calculations, along with the same initial
> vector psi0, and the same time steps.
>
> Let me know what other information is needed.  I?m not sure what could be
> the problem.  `func` doesn?t touch U at all (should it?).
>
> -Andrew
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/9e881670/attachment.html>

From haakon at hakostra.net  Fri Mar 20 16:02:04 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Fri, 20 Mar 2015 22:02:04 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
Message-ID: <550C8ACC.3030102@hakostra.net>

On 20. mars 2015 20:48, Barry Smith wrote:
> Why is 1 dimension a special case that is not worthy of its own format? The same thing would hold for 2d and 3d. One could then argue that we should have a single six dimensional format for the files for all vectors that PETSc produces. Then a 1d problem has five of the dimensions being 1.

This is a very good point, and support my view.

Let me come with two very simple example cases:


Case 1:
Create a list of grid points in an external preprocessor for the purpose 
of loading this into a Vec later:

   x = np.linspace(0.0, 1.0, num=21)
   f.create_dataset('/MESH/nodes/x', data=x)

vs.

   x = np.linspace(0.0, 1.0, num=21)
   x = x.reshape((21,1))
   f.create_dataset('/MESH/nodes/x', data=x)


Case 2:
Read three Vecs written to disk by PETSc, and calculate total "bounding 
box volume" of the grid:

   g = h5py.File('grid.h5', 'r')
   x = g['/MESH/nodes/x']
   y = g['/MESH/nodes/y']
   z = g['/MESH/nodes/z']
   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])

vs.

   g = h5py.File('grid.h5', 'r')
   x = g['/MESH/nodes/x'][:,0]
   y = g['/MESH/nodes/y'][:,0]
   z = g['/MESH/nodes/z'][:,0]
   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])


In both cases I think handling this extra, unnecessary dimension makes 
the code less attractive. It's not that either way is difficult, 
problematic or impossible, but it's just that 1D Vecs should intuitively 
be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for 
quite a while until I figured this out, even after having written an 
entire Navier-Stokes DNS solver using the PETSc library for everything 
except time integration and filling these simple 1D coordinate arrays!

Regards,
H?kon

From ansp6066 at colorado.edu  Fri Mar 20 16:09:44 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Fri, 20 Mar 2015 14:09:44 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <CAMYG4GnPFjc5pv-sPsm-x74zA8dyiqfBhhGmJZ0Y+f0Q3f28kg@mail.gmail.com>
References: <CAMYG4GnPFjc5pv-sPsm-x74zA8dyiqfBhhGmJZ0Y+f0Q3f28kg@mail.gmail.com>
Message-ID: <1426885784701.1d306762@Nodemailer>

So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. ?When I do that, it just zeros out the solution.




The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? ?In my case this is A(t). ?The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). ?In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).




Is this the case? or is there some other responsibility of said function?




-Andrew




>Ah ha!

>

>The function passed to TSSetRHSJacobian needs to zero the solution vector?

>

>As a point, this isn?t mentioned in any documentation that I can find.

>

>-Andrew


On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:

This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.

?? Matt

On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:



I have a fairly simple problem that I?m trying to timestep:




u? = A(t) u




I?m using the crank-nicholson method, which I understand (for this problem) to be:




u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]

or

[1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)




When I attempt to timestep using PETSc, the norm of `u` blows up.? When I do it directly (using the above), the norm of `u` doesn?t blow up.




It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1.? The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.




I?m not sure what I?m doing wrong.




PETSc code is taken out of the manual and is pretty simple:





? ? ? ? TSCreate( comm, &ts );

? ? ? ? TSSetProblemType( ts, TS_LINEAR);

? ? ? ? TSSetType( ts, TSCN );

? ? ? ? TSSetInitialTimeStep( ts, 0, 0.01 );

? ? ? ? TSSetDuration( ts, 5, 0.03 );

? ? ? ? TSSetFromOptions( ts );

? ? ? ? TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );

? ? ? ? TSSetRHSJacobian( ts, A, A, func, &cntx );

? ? ? ? TSSolve( ts, psi0 );




`func` just constructs A(t) at the time given.? The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.




Let me know what other information is needed.? I?m not sure what could be the problem. ?`func` doesn?t touch U at all (should it?).




-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/60670db1/attachment-0001.html>

From knepley at gmail.com  Fri Mar 20 16:26:06 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 20 Mar 2015 15:26:06 -0600
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <1426885784701.1d306762@Nodemailer>
References: <CAMYG4GnPFjc5pv-sPsm-x74zA8dyiqfBhhGmJZ0Y+f0Q3f28kg@mail.gmail.com>
	<1426885784701.1d306762@Nodemailer>
Message-ID: <CAMYG4GkuaZBwT=Ja9F0uNyh2jA2fp0s8iufTQKP3rZRWVLagFQ@mail.gmail.com>

On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:

>  So, it doesn?t seem that zeroing the given vector in the function passed
> to TSSetRHSJacobian is the problem.  When I do that, it just zeros out the
> solution.
>

I would think you would zero the residual vector (if you add to it to
construct the residual, as in FEM methods), not the solution.


> The function that is passed to TSSetRHSJacobian has only one
> responsibility ? to create the jacobian ? correct?  In my case this is
> A(t).  The solution vector is given for when you are solving nonlinear
> problems (A(t) also depends on U(t)).  In my case, I don?t even look at the
> solution vector (because my A(t) doesn?t depend on it).
>

Are you initializing the Jacobian to 0 first?

  Thanks,

     Matt


> Is this the case? or is there some other responsibility of said function?
>
> -Andrew
>
> >Ah ha!
> >
> >The function passed to TSSetRHSJacobian needs to zero the solution vector?
> >
> >As a point, this isn?t mentioned in any documentation that I can find.
> >
> >-Andrew
>
> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>,
> wrote:
>
>> This sounds like a problem in your calculation function where a Vec or
>> Mat does not get reset to 0, but it does in your by hand code.
>>
>>    Matt
>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>>
>>>  I have a fairly simple problem that I?m trying to timestep:
>>>
>>> u? = A(t) u
>>>
>>> I?m using the crank-nicholson method, which I understand (for this
>>> problem) to be:
>>>
>>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>> or
>>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>>
>>> When I attempt to timestep using PETSc, the norm of `u` blows up.  When
>>> I do it directly (using the above), the norm of `u` doesn?t blow up.
>>>
>>> It is important to note that the solution generated after the first step
>>> is identical for both, but the second step for Petsc has a norm of ~2,
>>> while for the directly calculated version it is ~1.  The third step for
>>> petsc has a norm of ~4, while the directly calculated version it is still
>>> ~1.
>>>
>>> I?m not sure what I?m doing wrong.
>>>
>>> PETSc code is taken out of the manual and is pretty simple:
>>>
>>>          TSCreate( comm, &ts );
>>>         TSSetProblemType( ts, TS_LINEAR);
>>>         TSSetType( ts, TSCN );
>>>         TSSetInitialTimeStep( ts, 0, 0.01 );
>>>         TSSetDuration( ts, 5, 0.03 );
>>>         TSSetFromOptions( ts );
>>>         TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>>         TSSetRHSJacobian( ts, A, A, func, &cntx );
>>>         TSSolve( ts, psi0 );
>>>
>>> `func` just constructs A(t) at the time given.  The same code for
>>> calculating A(t) is used in both calculations, along with the same initial
>>> vector psi0, and the same time steps.
>>>
>>> Let me know what other information is needed.  I?m not sure what could
>>> be the problem.  `func` doesn?t touch U at all (should it?).
>>>
>>> -Andrew
>>>
>>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/21d8f391/attachment.html>

From ansp6066 at colorado.edu  Fri Mar 20 17:00:08 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Fri, 20 Mar 2015 15:00:08 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <CAMYG4GkuaZBwT=Ja9F0uNyh2jA2fp0s8iufTQKP3rZRWVLagFQ@mail.gmail.com>
References: <CAMYG4GkuaZBwT=Ja9F0uNyh2jA2fp0s8iufTQKP3rZRWVLagFQ@mail.gmail.com>
Message-ID: <1426888808165.66abbd55@Nodemailer>

I?m sorry, I?m not trying to be difficult, but I?m not following.




The manual states (for my special case):


			
				
					
							u ? = A(t)u. Use





							TSSetProblemType(ts,TS LINEAR);
TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);





							where YourComputeRHSJacobian() is a function you provide that computes A as a function of
time. Or use ...

						

					

			

		


My `func` does this. ?It is 7 lines:





? ? context* c = static_cast<context*>( G_u );

? ? PetscScalar e = c->E( t_ );

? ? MatCopy( c->D, A, SAME_NONZERO_PATTERN );

? ? MatShift( A, e );

? ? MatDiagonalSet( A, c->H0, INSERT_VALUES);

? ? MatShift( A, std::complex<double>( 0, -1 ) );

? ? return 0;






SHOULD `func` touch U? ?If so, what should `func` do to U? ?I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).




-Andrew

On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com>
wrote:

> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>  So, it doesn?t seem that zeroing the given vector in the function passed
>> to TSSetRHSJacobian is the problem.  When I do that, it just zeros out the
>> solution.
>>
> I would think you would zero the residual vector (if you add to it to
> construct the residual, as in FEM methods), not the solution.
>> The function that is passed to TSSetRHSJacobian has only one
>> responsibility ? to create the jacobian ? correct?  In my case this is
>> A(t).  The solution vector is given for when you are solving nonlinear
>> problems (A(t) also depends on U(t)).  In my case, I don?t even look at the
>> solution vector (because my A(t) doesn?t depend on it).
>>
> Are you initializing the Jacobian to 0 first?
>   Thanks,
>      Matt
>> Is this the case? or is there some other responsibility of said function?
>>
>> -Andrew
>>
>> >Ah ha!
>> >
>> >The function passed to TSSetRHSJacobian needs to zero the solution vector?
>> >
>> >As a point, this isn?t mentioned in any documentation that I can find.
>> >
>> >-Andrew
>>
>> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>,
>> wrote:
>>
>>> This sounds like a problem in your calculation function where a Vec or
>>> Mat does not get reset to 0, but it does in your by hand code.
>>>
>>>    Matt
>>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>>>
>>>>  I have a fairly simple problem that I?m trying to timestep:
>>>>
>>>> u? = A(t) u
>>>>
>>>> I?m using the crank-nicholson method, which I understand (for this
>>>> problem) to be:
>>>>
>>>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>>> or
>>>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>>>
>>>> When I attempt to timestep using PETSc, the norm of `u` blows up.  When
>>>> I do it directly (using the above), the norm of `u` doesn?t blow up.
>>>>
>>>> It is important to note that the solution generated after the first step
>>>> is identical for both, but the second step for Petsc has a norm of ~2,
>>>> while for the directly calculated version it is ~1.  The third step for
>>>> petsc has a norm of ~4, while the directly calculated version it is still
>>>> ~1.
>>>>
>>>> I?m not sure what I?m doing wrong.
>>>>
>>>> PETSc code is taken out of the manual and is pretty simple:
>>>>
>>>>          TSCreate( comm, &ts );
>>>>         TSSetProblemType( ts, TS_LINEAR);
>>>>         TSSetType( ts, TSCN );
>>>>         TSSetInitialTimeStep( ts, 0, 0.01 );
>>>>         TSSetDuration( ts, 5, 0.03 );
>>>>         TSSetFromOptions( ts );
>>>>         TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>>>         TSSetRHSJacobian( ts, A, A, func, &cntx );
>>>>         TSSolve( ts, psi0 );
>>>>
>>>> `func` just constructs A(t) at the time given.  The same code for
>>>> calculating A(t) is used in both calculations, along with the same initial
>>>> vector psi0, and the same time steps.
>>>>
>>>> Let me know what other information is needed.  I?m not sure what could
>>>> be the problem.  `func` doesn?t touch U at all (should it?).
>>>>
>>>> -Andrew
>>>>
>>>>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/cf168150/attachment.html>

From bsmith at mcs.anl.gov  Fri Mar 20 17:45:02 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 17:45:02 -0500
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <1426888808165.66abbd55@Nodemailer>
References: <CAMYG4GkuaZBwT=Ja9F0uNyh2jA2fp0s8iufTQKP3rZRWVLagFQ@mail.gmail.com>
	<1426888808165.66abbd55@Nodemailer>
Message-ID: <DD55795F-5404-41E2-89AC-767E65D7274C@mcs.anl.gov>

Andrew,

  Send your entire code. It will be easier and faster than talking past each other.

  Barry

> On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
> 
> I?m sorry, I?m not trying to be difficult, but I?m not following.
> 
> The manual states (for my special case):
> 	? u ? = A(t)u. Use
> 
> TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
> 
> where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ...
> My `func` does this.  It is 7 lines:
> 
>     context* c = static_cast<context*>( G_u );
>     PetscScalar e = c->E( t_ );
>     MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>     MatShift( A, e );
>     MatDiagonalSet( A, c->H0, INSERT_VALUES);
>     MatShift( A, std::complex<double>( 0, -1 ) );
>     return 0;
> 
> SHOULD `func` touch U?  If so, what should `func` do to U?  I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).
> 
> -Andrew
> 
> 
> 
> On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
> So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem.  When I do that, it just zeros out the solution.
> 
> I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution.
>  
> The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct?  In my case this is A(t).  The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)).  In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).
> 
> Are you initializing the Jacobian to 0 first?
> 
>   Thanks,
> 
>      Matt
>  
> Is this the case? or is there some other responsibility of said function?
> 
> -Andrew
> 
> >Ah ha!
> >
> >The function passed to TSSetRHSJacobian needs to zero the solution vector?
> >
> >As a point, this isn?t mentioned in any documentation that I can find.
> >
> >-Andrew
> 
> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:
> This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.
> 
>    Matt
> 
> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
> I have a fairly simple problem that I?m trying to timestep:
> 
> u? = A(t) u
> 
> I?m using the crank-nicholson method, which I understand (for this problem) to be:
> 
> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
> or
> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
> 
> When I attempt to timestep using PETSc, the norm of `u` blows up.  When I do it directly (using the above), the norm of `u` doesn?t blow up.
> 
> It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1.  The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.
> 
> I?m not sure what I?m doing wrong.
> 
> PETSc code is taken out of the manual and is pretty simple:
> 
>         TSCreate( comm, &ts );
>         TSSetProblemType( ts, TS_LINEAR);
>         TSSetType( ts, TSCN );
>         TSSetInitialTimeStep( ts, 0, 0.01 );
>         TSSetDuration( ts, 5, 0.03 );
>         TSSetFromOptions( ts );
>         TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>         TSSetRHSJacobian( ts, A, A, func, &cntx );
>         TSSolve( ts, psi0 );
> 
> `func` just constructs A(t) at the time given.  The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.
> 
> Let me know what other information is needed.  I?m not sure what could be the problem.  `func` doesn?t touch U at all (should it?).
> 
> -Andrew
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From ansp6066 at colorado.edu  Fri Mar 20 18:39:50 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Fri, 20 Mar 2015 16:39:50 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <DD55795F-5404-41E2-89AC-767E65D7274C@mcs.anl.gov>
References: <DD55795F-5404-41E2-89AC-767E65D7274C@mcs.anl.gov>
Message-ID: <1426894790492.f3f66657@Nodemailer>

Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)





https://gist.github.com/spott/aea8070f35e79e7249e6




The first section does it using the time stepper. ?The second section does it by explicitly doing the steps. ?The output is:




//first section, using TimeStepper:


t: 0 step: 0 norm-1: 0

t: 0.01 step: 1 norm-1: 0

t: 0.02 step: 2 norm-1: 0.999995

t: 0.03 step: 3 norm-1: 2.99998




//Second section, using explicit code.

t: 0.01 norm-1: 0

t: 0.02 norm-1: 0

t: 0.02 norm-1: 2.22045e-16

On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> Andrew,
>   Send your entire code. It will be easier and faster than talking past each other.
>   Barry
>> On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>> 
>> I?m sorry, I?m not trying to be difficult, but I?m not following.
>> 
>> The manual states (for my special case):
>> 	? u ? = A(t)u. Use
>> 
>> TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>> 
>> where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ...
>> My `func` does this.  It is 7 lines:
>> 
>>     context* c = static_cast<context*>( G_u );
>>     PetscScalar e = c->E( t_ );
>>     MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>>     MatShift( A, e );
>>     MatDiagonalSet( A, c->H0, INSERT_VALUES);
>>     MatShift( A, std::complex<double>( 0, -1 ) );
>>     return 0;
>> 
>> SHOULD `func` touch U?  If so, what should `func` do to U?  I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).
>> 
>> -Andrew
>> 
>> 
>> 
>> On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>> So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem.  When I do that, it just zeros out the solution.
>> 
>> I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution.
>>  
>> The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct?  In my case this is A(t).  The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)).  In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).
>> 
>> Are you initializing the Jacobian to 0 first?
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> Is this the case? or is there some other responsibility of said function?
>> 
>> -Andrew
>> 
>> >Ah ha!
>> >
>> >The function passed to TSSetRHSJacobian needs to zero the solution vector?
>> >
>> >As a point, this isn?t mentioned in any documentation that I can find.
>> >
>> >-Andrew
>> 
>> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:
>> This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.
>> 
>>    Matt
>> 
>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>> I have a fairly simple problem that I?m trying to timestep:
>> 
>> u? = A(t) u
>> 
>> I?m using the crank-nicholson method, which I understand (for this problem) to be:
>> 
>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>> or
>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>> 
>> When I attempt to timestep using PETSc, the norm of `u` blows up.  When I do it directly (using the above), the norm of `u` doesn?t blow up.
>> 
>> It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1.  The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.
>> 
>> I?m not sure what I?m doing wrong.
>> 
>> PETSc code is taken out of the manual and is pretty simple:
>> 
>>         TSCreate( comm, &ts );
>>         TSSetProblemType( ts, TS_LINEAR);
>>         TSSetType( ts, TSCN );
>>         TSSetInitialTimeStep( ts, 0, 0.01 );
>>         TSSetDuration( ts, 5, 0.03 );
>>         TSSetFromOptions( ts );
>>         TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>         TSSetRHSJacobian( ts, A, A, func, &cntx );
>>         TSSolve( ts, psi0 );
>> 
>> `func` just constructs A(t) at the time given.  The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.
>> 
>> Let me know what other information is needed.  I?m not sure what could be the problem.  `func` doesn?t touch U at all (should it?).
>> 
>> -Andrew
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/02a9a356/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri Mar 20 20:25:55 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 20:25:55 -0500
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <1426894790492.f3f66657@Nodemailer>
References: <DD55795F-5404-41E2-89AC-767E65D7274C@mcs.anl.gov>
	<1426894790492.f3f66657@Nodemailer>
Message-ID: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>

Data files are needed

 PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
    VecLoad( H0, view );
    PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );

BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded.

  Barry


> On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
> 
> Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)
> 
> https://gist.github.com/spott/aea8070f35e79e7249e6
> 
> The first section does it using the time stepper.  The second section does it by explicitly doing the steps.  The output is:
> 
> //first section, using TimeStepper:
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0.999995
> t: 0.03 step: 3 norm-1: 2.99998
> 
> //Second section, using explicit code.
> t: 0.01 norm-1: 0
> t: 0.02 norm-1: 0
> t: 0.02 norm-1: 2.22045e-16
> 
> 
> 
> On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> Andrew, 
> 
> Send your entire code. It will be easier and faster than talking past each other. 
> 
> Barry 
> 
> > On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > 
> > I?m sorry, I?m not trying to be difficult, but I?m not following. 
> > 
> > The manual states (for my special case): 
> > ? u ? = A(t)u. Use 
> > 
> > TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx); 
> > 
> > where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ... 
> > My `func` does this. It is 7 lines: 
> > 
> > context* c = static_cast<context*>( G_u ); 
> > PetscScalar e = c->E( t_ ); 
> > MatCopy( c->D, A, SAME_NONZERO_PATTERN ); 
> > MatShift( A, e ); 
> > MatDiagonalSet( A, c->H0, INSERT_VALUES); 
> > MatShift( A, std::complex<double>( 0, -1 ) ); 
> > return 0; 
> > 
> > SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case). 
> > 
> > -Andrew 
> > 
> > 
> > 
> > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote: 
> > 
> > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution. 
> > 
> > I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution. 
> > 
> > The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it). 
> > 
> > Are you initializing the Jacobian to 0 first? 
> > 
> > Thanks, 
> > 
> > Matt 
> > 
> > Is this the case? or is there some other responsibility of said function? 
> > 
> > -Andrew 
> > 
> > >Ah ha! 
> > > 
> > >The function passed to TSSetRHSJacobian needs to zero the solution vector? 
> > > 
> > >As a point, this isn?t mentioned in any documentation that I can find. 
> > > 
> > >-Andrew 
> > 
> > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote: 
> > This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code. 
> > 
> > Matt 
> > 
> > On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote: 
> > I have a fairly simple problem that I?m trying to timestep: 
> > 
> > u? = A(t) u 
> > 
> > I?m using the crank-nicholson method, which I understand (for this problem) to be: 
> > 
> > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)] 
> > or 
> > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t) 
> > 
> > When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up. 
> > 
> > It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1. 
> > 
> > I?m not sure what I?m doing wrong. 
> > 
> > PETSc code is taken out of the manual and is pretty simple: 
> > 
> > TSCreate( comm, &ts ); 
> > TSSetProblemType( ts, TS_LINEAR); 
> > TSSetType( ts, TSCN ); 
> > TSSetInitialTimeStep( ts, 0, 0.01 ); 
> > TSSetDuration( ts, 5, 0.03 ); 
> > TSSetFromOptions( ts ); 
> > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL ); 
> > TSSetRHSJacobian( ts, A, A, func, &cntx ); 
> > TSSolve( ts, psi0 ); 
> > 
> > `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps. 
> > 
> > Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?). 
> > 
> > -Andrew 
> > 
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> > -- Norbert Wiener 
> > 
> 
> 
> 


From hus003 at ucsd.edu  Fri Mar 20 20:30:43 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sat, 21 Mar 2015 01:30:43 +0000
Subject: [petsc-users] Parallelize the Schur complement
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010F2520@XMAIL-MBX-BH1.AD.UCSD.EDU>

I have a KSP with Mat [A00, A01; A10, A11], and I want to do Schur complement for the Mat
S=A11-A10*A00^(-1)*A01

How do I parallelize it? I mean, I use DMCreateMatrix to create A00, and I can parallelize the rows of A01 and A10^t by the same DM object. But I don't know how to parallelize the columns of A01 and A10^t. So right now, I can form a sequential S, and solve for the KSP that is corresponding to S sequentially. How can I create a parallelized S?

Thank you!

Best,
Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150321/a93a8a67/attachment.html>

From ansp6066 at colorado.edu  Fri Mar 20 22:18:55 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Fri, 20 Mar 2015 20:18:55 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems.
In-Reply-To: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
Message-ID: <1426907935637.9e8be15c@Nodemailer>

here are the data files.


dipole_matrix.dat:


https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0




energy_eigenvalues_vector.dat


https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0




-Andrew

On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> Data files are needed
>  PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>     VecLoad( H0, view );
>     PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded.
>   Barry
>> On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>> 
>> Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)
>> 
>> https://gist.github.com/spott/aea8070f35e79e7249e6
>> 
>> The first section does it using the time stepper.  The second section does it by explicitly doing the steps.  The output is:
>> 
>> //first section, using TimeStepper:
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0.999995
>> t: 0.03 step: 3 norm-1: 2.99998
>> 
>> //Second section, using explicit code.
>> t: 0.01 norm-1: 0
>> t: 0.02 norm-1: 0
>> t: 0.02 norm-1: 2.22045e-16
>> 
>> 
>> 
>> On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> Andrew, 
>> 
>> Send your entire code. It will be easier and faster than talking past each other. 
>> 
>> Barry 
>> 
>> > On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > 
>> > I?m sorry, I?m not trying to be difficult, but I?m not following. 
>> > 
>> > The manual states (for my special case): 
>> > ? u ? = A(t)u. Use 
>> > 
>> > TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx); 
>> > 
>> > where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ... 
>> > My `func` does this. It is 7 lines: 
>> > 
>> > context* c = static_cast<context*>( G_u ); 
>> > PetscScalar e = c->E( t_ ); 
>> > MatCopy( c->D, A, SAME_NONZERO_PATTERN ); 
>> > MatShift( A, e ); 
>> > MatDiagonalSet( A, c->H0, INSERT_VALUES); 
>> > MatShift( A, std::complex<double>( 0, -1 ) ); 
>> > return 0; 
>> > 
>> > SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case). 
>> > 
>> > -Andrew 
>> > 
>> > 
>> > 
>> > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote: 
>> > 
>> > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution. 
>> > 
>> > I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution. 
>> > 
>> > The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it). 
>> > 
>> > Are you initializing the Jacobian to 0 first? 
>> > 
>> > Thanks, 
>> > 
>> > Matt 
>> > 
>> > Is this the case? or is there some other responsibility of said function? 
>> > 
>> > -Andrew 
>> > 
>> > >Ah ha! 
>> > > 
>> > >The function passed to TSSetRHSJacobian needs to zero the solution vector? 
>> > > 
>> > >As a point, this isn?t mentioned in any documentation that I can find. 
>> > > 
>> > >-Andrew 
>> > 
>> > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote: 
>> > This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code. 
>> > 
>> > Matt 
>> > 
>> > On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote: 
>> > I have a fairly simple problem that I?m trying to timestep: 
>> > 
>> > u? = A(t) u 
>> > 
>> > I?m using the crank-nicholson method, which I understand (for this problem) to be: 
>> > 
>> > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)] 
>> > or 
>> > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t) 
>> > 
>> > When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up. 
>> > 
>> > It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1. 
>> > 
>> > I?m not sure what I?m doing wrong. 
>> > 
>> > PETSc code is taken out of the manual and is pretty simple: 
>> > 
>> > TSCreate( comm, &ts ); 
>> > TSSetProblemType( ts, TS_LINEAR); 
>> > TSSetType( ts, TSCN ); 
>> > TSSetInitialTimeStep( ts, 0, 0.01 ); 
>> > TSSetDuration( ts, 5, 0.03 ); 
>> > TSSetFromOptions( ts ); 
>> > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL ); 
>> > TSSetRHSJacobian( ts, A, A, func, &cntx ); 
>> > TSSolve( ts, psi0 ); 
>> > 
>> > `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps. 
>> > 
>> > Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?). 
>> > 
>> > -Andrew 
>> > 
>> > 
>> > 
>> > 
>> > -- 
>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> > -- Norbert Wiener 
>> > 
>> 
>> 
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/ef0fae6d/attachment.html>

From bsmith at mcs.anl.gov  Fri Mar 20 22:57:10 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Mar 2015 22:57:10 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1426907935637.9e8be15c@Nodemailer>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
Message-ID: <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>

 
  Andrew,

   I'm afraid Emil will have to take a look at this and explain it.  The -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but the -ts_type cn is not stable. It turns out that -ts_type cn is equivalent to -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint and somehow this endpoint business (which I don't understand) is causing a problem. Meanwhile if I add -ts_theta_adapt to the endpoint one it becomes stable ? Anyways all cases are displayed below.

  Emil,

    What's up with this? Does the endpoint business have a bug or can it not be used for this problem (the matrix A is a function of t.)

  Barry


$ ./ex2  -ts_type cn
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 1
t: 0.03 step: 3 norm-1: 3
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 0
t: 0.03 step: 3 norm-1: 0
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta -ts_theta_theta .5
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 0
t: 0.03 step: 3 norm-1: 0
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 1
t: 0.03 step: 3 norm-1: 3
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 0
t: 0.03 step: 3 norm-1: 0
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor
0 TS dt 0.01 time 0 
t: 0 step: 0 norm-1: 0
0 TS dt 0.01 time 0 
1 TS dt 0.1 time 0.01 
t: 0.01 step: 1 norm-1: 0
1 TS dt 0.1 time 0.01 
2 TS dt 0.1 time 0.02 
t: 0.02 step: 2 norm-1: 0
2 TS dt 0.1 time 0.02 
3 TS dt 0.1 time 0.03 
t: 0.03 step: 3 norm-1: 0
3 TS dt 0.1 time 0.03 
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor -ts_adapt_monitor
0 TS dt 0.01 time 0 
t: 0 step: 0 norm-1: 0
0 TS dt 0.01 time 0 
      TSAdapt 'basic': step   0 accepted t=0          + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
1 TS dt 0.1 time 0.01 
t: 0.01 step: 1 norm-1: 0
1 TS dt 0.1 time 0.01 
      TSAdapt 'basic': step   1 rejected t=0.01       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02 
      TSAdapt 'basic': step   1 accepted t=0.01       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
2 TS dt 0.1 time 0.02 
t: 0.02 step: 2 norm-1: 0
2 TS dt 0.1 time 0.02 
      TSAdapt 'basic': step   2 rejected t=0.02       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02 
      TSAdapt 'basic': step   2 accepted t=0.02       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
3 TS dt 0.1 time 0.03 
t: 0.03 step: 3 norm-1: 0
3 TS dt 0.1 time 0.03 
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type beuler
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 0
t: 0.03 step: 3 norm-1: 0
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
$ ./ex2  -ts_type euler
t: 0 step: 0 norm-1: 0
t: 0.01 step: 1 norm-1: 0
t: 0.02 step: 2 norm-1: 0
t: 0.03 step: 3 norm-1: 0
~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug


> On Mar 20, 2015, at 10:18 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
> 
> here are the data files.
> 
> dipole_matrix.dat:
> https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
> 
> energy_eigenvalues_vector.dat
> https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
> 
> -Andrew
> 
> 
> 
> On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> Data files are needed 
> 
> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view ); 
> VecLoad( H0, view ); 
> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view ); 
> 
> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded. 
> 
> Barry 
> 
> 
> > On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > 
> > Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?) 
> > 
> > https://gist.github.com/spott/aea8070f35e79e7249e6 
> > 
> > The first section does it using the time stepper. The second section does it by explicitly doing the steps. The output is: 
> > 
> > //first section, using TimeStepper: 
> > t: 0 step: 0 norm-1: 0 
> > t: 0.01 step: 1 norm-1: 0 
> > t: 0.02 step: 2 norm-1: 0.999995 
> > t: 0.03 step: 3 norm-1: 2.99998 
> > 
> > //Second section, using explicit code. 
> > t: 0.01 norm-1: 0 
> > t: 0.02 norm-1: 0 
> > t: 0.02 norm-1: 2.22045e-16 
> > 
> > 
> > 
> > On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote: 
> > 
> > Andrew, 
> > 
> > Send your entire code. It will be easier and faster than talking past each other. 
> > 
> > Barry 
> > 
> > > On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > > 
> > > I?m sorry, I?m not trying to be difficult, but I?m not following. 
> > > 
> > > The manual states (for my special case): 
> > > ? u ? = A(t)u. Use 
> > > 
> > > TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx); 
> > > 
> > > where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ... 
> > > My `func` does this. It is 7 lines: 
> > > 
> > > context* c = static_cast<context*>( G_u ); 
> > > PetscScalar e = c->E( t_ ); 
> > > MatCopy( c->D, A, SAME_NONZERO_PATTERN ); 
> > > MatShift( A, e ); 
> > > MatDiagonalSet( A, c->H0, INSERT_VALUES); 
> > > MatShift( A, std::complex<double>( 0, -1 ) ); 
> > > return 0; 
> > > 
> > > SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case). 
> > > 
> > > -Andrew 
> > > 
> > > 
> > > 
> > > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote: 
> > > 
> > > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
> > > So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution. 
> > > 
> > > I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution. 
> > > 
> > > The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it). 
> > > 
> > > Are you initializing the Jacobian to 0 first? 
> > > 
> > > Thanks, 
> > > 
> > > Matt 
> > > 
> > > Is this the case? or is there some other responsibility of said function? 
> > > 
> > > -Andrew 
> > > 
> > > >Ah ha! 
> > > > 
> > > >The function passed to TSSetRHSJacobian needs to zero the solution vector? 
> > > > 
> > > >As a point, this isn?t mentioned in any documentation that I can find. 
> > > > 
> > > >-Andrew 
> > > 
> > > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote: 
> > > This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code. 
> > > 
> > > Matt 
> > > 
> > > On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote: 
> > > I have a fairly simple problem that I?m trying to timestep: 
> > > 
> > > u? = A(t) u 
> > > 
> > > I?m using the crank-nicholson method, which I understand (for this problem) to be: 
> > > 
> > > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)] 
> > > or 
> > > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t) 
> > > 
> > > When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up. 
> > > 
> > > It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1. 
> > > 
> > > I?m not sure what I?m doing wrong. 
> > > 
> > > PETSc code is taken out of the manual and is pretty simple: 
> > > 
> > > TSCreate( comm, &ts ); 
> > > TSSetProblemType( ts, TS_LINEAR); 
> > > TSSetType( ts, TSCN ); 
> > > TSSetInitialTimeStep( ts, 0, 0.01 ); 
> > > TSSetDuration( ts, 5, 0.03 ); 
> > > TSSetFromOptions( ts ); 
> > > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL ); 
> > > TSSetRHSJacobian( ts, A, A, func, &cntx ); 
> > > TSSolve( ts, psi0 ); 
> > > 
> > > `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps. 
> > > 
> > > Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?). 
> > > 
> > > -Andrew 
> > > 
> > > 
> > > 
> > > 
> > > -- 
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> > > -- Norbert Wiener 
> > > 
> > 
> > 
> > 
> 
> 
> 


From jed at jedbrown.org  Sat Mar 21 00:32:00 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 20 Mar 2015 23:32:00 -0600
Subject: [petsc-users] Parallelize the Schur complement
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010F2520@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010F2520@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <877fua9ar3.fsf@jedbrown.org>

"Sun, Hui" <hus003 at ucsd.edu> writes:
> How do I parallelize it? I mean, I use DMCreateMatrix to create A00,
> and I can parallelize the rows of A01 and A10^t by the same DM
> object. But I don't know how to parallelize the columns of A01 and
> A10^t. 

You create the matrix.  We don't have DM support for non-square
matrices, but you can get their sizes.  You might consider using
DMComposite or a multi-field DM when creating the matrix.  It's also
pretty likely that you don't want to actually form the Schur complement
(usually dense), and instead would be better off approximating it and
solving with it iteratively.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150320/eaec9d7b/attachment.pgp>

From hus003 at ucsd.edu  Sat Mar 21 01:02:35 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Sat, 21 Mar 2015 06:02:35 +0000
Subject: [petsc-users] Parallelize the Schur complement
In-Reply-To: <877fua9ar3.fsf@jedbrown.org>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010F2520@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<877fua9ar3.fsf@jedbrown.org>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010F254F@XMAIL-MBX-BH1.AD.UCSD.EDU>

Hi Jed, thank you for your answers. However, I still don't think I get the answer. Maybe I should ask in a clearer way. 

My A00 is of size n^3 times n^3, while my A11 is of size m times m. A00 has DM, but A11 does not. 

Let's suppose I have 16 cores, and I use all 16 cores to create the DM for A00, and my A01 and A10' has 16 cores parallelization in rows, but they are sequential in columns, and A11 is also sequential. 

I want to make A11 parallelized. So maybe I can try the following, I use 8 cores to create the DM for A00, and when I do A10*A00^(-1)*A01*v, I need the same 8 cores for the rows of A01 and A10' so that the matrix multiplications can carry over. But I want to parallelize A11 as well, so maybe I want to use 2 cores for the rows of A11 and 2 cores for the columns of A11, and hence I should also have 2 cores for the columns of A01 and A10'. Then, for matrix A00 I use 8 cores, and for A10 and A01 I use 8 times 2 which is 16 cores, and for A11 I use 4 cores. However, this doesn't seem right, because since there are 16 cores for A01, these 16 cores should all have access to part of the matrix A00 because we have the operation A00^(-1)*A01. But I use 8 cores for DM. That means I should have two copies of A00, which doesn't seem quite reasonable. 

But what is a reasonable way to do this task: that is, to parallelize all four components A00, A01, A10, A11 in a consistent, efficient, and natural way. 

Best,
Hui


________________________________________
From: Jed Brown [jed at jedbrown.org]
Sent: Friday, March 20, 2015 10:32 PM
To: Sun, Hui; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Parallelize the Schur complement

"Sun, Hui" <hus003 at ucsd.edu> writes:
> How do I parallelize it? I mean, I use DMCreateMatrix to create A00,
> and I can parallelize the rows of A01 and A10^t by the same DM
> object. But I don't know how to parallelize the columns of A01 and
> A10^t.

You create the matrix.  We don't have DM support for non-square
matrices, but you can get their sizes.  You might consider using
DMComposite or a multi-field DM when creating the matrix.  It's also
pretty likely that you don't want to actually form the Schur complement
(usually dense), and instead would be better off approximating it and
solving with it iteratively.

From emconsta at mcs.anl.gov  Sat Mar 21 09:32:33 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Sat, 21 Mar 2015 09:32:33 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
Message-ID: <550D8101.6070006@mcs.anl.gov>

I haven't been able to compile and run. But here are a few quick notes.

The problem appears to be very stiff.

Theta and theta_endpoint are defining different methods:

1) -ts_type beuler OR -ts_theta_theta 1.0: is backward Euler

u(t + h) = u(t) + h*A(t+h)*u(t+h)

2) -ts_theta_theta 0.5: is the implicit midpoint rule

u(t + h) = u(t) + h*[A(t+h/2)*(u(t+h)+u(t))/2]

3) -ts_type cn OR -ts_theta_theta 0.5 -ts_theta_endpoint: is 
Crank-Nicholson/trapezoidal

u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]

Note that the last two are different. -ts_type theta -ts_theta_theta .5 
is different from -ts_type cn. They the same linear stability properties 
if A(t)=A; but not if A depends on t.

When -ts_theta_adapt is used, then it detects the instability as an 
error and reduces the step by a lot! wlte=1.24e+03 which means that the 
reduction should be severe but the controller tries 0.1*dt and that 
seems to pass but it "jig-saws" (take a look at the next attempted 
step), which means that it is likely unstable.

I'll try to build the example to get more insight.

Emil

On 3/20/15 10:57 PM, Barry Smith wrote:
>
>    Andrew,
>
>     I'm afraid Emil will have to take a look at this and explain it.  The -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but the -ts_type cn is not stable. It turns out that -ts_type cn is equivalent to -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint and somehow this endpoint business (which I don't understand) is causing a problem. Meanwhile if I add -ts_theta_adapt to the endpoint one it becomes stable ? Anyways all cases are displayed below.
>
>    Emil,
>
>      What's up with this? Does the endpoint business have a bug or can it not be used for this problem (the matrix A is a function of t.)
>
>    Barry
>
>
> $ ./ex2  -ts_type cn
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 1
> t: 0.03 step: 3 norm-1: 3
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 1
> t: 0.03 step: 3 norm-1: 3
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor
> 0 TS dt 0.01 time 0
> t: 0 step: 0 norm-1: 0
> 0 TS dt 0.01 time 0
> 1 TS dt 0.1 time 0.01
> t: 0.01 step: 1 norm-1: 0
> 1 TS dt 0.1 time 0.01
> 2 TS dt 0.1 time 0.02
> t: 0.02 step: 2 norm-1: 0
> 2 TS dt 0.1 time 0.02
> 3 TS dt 0.1 time 0.03
> t: 0.03 step: 3 norm-1: 0
> 3 TS dt 0.1 time 0.03
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor -ts_adapt_monitor
> 0 TS dt 0.01 time 0
> t: 0 step: 0 norm-1: 0
> 0 TS dt 0.01 time 0
>        TSAdapt 'basic': step   0 accepted t=0          + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
> 1 TS dt 0.1 time 0.01
> t: 0.01 step: 1 norm-1: 0
> 1 TS dt 0.1 time 0.01
>        TSAdapt 'basic': step   1 rejected t=0.01       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>        TSAdapt 'basic': step   1 accepted t=0.01       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
> 2 TS dt 0.1 time 0.02
> t: 0.02 step: 2 norm-1: 0
> 2 TS dt 0.1 time 0.02
>        TSAdapt 'basic': step   2 rejected t=0.02       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>        TSAdapt 'basic': step   2 accepted t=0.02       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
> 3 TS dt 0.1 time 0.03
> t: 0.03 step: 3 norm-1: 0
> 3 TS dt 0.1 time 0.03
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type beuler
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type euler
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>
>
>> On Mar 20, 2015, at 10:18 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>
>> here are the data files.
>>
>> dipole_matrix.dat:
>> https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>>
>> energy_eigenvalues_vector.dat
>> https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>>
>> -Andrew
>>
>>
>>
>> On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>> Data files are needed
>>
>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>> VecLoad( H0, view );
>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
>>
>> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded.
>>
>> Barry
>>
>>
>>> On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>
>>> Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)
>>>
>>> https://gist.github.com/spott/aea8070f35e79e7249e6
>>>
>>> The first section does it using the time stepper. The second section does it by explicitly doing the steps. The output is:
>>>
>>> //first section, using TimeStepper:
>>> t: 0 step: 0 norm-1: 0
>>> t: 0.01 step: 1 norm-1: 0
>>> t: 0.02 step: 2 norm-1: 0.999995
>>> t: 0.03 step: 3 norm-1: 2.99998
>>>
>>> //Second section, using explicit code.
>>> t: 0.01 norm-1: 0
>>> t: 0.02 norm-1: 0
>>> t: 0.02 norm-1: 2.22045e-16
>>>
>>>
>>>
>>> On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>> Andrew,
>>>
>>> Send your entire code. It will be easier and faster than talking past each other.
>>>
>>> Barry
>>>
>>>> On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>
>>>> I?m sorry, I?m not trying to be difficult, but I?m not following.
>>>>
>>>> The manual states (for my special case):
>>>> ? u ? = A(t)u. Use
>>>>
>>>> TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>>>>
>>>> where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ...
>>>> My `func` does this. It is 7 lines:
>>>>
>>>> context* c = static_cast<context*>( G_u );
>>>> PetscScalar e = c->E( t_ );
>>>> MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>>>> MatShift( A, e );
>>>> MatDiagonalSet( A, c->H0, INSERT_VALUES);
>>>> MatShift( A, std::complex<double>( 0, -1 ) );
>>>> return 0;
>>>>
>>>> SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).
>>>>
>>>> -Andrew
>>>>
>>>>
>>>>
>>>> On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>
>>>> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>> So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution.
>>>>
>>>> I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution.
>>>>
>>>> The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).
>>>>
>>>> Are you initializing the Jacobian to 0 first?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>> Is this the case? or is there some other responsibility of said function?
>>>>
>>>> -Andrew
>>>>
>>>>> Ah ha!
>>>>>
>>>>> The function passed to TSSetRHSJacobian needs to zero the solution vector?
>>>>>
>>>>> As a point, this isn?t mentioned in any documentation that I can find.
>>>>>
>>>>> -Andrew
>>>>
>>>> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:
>>>> This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.
>>>>
>>>> Matt
>>>>
>>>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>>>> I have a fairly simple problem that I?m trying to timestep:
>>>>
>>>> u? = A(t) u
>>>>
>>>> I?m using the crank-nicholson method, which I understand (for this problem) to be:
>>>>
>>>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>>> or
>>>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>>>
>>>> When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up.
>>>>
>>>> It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.
>>>>
>>>> I?m not sure what I?m doing wrong.
>>>>
>>>> PETSc code is taken out of the manual and is pretty simple:
>>>>
>>>> TSCreate( comm, &ts );
>>>> TSSetProblemType( ts, TS_LINEAR);
>>>> TSSetType( ts, TSCN );
>>>> TSSetInitialTimeStep( ts, 0, 0.01 );
>>>> TSSetDuration( ts, 5, 0.03 );
>>>> TSSetFromOptions( ts );
>>>> TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>>> TSSetRHSJacobian( ts, A, A, func, &cntx );
>>>> TSSolve( ts, psi0 );
>>>>
>>>> `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.
>>>>
>>>> Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?).
>>>>
>>>> -Andrew
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>>
>>
>>
>>
>

From jed at jedbrown.org  Sat Mar 21 09:47:09 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sat, 21 Mar 2015 08:47:09 -0600
Subject: [petsc-users] Parallelize the Schur complement
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010F254F@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010F2520@XMAIL-MBX-BH1.AD.UCSD.EDU>
	<877fua9ar3.fsf@jedbrown.org>
	<7501CC2B7BBCC44A92ECEEC316170ECB010F254F@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <87zj7676he.fsf@jedbrown.org>

"Sun, Hui" <hus003 at ucsd.edu> writes:

> Hi Jed, thank you for your answers. However, I still don't think I get the answer. Maybe I should ask in a clearer way. 
>
> My A00 is of size n^3 times n^3, while my A11 is of size m times m. A00 has DM, but A11 does not. 
>
> Let's suppose I have 16 cores, and I use all 16 cores to create the DM for A00, and my A01 and A10' has 16 cores parallelization in rows, but they are sequential in columns, and A11 is also sequential. 

Why would you make A11 sequential?

> I want to make A11 parallelized. So maybe I can try the following, I
> use 8 cores to create the DM for A00, and when I do
> A10*A00^(-1)*A01*v, I need the same 8 cores for the rows of A01 and
> A10' so that the matrix multiplications can carry over. But I want to
> parallelize A11 as well, so maybe I want to use 2 cores for the rows
> of A11 and 2 cores for the columns of A11, and hence I should also
> have 2 cores for the columns of A01 and A10'. Then, for matrix A00 I
> use 8 cores, and for A10 and A01 I use 8 times 2 which is 16 cores,
> and for A11 I use 4 cores. However, this doesn't seem right, because
> since there are 16 cores for A01, these 16 cores should all have
> access to part of the matrix A00 because we have the operation
> A00^(-1)*A01. But I use 8 cores for DM. That means I should have two
> copies of A00, which doesn't seem quite reasonable.

You're over-thinking this.  Distribute both over all cores (PETSc
matrices are row distributions unless you transpose, so just use the row
distributions).  Set the row distribution of A01 equal to A00 and the
column distribution of A01 equal to A11.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150321/ba360bfc/attachment.pgp>

From bsmith at mcs.anl.gov  Sat Mar 21 11:26:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 21 Mar 2015 11:26:00 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <550D8101.6070006@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
Message-ID: <EC3202AB-69EF-445E-9A81-33C5B47FE9C0@mcs.anl.gov>


  Depending on your C++ compiler you may need to build PETSc with the additional ./configure option --with-cxx-dialect=C++11


> On Mar 21, 2015, at 9:32 AM, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
> 
> I haven't been able to compile and run. But here are a few quick notes.
> 
> The problem appears to be very stiff.
> 
> Theta and theta_endpoint are defining different methods:
> 
> 1) -ts_type beuler OR -ts_theta_theta 1.0: is backward Euler
> 
> u(t + h) = u(t) + h*A(t+h)*u(t+h)
> 
> 2) -ts_theta_theta 0.5: is the implicit midpoint rule
> 
> u(t + h) = u(t) + h*[A(t+h/2)*(u(t+h)+u(t))/2]
> 
> 3) -ts_type cn OR -ts_theta_theta 0.5 -ts_theta_endpoint: is Crank-Nicholson/trapezoidal
> 
> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
> 
> Note that the last two are different. -ts_type theta -ts_theta_theta .5 is different from -ts_type cn. They the same linear stability properties if A(t)=A; but not if A depends on t.
> 
> When -ts_theta_adapt is used, then it detects the instability as an error and reduces the step by a lot! wlte=1.24e+03 which means that the reduction should be severe but the controller tries 0.1*dt and that seems to pass but it "jig-saws" (take a look at the next attempted step), which means that it is likely unstable.
> 
> I'll try to build the example to get more insight.
> 
> Emil
> 
> On 3/20/15 10:57 PM, Barry Smith wrote:
>> 
>>   Andrew,
>> 
>>    I'm afraid Emil will have to take a look at this and explain it.  The -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but the -ts_type cn is not stable. It turns out that -ts_type cn is equivalent to -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint and somehow this endpoint business (which I don't understand) is causing a problem. Meanwhile if I add -ts_theta_adapt to the endpoint one it becomes stable ? Anyways all cases are displayed below.
>> 
>>   Emil,
>> 
>>     What's up with this? Does the endpoint business have a bug or can it not be used for this problem (the matrix A is a function of t.)
>> 
>>   Barry
>> 
>> 
>> $ ./ex2  -ts_type cn
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 1
>> t: 0.03 step: 3 norm-1: 3
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 1
>> t: 0.03 step: 3 norm-1: 3
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor
>> 0 TS dt 0.01 time 0
>> t: 0 step: 0 norm-1: 0
>> 0 TS dt 0.01 time 0
>> 1 TS dt 0.1 time 0.01
>> t: 0.01 step: 1 norm-1: 0
>> 1 TS dt 0.1 time 0.01
>> 2 TS dt 0.1 time 0.02
>> t: 0.02 step: 2 norm-1: 0
>> 2 TS dt 0.1 time 0.02
>> 3 TS dt 0.1 time 0.03
>> t: 0.03 step: 3 norm-1: 0
>> 3 TS dt 0.1 time 0.03
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor -ts_adapt_monitor
>> 0 TS dt 0.01 time 0
>> t: 0 step: 0 norm-1: 0
>> 0 TS dt 0.01 time 0
>>       TSAdapt 'basic': step   0 accepted t=0          + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 1 TS dt 0.1 time 0.01
>> t: 0.01 step: 1 norm-1: 0
>> 1 TS dt 0.1 time 0.01
>>       TSAdapt 'basic': step   1 rejected t=0.01       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>>       TSAdapt 'basic': step   1 accepted t=0.01       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 2 TS dt 0.1 time 0.02
>> t: 0.02 step: 2 norm-1: 0
>> 2 TS dt 0.1 time 0.02
>>       TSAdapt 'basic': step   2 rejected t=0.02       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>>       TSAdapt 'basic': step   2 accepted t=0.02       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 3 TS dt 0.1 time 0.03
>> t: 0.03 step: 3 norm-1: 0
>> 3 TS dt 0.1 time 0.03
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type beuler
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type euler
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> 
>> 
>>> On Mar 20, 2015, at 10:18 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>> 
>>> here are the data files.
>>> 
>>> dipole_matrix.dat:
>>> https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>>> 
>>> energy_eigenvalues_vector.dat
>>> https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>>> 
>>> -Andrew
>>> 
>>> 
>>> 
>>> On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> Data files are needed
>>> 
>>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>>> VecLoad( H0, view );
>>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
>>> 
>>> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded.
>>> 
>>> Barry
>>> 
>>> 
>>>> On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>> 
>>>> Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)
>>>> 
>>>> https://gist.github.com/spott/aea8070f35e79e7249e6
>>>> 
>>>> The first section does it using the time stepper. The second section does it by explicitly doing the steps. The output is:
>>>> 
>>>> //first section, using TimeStepper:
>>>> t: 0 step: 0 norm-1: 0
>>>> t: 0.01 step: 1 norm-1: 0
>>>> t: 0.02 step: 2 norm-1: 0.999995
>>>> t: 0.03 step: 3 norm-1: 2.99998
>>>> 
>>>> //Second section, using explicit code.
>>>> t: 0.01 norm-1: 0
>>>> t: 0.02 norm-1: 0
>>>> t: 0.02 norm-1: 2.22045e-16
>>>> 
>>>> 
>>>> 
>>>> On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> Andrew,
>>>> 
>>>> Send your entire code. It will be easier and faster than talking past each other.
>>>> 
>>>> Barry
>>>> 
>>>>> On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>> 
>>>>> I?m sorry, I?m not trying to be difficult, but I?m not following.
>>>>> 
>>>>> The manual states (for my special case):
>>>>> ? u ? = A(t)u. Use
>>>>> 
>>>>> TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>>>>> 
>>>>> where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ...
>>>>> My `func` does this. It is 7 lines:
>>>>> 
>>>>> context* c = static_cast<context*>( G_u );
>>>>> PetscScalar e = c->E( t_ );
>>>>> MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>>>>> MatShift( A, e );
>>>>> MatDiagonalSet( A, c->H0, INSERT_VALUES);
>>>>> MatShift( A, std::complex<double>( 0, -1 ) );
>>>>> return 0;
>>>>> 
>>>>> SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).
>>>>> 
>>>>> -Andrew
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>> So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution.
>>>>> 
>>>>> I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution.
>>>>> 
>>>>> The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).
>>>>> 
>>>>> Are you initializing the Jacobian to 0 first?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Matt
>>>>> 
>>>>> Is this the case? or is there some other responsibility of said function?
>>>>> 
>>>>> -Andrew
>>>>> 
>>>>>> Ah ha!
>>>>>> 
>>>>>> The function passed to TSSetRHSJacobian needs to zero the solution vector?
>>>>>> 
>>>>>> As a point, this isn?t mentioned in any documentation that I can find.
>>>>>> 
>>>>>> -Andrew
>>>>> 
>>>>> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:
>>>>> This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.
>>>>> 
>>>>> Matt
>>>>> 
>>>>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>>>>> I have a fairly simple problem that I?m trying to timestep:
>>>>> 
>>>>> u? = A(t) u
>>>>> 
>>>>> I?m using the crank-nicholson method, which I understand (for this problem) to be:
>>>>> 
>>>>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>>>> or
>>>>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>>>> 
>>>>> When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up.
>>>>> 
>>>>> It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.
>>>>> 
>>>>> I?m not sure what I?m doing wrong.
>>>>> 
>>>>> PETSc code is taken out of the manual and is pretty simple:
>>>>> 
>>>>> TSCreate( comm, &ts );
>>>>> TSSetProblemType( ts, TS_LINEAR);
>>>>> TSSetType( ts, TSCN );
>>>>> TSSetInitialTimeStep( ts, 0, 0.01 );
>>>>> TSSetDuration( ts, 5, 0.03 );
>>>>> TSSetFromOptions( ts );
>>>>> TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>>>> TSSetRHSJacobian( ts, A, A, func, &cntx );
>>>>> TSSolve( ts, psi0 );
>>>>> 
>>>>> `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.
>>>>> 
>>>>> Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?).
>>>>> 
>>>>> -Andrew
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 


From ansp6066 at colorado.edu  Sat Mar 21 11:26:35 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Sat, 21 Mar 2015 09:26:35 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <550D8101.6070006@mcs.anl.gov>
References: <550D8101.6070006@mcs.anl.gov>
Message-ID: <1426955194929.0192b66f@Nodemailer>

>I haven?t been able to compile and run.




You might need -std=c++11 or -std=c++1y. ?The code uses a lambda. ?If you want, I can refactor that away.?




>The problem appears to be very stiff.





So, one of the things that is weird is that when I explicitly calculate the first few steps using the crank-nicholson method, I don?t have the exponential blowup. ?However, when I use the TimeStepper, I DO get an exponential blowup.




Since, theoretically, the two methods should be equivalent and I?m using the same recurrence relationship for the Crank Nicholson method that you are, where does the difference come from?

On Sat, Mar 21, 2015 at 8:32 AM, Emil Constantinescu <emconsta at mcs.anl.gov>
wrote:

> I haven't been able to compile and run. But here are a few quick notes.
> The problem appears to be very stiff.
> Theta and theta_endpoint are defining different methods:
> 1) -ts_type beuler OR -ts_theta_theta 1.0: is backward Euler
> u(t + h) = u(t) + h*A(t+h)*u(t+h)
> 2) -ts_theta_theta 0.5: is the implicit midpoint rule
> u(t + h) = u(t) + h*[A(t+h/2)*(u(t+h)+u(t))/2]
> 3) -ts_type cn OR -ts_theta_theta 0.5 -ts_theta_endpoint: is 
> Crank-Nicholson/trapezoidal
> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
> Note that the last two are different. -ts_type theta -ts_theta_theta .5 
> is different from -ts_type cn. They the same linear stability properties 
> if A(t)=A; but not if A depends on t.
> When -ts_theta_adapt is used, then it detects the instability as an 
> error and reduces the step by a lot! wlte=1.24e+03 which means that the 
> reduction should be severe but the controller tries 0.1*dt and that 
> seems to pass but it "jig-saws" (take a look at the next attempted 
> step), which means that it is likely unstable.
> I'll try to build the example to get more insight.
> Emil
> On 3/20/15 10:57 PM, Barry Smith wrote:
>>
>>    Andrew,
>>
>>     I'm afraid Emil will have to take a look at this and explain it.  The -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but the -ts_type cn is not stable. It turns out that -ts_type cn is equivalent to -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint and somehow this endpoint business (which I don't understand) is causing a problem. Meanwhile if I add -ts_theta_adapt to the endpoint one it becomes stable ? Anyways all cases are displayed below.
>>
>>    Emil,
>>
>>      What's up with this? Does the endpoint business have a bug or can it not be used for this problem (the matrix A is a function of t.)
>>
>>    Barry
>>
>>
>> $ ./ex2  -ts_type cn
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 1
>> t: 0.03 step: 3 norm-1: 3
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 1
>> t: 0.03 step: 3 norm-1: 3
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor
>> 0 TS dt 0.01 time 0
>> t: 0 step: 0 norm-1: 0
>> 0 TS dt 0.01 time 0
>> 1 TS dt 0.1 time 0.01
>> t: 0.01 step: 1 norm-1: 0
>> 1 TS dt 0.1 time 0.01
>> 2 TS dt 0.1 time 0.02
>> t: 0.02 step: 2 norm-1: 0
>> 2 TS dt 0.1 time 0.02
>> 3 TS dt 0.1 time 0.03
>> t: 0.03 step: 3 norm-1: 0
>> 3 TS dt 0.1 time 0.03
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor -ts_adapt_monitor
>> 0 TS dt 0.01 time 0
>> t: 0 step: 0 norm-1: 0
>> 0 TS dt 0.01 time 0
>>        TSAdapt 'basic': step   0 accepted t=0          + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 1 TS dt 0.1 time 0.01
>> t: 0.01 step: 1 norm-1: 0
>> 1 TS dt 0.1 time 0.01
>>        TSAdapt 'basic': step   1 rejected t=0.01       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>>        TSAdapt 'basic': step   1 accepted t=0.01       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 2 TS dt 0.1 time 0.02
>> t: 0.02 step: 2 norm-1: 0
>> 2 TS dt 0.1 time 0.02
>>        TSAdapt 'basic': step   2 rejected t=0.02       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02
>>        TSAdapt 'basic': step   2 accepted t=0.02       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01
>> 3 TS dt 0.1 time 0.03
>> t: 0.03 step: 3 norm-1: 0
>> 3 TS dt 0.1 time 0.03
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type beuler
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> $ ./ex2  -ts_type euler
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0
>> t: 0.03 step: 3 norm-1: 0
>> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>
>>
>>> On Mar 20, 2015, at 10:18 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>
>>> here are the data files.
>>>
>>> dipole_matrix.dat:
>>> https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>>>
>>> energy_eigenvalues_vector.dat
>>> https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>>>
>>> -Andrew
>>>
>>>
>>>
>>> On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>> Data files are needed
>>>
>>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>>> VecLoad( H0, view );
>>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
>>>
>>> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded.
>>>
>>> Barry
>>>
>>>
>>>> On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>
>>>> Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?)
>>>>
>>>> https://gist.github.com/spott/aea8070f35e79e7249e6
>>>>
>>>> The first section does it using the time stepper. The second section does it by explicitly doing the steps. The output is:
>>>>
>>>> //first section, using TimeStepper:
>>>> t: 0 step: 0 norm-1: 0
>>>> t: 0.01 step: 1 norm-1: 0
>>>> t: 0.02 step: 2 norm-1: 0.999995
>>>> t: 0.03 step: 3 norm-1: 2.99998
>>>>
>>>> //Second section, using explicit code.
>>>> t: 0.01 norm-1: 0
>>>> t: 0.02 norm-1: 0
>>>> t: 0.02 norm-1: 2.22045e-16
>>>>
>>>>
>>>>
>>>> On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>
>>>> Andrew,
>>>>
>>>> Send your entire code. It will be easier and faster than talking past each other.
>>>>
>>>> Barry
>>>>
>>>>> On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>>
>>>>> I?m sorry, I?m not trying to be difficult, but I?m not following.
>>>>>
>>>>> The manual states (for my special case):
>>>>> ? u ? = A(t)u. Use
>>>>>
>>>>> TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>>>>>
>>>>> where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ...
>>>>> My `func` does this. It is 7 lines:
>>>>>
>>>>> context* c = static_cast<context*>( G_u );
>>>>> PetscScalar e = c->E( t_ );
>>>>> MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>>>>> MatShift( A, e );
>>>>> MatDiagonalSet( A, c->H0, INSERT_VALUES);
>>>>> MatShift( A, std::complex<double>( 0, -1 ) );
>>>>> return 0;
>>>>>
>>>>> SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case).
>>>>>
>>>>> -Andrew
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>
>>>>> On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>>>>> So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution.
>>>>>
>>>>> I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution.
>>>>>
>>>>> The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it).
>>>>>
>>>>> Are you initializing the Jacobian to 0 first?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>> Is this the case? or is there some other responsibility of said function?
>>>>>
>>>>> -Andrew
>>>>>
>>>>>> Ah ha!
>>>>>>
>>>>>> The function passed to TSSetRHSJacobian needs to zero the solution vector?
>>>>>>
>>>>>> As a point, this isn?t mentioned in any documentation that I can find.
>>>>>>
>>>>>> -Andrew
>>>>>
>>>>> On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote:
>>>>> This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code.
>>>>>
>>>>> Matt
>>>>>
>>>>> On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote:
>>>>> I have a fairly simple problem that I?m trying to timestep:
>>>>>
>>>>> u? = A(t) u
>>>>>
>>>>> I?m using the crank-nicholson method, which I understand (for this problem) to be:
>>>>>
>>>>> u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>>>> or
>>>>> [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>>>>
>>>>> When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up.
>>>>>
>>>>> It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1.
>>>>>
>>>>> I?m not sure what I?m doing wrong.
>>>>>
>>>>> PETSc code is taken out of the manual and is pretty simple:
>>>>>
>>>>> TSCreate( comm, &ts );
>>>>> TSSetProblemType( ts, TS_LINEAR);
>>>>> TSSetType( ts, TSCN );
>>>>> TSSetInitialTimeStep( ts, 0, 0.01 );
>>>>> TSSetDuration( ts, 5, 0.03 );
>>>>> TSSetFromOptions( ts );
>>>>> TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>>>> TSSetRHSJacobian( ts, A, A, func, &cntx );
>>>>> TSSolve( ts, psi0 );
>>>>>
>>>>> `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps.
>>>>>
>>>>> Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?).
>>>>>
>>>>> -Andrew
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150321/39111395/attachment.html>

From may at bu.edu  Sat Mar 21 11:55:07 2015
From: may at bu.edu (Young, Matthew, Adam)
Date: Sat, 21 Mar 2015 16:55:07 +0000
Subject: [petsc-users] PETSc within a larger simulation
In-Reply-To: <B677DA97-8451-437A-A8A3-F2A20A5BF433@mcs.anl.gov>
References: <17A35C213185A84BB8ED54C88FBFD712B4CA356A@IST-EX10MBX-4.ad.bu.edu>,
	<B677DA97-8451-437A-A8A3-F2A20A5BF433@mcs.anl.gov>
Message-ID: <17A35C213185A84BB8ED54C88FBFD712B4CA44B1@IST-EX10MBX-4.ad.bu.edu>

Barry, 

The PIC and fluid codes are both running on MPI_COMM_WORLD, so it looks like we're doing the initialization correctly. Since my first email, I've identified some other parts in my PETSc solver that could be causing my communicator problems. I'll work on fixing those and come back with specific questions if I have them.

Thanks!
--------------------------------------------------------------
Matthew Young
Graduate Student
Boston University Dept. of Astronomy
--------------------------------------------------------------


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Friday, March 20, 2015 2:11 PM
To: Young, Matthew, Adam
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] PETSc within a larger simulation

> On Mar 20, 2015, at 1:02 PM, Young, Matthew, Adam <may at bu.edu> wrote:
>
> I have a PETSc routine that I am trying to use as an electric-field solver within a particle-in-cell code. The existing PIC code is parallelized with MPI. The pure PIC code (written many years ago by my advisor) is well-tested and I am trying to add an option for it to operate as a hybrid fluid/PIC code. This requires solving Ax=b at every time step but I don't think I understand how to manage communication between the existing code and my PETSc solver. Since we may not always run the hybrid version, the code calls MPI_Init() before PetscInitialize(). Section 14.1 of the manual suggests that I use PetscSetCommWorld between MPI_Init() and PetscInitialize(), but that routine appears to not be supported anymore. Is that correct? Can you point me to examples of simulations that use PETSc within a larger code?

   Matt,

    In the standard case where the PIC and fluid code are both running on the same communicator (MPI_COMM_WORLD) you don't need to do anything special, just call the MPI_Init() and then the PetscInitialize(). If the PETSc code runs on a smaller communicator then BEFORE the call to PetscInitialize() you just need to set PETSC_COMM_WORLD = comm (where comm is that smaller communicator). (This replaces the use of the function call PetscSetCommWorld() that once existed.)

  Barry

>
> --Matt
>
> --------------------------------------------------------------
> Matthew Young
> Graduate Student
> Boston University Dept. of Astronomy
> --------------------------------------------------------------


From ansp6066 at colorado.edu  Sun Mar 22 15:29:53 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Sun, 22 Mar 2015 13:29:53 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
References: <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
Message-ID: <1427056193205.f585a3ea@Nodemailer>

So, I?m now even more confused.




I?m attempting to solve an equation that looks like this:




u? = -i(H0 + e(t) D) u




Where H0 is a purely real diagonal matrix, D is an off-diagonal block matrix, and e(t) is a function of time (The schr?dinger equation in the energy basis).




I?ve rewritten the e(t) function in my code to just return 0.0. ?So the new equation is just u? = -iH0 u. ?The matrix is time independent and diagonal (I?ve checked this). ?H0[0] ~= -.5 (with no imaginary component). ?and u(t=0) = [1,0,0,0,..]




This problem SHOULD be incredibly simple: u? = i (0.5) u.




However, I?m still getting the same blowup with the TS.:




//with e(t) == 0



//TS

t: 0 step: 0 norm-1: 0

t: 0.01 step: 1 norm-1: 0

t: 0.02 step: 2 norm-1: 0.9999953125635765

t: 0.03 step: 3 norm-1: 2.999981250276277

//Hand rolled

t: 0.01 norm-1: 0 ef 0

t: 0.02 norm-1: 0 ef 0

t: 0.03 norm-1: -1.110223024625157e-16 ef 0



??????????????????????????????

//with e(t) != 0

//TS


t: 0 step: 0 norm-1: 0

t: 0.01 step: 1 norm-1: 0

t: 0.02 step: 2 norm-1: 0.9999953125635765

t: 0.03 step: 3 norm-1: 2.999981250276277

//Hand rolled

t: 0.01 norm-1: 0 ef 9.474814647559372e-11

t: 0.02 norm-1: 0 ef 7.57983838406065e-10

t: 0.03 norm-1: -1.110223024625157e-16 ef 2.558187954267552e-09






I?ve updated the gist.




-Andrew

On Fri, Mar 20, 2015 at 9:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>  
>   Andrew,
>    I'm afraid Emil will have to take a look at this and explain it.  The -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but the -ts_type cn is not stable. It turns out that -ts_type cn is equivalent to -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint and somehow this endpoint business (which I don't understand) is causing a problem. Meanwhile if I add -ts_theta_adapt to the endpoint one it becomes stable ? Anyways all cases are displayed below.
>   Emil,
>     What's up with this? Does the endpoint business have a bug or can it not be used for this problem (the matrix A is a function of t.)
>   Barry
> $ ./ex2  -ts_type cn
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 1
> t: 0.03 step: 3 norm-1: 3
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 1
> t: 0.03 step: 3 norm-1: 3
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor
> 0 TS dt 0.01 time 0 
> t: 0 step: 0 norm-1: 0
> 0 TS dt 0.01 time 0 
> 1 TS dt 0.1 time 0.01 
> t: 0.01 step: 1 norm-1: 0
> 1 TS dt 0.1 time 0.01 
> 2 TS dt 0.1 time 0.02 
> t: 0.02 step: 2 norm-1: 0
> 2 TS dt 0.1 time 0.02 
> 3 TS dt 0.1 time 0.03 
> t: 0.03 step: 3 norm-1: 0
> 3 TS dt 0.1 time 0.03 
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type theta -ts_theta_theta .5  -ts_theta_endpoint -ts_theta_adapt -ts_monitor -ts_adapt_monitor
> 0 TS dt 0.01 time 0 
> t: 0 step: 0 norm-1: 0
> 0 TS dt 0.01 time 0 
>       TSAdapt 'basic': step   0 accepted t=0          + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
> 1 TS dt 0.1 time 0.01 
> t: 0.01 step: 1 norm-1: 0
> 1 TS dt 0.1 time 0.01 
>       TSAdapt 'basic': step   1 rejected t=0.01       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02 
>       TSAdapt 'basic': step   1 accepted t=0.01       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
> 2 TS dt 0.1 time 0.02 
> t: 0.02 step: 2 norm-1: 0
> 2 TS dt 0.1 time 0.02 
>       TSAdapt 'basic': step   2 rejected t=0.02       + 1.000e-01 wlte=1.24e+03 family='theta' scheme=0:'(null)' dt=1.000e-02 
>       TSAdapt 'basic': step   2 accepted t=0.02       + 1.000e-02 wlte=    0 family='theta' scheme=0:'(null)' dt=1.000e-01 
> 3 TS dt 0.1 time 0.03 
> t: 0.03 step: 3 norm-1: 0
> 3 TS dt 0.1 time 0.03 
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type beuler
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
> $ ./ex2  -ts_type euler
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0
> t: 0.03 step: 3 norm-1: 0
> ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>> On Mar 20, 2015, at 10:18 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>> 
>> here are the data files.
>> 
>> dipole_matrix.dat:
>> https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>> 
>> energy_eigenvalues_vector.dat
>> https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>> 
>> -Andrew
>> 
>> 
>> 
>> On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> Data files are needed 
>> 
>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view ); 
>> VecLoad( H0, view ); 
>> PetscViewerBinaryOpen( PETSC_COMM_WORLD, "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view ); 
>> 
>> BTW: You do not need to call Mat/VecAssembly on Mats and Vecs after they have been loaded. 
>> 
>> Barry 
>> 
>> 
>> > On Mar 20, 2015, at 6:39 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > 
>> > Sorry it took so long, I wanted to create a ?reduced? case (without all my parameter handling and other stuff?) 
>> > 
>> > https://gist.github.com/spott/aea8070f35e79e7249e6 
>> > 
>> > The first section does it using the time stepper. The second section does it by explicitly doing the steps. The output is: 
>> > 
>> > //first section, using TimeStepper: 
>> > t: 0 step: 0 norm-1: 0 
>> > t: 0.01 step: 1 norm-1: 0 
>> > t: 0.02 step: 2 norm-1: 0.999995 
>> > t: 0.03 step: 3 norm-1: 2.99998 
>> > 
>> > //Second section, using explicit code. 
>> > t: 0.01 norm-1: 0 
>> > t: 0.02 norm-1: 0 
>> > t: 0.02 norm-1: 2.22045e-16 
>> > 
>> > 
>> > 
>> > On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote: 
>> > 
>> > Andrew, 
>> > 
>> > Send your entire code. It will be easier and faster than talking past each other. 
>> > 
>> > Barry 
>> > 
>> > > On Mar 20, 2015, at 5:00 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > > 
>> > > I?m sorry, I?m not trying to be difficult, but I?m not following. 
>> > > 
>> > > The manual states (for my special case): 
>> > > ? u ? = A(t)u. Use 
>> > > 
>> > > TSSetProblemType(ts,TS LINEAR); TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL); TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx); 
>> > > 
>> > > where YourComputeRHSJacobian() is a function you provide that computes A as a function of time. Or use ... 
>> > > My `func` does this. It is 7 lines: 
>> > > 
>> > > context* c = static_cast<context*>( G_u ); 
>> > > PetscScalar e = c->E( t_ ); 
>> > > MatCopy( c->D, A, SAME_NONZERO_PATTERN ); 
>> > > MatShift( A, e ); 
>> > > MatDiagonalSet( A, c->H0, INSERT_VALUES); 
>> > > MatShift( A, std::complex<double>( 0, -1 ) ); 
>> > > return 0; 
>> > > 
>> > > SHOULD `func` touch U? If so, what should `func` do to U? I thought that the RHSJacobian function was only meant to create A, since dG/du = A(t) (for this special case). 
>> > > 
>> > > -Andrew 
>> > > 
>> > > 
>> > > 
>> > > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley <knepley at gmail.com> wrote: 
>> > > 
>> > > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > > So, it doesn?t seem that zeroing the given vector in the function passed to TSSetRHSJacobian is the problem. When I do that, it just zeros out the solution. 
>> > > 
>> > > I would think you would zero the residual vector (if you add to it to construct the residual, as in FEM methods), not the solution. 
>> > > 
>> > > The function that is passed to TSSetRHSJacobian has only one responsibility ? to create the jacobian ? correct? In my case this is A(t). The solution vector is given for when you are solving nonlinear problems (A(t) also depends on U(t)). In my case, I don?t even look at the solution vector (because my A(t) doesn?t depend on it). 
>> > > 
>> > > Are you initializing the Jacobian to 0 first? 
>> > > 
>> > > Thanks, 
>> > > 
>> > > Matt 
>> > > 
>> > > Is this the case? or is there some other responsibility of said function? 
>> > > 
>> > > -Andrew 
>> > > 
>> > > >Ah ha! 
>> > > > 
>> > > >The function passed to TSSetRHSJacobian needs to zero the solution vector? 
>> > > > 
>> > > >As a point, this isn?t mentioned in any documentation that I can find. 
>> > > > 
>> > > >-Andrew 
>> > > 
>> > > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com>, wrote: 
>> > > This sounds like a problem in your calculation function where a Vec or Mat does not get reset to 0, but it does in your by hand code. 
>> > > 
>> > > Matt 
>> > > 
>> > > On Mar 20, 2015 2:52 PM, "Andrew Spott" <ansp6066 at colorado.edu> wrote: 
>> > > I have a fairly simple problem that I?m trying to timestep: 
>> > > 
>> > > u? = A(t) u 
>> > > 
>> > > I?m using the crank-nicholson method, which I understand (for this problem) to be: 
>> > > 
>> > > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)] 
>> > > or 
>> > > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t) 
>> > > 
>> > > When I attempt to timestep using PETSc, the norm of `u` blows up. When I do it directly (using the above), the norm of `u` doesn?t blow up. 
>> > > 
>> > > It is important to note that the solution generated after the first step is identical for both, but the second step for Petsc has a norm of ~2, while for the directly calculated version it is ~1. The third step for petsc has a norm of ~4, while the directly calculated version it is still ~1. 
>> > > 
>> > > I?m not sure what I?m doing wrong. 
>> > > 
>> > > PETSc code is taken out of the manual and is pretty simple: 
>> > > 
>> > > TSCreate( comm, &ts ); 
>> > > TSSetProblemType( ts, TS_LINEAR); 
>> > > TSSetType( ts, TSCN ); 
>> > > TSSetInitialTimeStep( ts, 0, 0.01 ); 
>> > > TSSetDuration( ts, 5, 0.03 ); 
>> > > TSSetFromOptions( ts ); 
>> > > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL ); 
>> > > TSSetRHSJacobian( ts, A, A, func, &cntx ); 
>> > > TSSolve( ts, psi0 ); 
>> > > 
>> > > `func` just constructs A(t) at the time given. The same code for calculating A(t) is used in both calculations, along with the same initial vector psi0, and the same time steps. 
>> > > 
>> > > Let me know what other information is needed. I?m not sure what could be the problem. `func` doesn?t touch U at all (should it?). 
>> > > 
>> > > -Andrew 
>> > > 
>> > > 
>> > > 
>> > > 
>> > > -- 
>> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> > > -- Norbert Wiener 
>> > > 
>> > 
>> > 
>> > 
>> 
>> 
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150322/99438c52/attachment.html>

From stm8086 at yahoo.com  Sun Mar 22 17:50:20 2015
From: stm8086 at yahoo.com (Steena M)
Date: Sun, 22 Mar 2015 15:50:20 -0700
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
Message-ID: <1427064620.9164.YahooMailBasic@web125404.mail.ne1.yahoo.com>

Hello, 

I need to distribute a sparse matrix such that each proc owns an unequal number of blocked rows before I proceed with MPI MatMult. My initial thoughts on doing this:

1) Use  MatGetSubMatrices() on the test MATMPIBAIJ matrix to produce a new matrix where each proc has an unequal number of rows.

2) Provide scatter context for vector X (for MatMult )using IS iscol from MatGetSubMatrices() while creating the vector X.

3) Call MatMult()

 Will MatMult_MPIBAIJ continue to scatter this matrix and vector such that each proc will own an equal number of matrix rows and corresponding diagonal vector elements? Should I write my own MPIMatMult function to retain my redistribution of the matrix and vector?

Thanks in advance,
Steena

From bsmith at mcs.anl.gov  Sun Mar 22 17:58:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 22 Mar 2015 17:58:40 -0500
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <1427064620.9164.YahooMailBasic@web125404.mail.ne1.yahoo.com>
References: <1427064620.9164.YahooMailBasic@web125404.mail.ne1.yahoo.com>
Message-ID: <9B95D6AB-DCFC-483B-84D3-4C04C1021DFE@mcs.anl.gov>


  Steena,

   I am a little unsure of your question.  

   1) you can create a MPIBAIJ matrix with any distribution of block rows per process you want, just set the local row size for each process to be what you like.  Use MatCreateVecs() to get correspondingly laid out vectors.

   or 2) if you have a MPIBAIJ matrix with "equal" row layout and you want a new one with uneven row layout you can simply use MatGetSubMatrix() to create that new matrix.

  Barry

Unless you have another reason to have the matrix with an equal number row layout I would just generate the matrix with the layout you want.


> On Mar 22, 2015, at 5:50 PM, Steena M <stm8086 at yahoo.com> wrote:
> 
> Hello, 
> 
> I need to distribute a sparse matrix such that each proc owns an unequal number of blocked rows before I proceed with MPI MatMult. My initial thoughts on doing this:
> 
> 1) Use  MatGetSubMatrices() on the test MATMPIBAIJ matrix to produce a new matrix where each proc has an unequal number of rows.
> 
> 2) Provide scatter context for vector X (for MatMult )using IS iscol from MatGetSubMatrices() while creating the vector X.
> 
> 3) Call MatMult()
> 
> Will MatMult_MPIBAIJ continue to scatter this matrix and vector such that each proc will own an equal number of matrix rows and corresponding diagonal vector elements? Should I write my own MPIMatMult function to retain my redistribution of the matrix and vector?
> 
> Thanks in advance,
> Steena


From emconsta at mcs.anl.gov  Sun Mar 22 21:21:04 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Sun, 22 Mar 2015 21:21:04 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1427056193205.f585a3ea@Nodemailer>
References: <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<1427056193205.f585a3ea@Nodemailer>
Message-ID: <550F7890.4020904@mcs.anl.gov>

Hi Andrew,

I can reproduce this issue and I agree that there is something wrong. 
I'll look into it.

Emil


On 3/22/15 3:29 PM, Andrew Spott wrote:
> So, I?m now even more confused.
>
> I?m attempting to solve an equation that looks like this:
>
> u? = -i(H0 + e(t) D) u
>
> Where H0 is a purely real diagonal matrix, D is an off-diagonal block
> matrix, and e(t) is a function of time (The schr?dinger equation in the
> energy basis).
>
> I?ve rewritten the e(t) function in my code to just return 0.0.  So the
> new equation is just u? = -iH0 u.  The matrix is time independent and
> diagonal (I?ve checked this).  H0[0] ~= -.5 (with no imaginary
> component).  and u(t=0) = [1,0,0,0,..]
>
> This problem SHOULD be incredibly simple: u? = i (0.5) u.
>
> However, I?m still getting the same blowup with the TS.:
>
> //with e(t) == 0
> //TS
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0.9999953125635765
> t: 0.03 step: 3 norm-1: 2.999981250276277
> //Hand rolled
> t: 0.01 norm-1: 0 ef 0
> t: 0.02 norm-1: 0 ef 0
> t: 0.03 norm-1: -1.110223024625157e-16 ef 0
> ??????????????????????????????
> //with e(t) != 0
> //TS
> t: 0 step: 0 norm-1: 0
> t: 0.01 step: 1 norm-1: 0
> t: 0.02 step: 2 norm-1: 0.9999953125635765
> t: 0.03 step: 3 norm-1: 2.999981250276277
> //Hand rolled
> t: 0.01 norm-1: 0 ef 9.474814647559372e-11
> t: 0.02 norm-1: 0 ef 7.57983838406065e-10
> t: 0.03 norm-1: -1.110223024625157e-16 ef 2.558187954267552e-09
>
> I?ve updated the gist.
>
> -Andrew
>
>
>
> On Fri, Mar 20, 2015 at 9:57 PM, Barry Smith <bsmith at mcs.anl.gov
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>     Andrew,
>
>     I'm afraid Emil will have to take a look at this and explain it. The
>     -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but
>     the -ts_type cn is not stable. It turns out that -ts_type cn is
>     equivalent to -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>     and somehow this endpoint business (which I don't understand) is
>     causing a problem. Meanwhile if I add -ts_theta_adapt to the
>     endpoint one it becomes stable ? Anyways all cases are displayed below.
>
>     Emil,
>
>     What's up with this? Does the endpoint business have a bug or can it
>     not be used for this problem (the matrix A is a function of t.)
>
>     Barry
>
>
>     $ ./ex2 -ts_type cn
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 1
>     t: 0.03 step: 3 norm-1: 3
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 0
>     t: 0.03 step: 3 norm-1: 0
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta -ts_theta_theta .5
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 0
>     t: 0.03 step: 3 norm-1: 0
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 1
>     t: 0.03 step: 3 norm-1: 3
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>     -ts_theta_adapt
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 0
>     t: 0.03 step: 3 norm-1: 0
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>     -ts_theta_adapt -ts_monitor
>     0 TS dt 0.01 time 0
>     t: 0 step: 0 norm-1: 0
>     0 TS dt 0.01 time 0
>     1 TS dt 0.1 time 0.01
>     t: 0.01 step: 1 norm-1: 0
>     1 TS dt 0.1 time 0.01
>     2 TS dt 0.1 time 0.02
>     t: 0.02 step: 2 norm-1: 0
>     2 TS dt 0.1 time 0.02
>     3 TS dt 0.1 time 0.03
>     t: 0.03 step: 3 norm-1: 0
>     3 TS dt 0.1 time 0.03
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>     -ts_theta_adapt -ts_monitor -ts_adapt_monitor
>     0 TS dt 0.01 time 0
>     t: 0 step: 0 norm-1: 0
>     0 TS dt 0.01 time 0
>     TSAdapt 'basic': step 0 accepted t=0 + 1.000e-02 wlte= 0
>     family='theta' scheme=0:'(null)' dt=1.000e-01
>     1 TS dt 0.1 time 0.01
>     t: 0.01 step: 1 norm-1: 0
>     1 TS dt 0.1 time 0.01
>     TSAdapt 'basic': step 1 rejected t=0.01 + 1.000e-01 wlte=1.24e+03
>     family='theta' scheme=0:'(null)' dt=1.000e-02
>     TSAdapt 'basic': step 1 accepted t=0.01 + 1.000e-02 wlte= 0
>     family='theta' scheme=0:'(null)' dt=1.000e-01
>     2 TS dt 0.1 time 0.02
>     t: 0.02 step: 2 norm-1: 0
>     2 TS dt 0.1 time 0.02
>     TSAdapt 'basic': step 2 rejected t=0.02 + 1.000e-01 wlte=1.24e+03
>     family='theta' scheme=0:'(null)' dt=1.000e-02
>     TSAdapt 'basic': step 2 accepted t=0.02 + 1.000e-02 wlte= 0
>     family='theta' scheme=0:'(null)' dt=1.000e-01
>     3 TS dt 0.1 time 0.03
>     t: 0.03 step: 3 norm-1: 0
>     3 TS dt 0.1 time 0.03
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type beuler
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 0
>     t: 0.03 step: 3 norm-1: 0
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>     $ ./ex2 -ts_type euler
>     t: 0 step: 0 norm-1: 0
>     t: 0.01 step: 1 norm-1: 0
>     t: 0.02 step: 2 norm-1: 0
>     t: 0.03 step: 3 norm-1: 0
>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>
>
>      > On Mar 20, 2015, at 10:18 PM, Andrew Spott
>     <ansp6066 at colorado.edu> wrote:
>      >
>      > here are the data files.
>      >
>      > dipole_matrix.dat:
>      > https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>      >
>      > energy_eigenvalues_vector.dat
>      >
>     https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>
>      >
>      > -Andrew
>      >
>      >
>      >
>      > On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov>
>     wrote:
>      >
>      > Data files are needed
>      >
>      > PetscViewerBinaryOpen( PETSC_COMM_WORLD,
>     "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>      > VecLoad( H0, view );
>      > PetscViewerBinaryOpen( PETSC_COMM_WORLD,
>     "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
>      >
>      > BTW: You do not need to call Mat/VecAssembly on Mats and Vecs
>     after they have been loaded.
>      >
>      > Barry
>      >
>      >
>      > > On Mar 20, 2015, at 6:39 PM, Andrew Spott
>     <ansp6066 at colorado.edu> wrote:
>      > >
>      > > Sorry it took so long, I wanted to create a ?reduced? case
>     (without all my parameter handling and other stuff?)
>      > >
>      > > https://gist.github.com/spott/aea8070f35e79e7249e6
>      > >
>      > > The first section does it using the time stepper. The second
>     section does it by explicitly doing the steps. The output is:
>      > >
>      > > //first section, using TimeStepper:
>      > > t: 0 step: 0 norm-1: 0
>      > > t: 0.01 step: 1 norm-1: 0
>      > > t: 0.02 step: 2 norm-1: 0.999995
>      > > t: 0.03 step: 3 norm-1: 2.99998
>      > >
>      > > //Second section, using explicit code.
>      > > t: 0.01 norm-1: 0
>      > > t: 0.02 norm-1: 0
>      > > t: 0.02 norm-1: 2.22045e-16
>      > >
>      > >
>      > >
>      > > On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith
>     <bsmith at mcs.anl.gov> wrote:
>      > >
>      > > Andrew,
>      > >
>      > > Send your entire code. It will be easier and faster than
>     talking past each other.
>      > >
>      > > Barry
>      > >
>      > > > On Mar 20, 2015, at 5:00 PM, Andrew Spott
>     <ansp6066 at colorado.edu> wrote:
>      > > >
>      > > > I?m sorry, I?m not trying to be difficult, but I?m not
>     following.
>      > > >
>      > > > The manual states (for my special case):
>      > > > ? u ? = A(t)u. Use
>      > > >
>      > > > TSSetProblemType(ts,TS LINEAR);
>     TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
>     TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>      > > >
>      > > > where YourComputeRHSJacobian() is a function you provide that
>     computes A as a function of time. Or use ...
>      > > > My `func` does this. It is 7 lines:
>      > > >
>      > > > context* c = static_cast<context*>( G_u );
>      > > > PetscScalar e = c->E( t_ );
>      > > > MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>      > > > MatShift( A, e );
>      > > > MatDiagonalSet( A, c->H0, INSERT_VALUES);
>      > > > MatShift( A, std::complex<double>( 0, -1 ) );
>      > > > return 0;
>      > > >
>      > > > SHOULD `func` touch U? If so, what should `func` do to U? I
>     thought that the RHSJacobian function was only meant to create A,
>     since dG/du = A(t) (for this special case).
>      > > >
>      > > > -Andrew
>      > > >
>      > > >
>      > > >
>      > > > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley
>     <knepley at gmail.com> wrote:
>      > > >
>      > > > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott
>     <ansp6066 at colorado.edu> wrote:
>      > > > So, it doesn?t seem that zeroing the given vector in the
>     function passed to TSSetRHSJacobian is the problem. When I do that,
>     it just zeros out the solution.
>      > > >
>      > > > I would think you would zero the residual vector (if you add
>     to it to construct the residual, as in FEM methods), not the solution.
>      > > >
>      > > > The function that is passed to TSSetRHSJacobian has only one
>     responsibility ? to create the jacobian ? correct? In my case this
>     is A(t). The solution vector is given for when you are solving
>     nonlinear problems (A(t) also depends on U(t)). In my case, I don?t
>     even look at the solution vector (because my A(t) doesn?t depend on
>     it).
>      > > >
>      > > > Are you initializing the Jacobian to 0 first?
>      > > >
>      > > > Thanks,
>      > > >
>      > > > Matt
>      > > >
>      > > > Is this the case? or is there some other responsibility of
>     said function?
>      > > >
>      > > > -Andrew
>      > > >
>      > > > >Ah ha!
>      > > > >
>      > > > >The function passed to TSSetRHSJacobian needs to zero the
>     solution vector?
>      > > > >
>      > > > >As a point, this isn?t mentioned in any documentation that I
>     can find.
>      > > > >
>      > > > >-Andrew
>      > > >
>      > > > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley
>     <knepley at gmail.com>, wrote:
>      > > > This sounds like a problem in your calculation function where
>     a Vec or Mat does not get reset to 0, but it does in your by hand code.
>      > > >
>      > > > Matt
>      > > >
>      > > > On Mar 20, 2015 2:52 PM, "Andrew Spott"
>     <ansp6066 at colorado.edu> wrote:
>      > > > I have a fairly simple problem that I?m trying to timestep:
>      > > >
>      > > > u? = A(t) u
>      > > >
>      > > > I?m using the crank-nicholson method, which I understand (for
>     this problem) to be:
>      > > >
>      > > > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>      > > > or
>      > > > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>      > > >
>      > > > When I attempt to timestep using PETSc, the norm of `u` blows
>     up. When I do it directly (using the above), the norm of `u` doesn?t
>     blow up.
>      > > >
>      > > > It is important to note that the solution generated after the
>     first step is identical for both, but the second step for Petsc has
>     a norm of ~2, while for the directly calculated version it is ~1.
>     The third step for petsc has a norm of ~4, while the directly
>     calculated version it is still ~1.
>      > > >
>      > > > I?m not sure what I?m doing wrong.
>      > > >
>      > > > PETSc code is taken out of the manual and is pretty simple:
>      > > >
>      > > > TSCreate( comm, &ts );
>      > > > TSSetProblemType( ts, TS_LINEAR);
>      > > > TSSetType( ts, TSCN );
>      > > > TSSetInitialTimeStep( ts, 0, 0.01 );
>      > > > TSSetDuration( ts, 5, 0.03 );
>      > > > TSSetFromOptions( ts );
>      > > > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>      > > > TSSetRHSJacobian( ts, A, A, func, &cntx );
>      > > > TSSolve( ts, psi0 );
>      > > >
>      > > > `func` just constructs A(t) at the time given. The same code
>     for calculating A(t) is used in both calculations, along with the
>     same initial vector psi0, and the same time steps.
>      > > >
>      > > > Let me know what other information is needed. I?m not sure
>     what could be the problem. `func` doesn?t touch U at all (should it?).
>      > > >
>      > > > -Andrew
>      > > >
>      > > >
>      > > >
>      > > >
>      > > > --
>      > > > What most experimenters take for granted before they begin
>     their experiments is infinitely more interesting than any results to
>     which their experiments lead.
>      > > > -- Norbert Wiener
>      > > >
>      > >
>      > >
>      > >
>      >
>      >
>      >
>
>

From ansp6066 at colorado.edu  Sun Mar 22 21:23:26 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Sun, 22 Mar 2015 19:23:26 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <550F7890.4020904@mcs.anl.gov>
References: <550F7890.4020904@mcs.anl.gov>
Message-ID: <1427077405806.9002f01c@Nodemailer>

Thanks! Let me know if I can do anything to help.




-Andrew



?
Andrew

On Sun, Mar 22, 2015 at 8:21 PM, Emil Constantinescu <emconsta at mcs.anl.gov>
wrote:

> Hi Andrew,
> I can reproduce this issue and I agree that there is something wrong. 
> I'll look into it.
> Emil
> On 3/22/15 3:29 PM, Andrew Spott wrote:
>> So, I?m now even more confused.
>>
>> I?m attempting to solve an equation that looks like this:
>>
>> u? = -i(H0 + e(t) D) u
>>
>> Where H0 is a purely real diagonal matrix, D is an off-diagonal block
>> matrix, and e(t) is a function of time (The schr?dinger equation in the
>> energy basis).
>>
>> I?ve rewritten the e(t) function in my code to just return 0.0.  So the
>> new equation is just u? = -iH0 u.  The matrix is time independent and
>> diagonal (I?ve checked this).  H0[0] ~= -.5 (with no imaginary
>> component).  and u(t=0) = [1,0,0,0,..]
>>
>> This problem SHOULD be incredibly simple: u? = i (0.5) u.
>>
>> However, I?m still getting the same blowup with the TS.:
>>
>> //with e(t) == 0
>> //TS
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0.9999953125635765
>> t: 0.03 step: 3 norm-1: 2.999981250276277
>> //Hand rolled
>> t: 0.01 norm-1: 0 ef 0
>> t: 0.02 norm-1: 0 ef 0
>> t: 0.03 norm-1: -1.110223024625157e-16 ef 0
>> ??????????????????????????????
>> //with e(t) != 0
>> //TS
>> t: 0 step: 0 norm-1: 0
>> t: 0.01 step: 1 norm-1: 0
>> t: 0.02 step: 2 norm-1: 0.9999953125635765
>> t: 0.03 step: 3 norm-1: 2.999981250276277
>> //Hand rolled
>> t: 0.01 norm-1: 0 ef 9.474814647559372e-11
>> t: 0.02 norm-1: 0 ef 7.57983838406065e-10
>> t: 0.03 norm-1: -1.110223024625157e-16 ef 2.558187954267552e-09
>>
>> I?ve updated the gist.
>>
>> -Andrew
>>
>>
>>
>> On Fri, Mar 20, 2015 at 9:57 PM, Barry Smith <bsmith at mcs.anl.gov
>> <mailto:bsmith at mcs.anl.gov>> wrote:
>>
>>
>>     Andrew,
>>
>>     I'm afraid Emil will have to take a look at this and explain it. The
>>     -ts_type beuler and -ts_type theta -ts_theta_theta .5 are stable but
>>     the -ts_type cn is not stable. It turns out that -ts_type cn is
>>     equivalent to -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>>     and somehow this endpoint business (which I don't understand) is
>>     causing a problem. Meanwhile if I add -ts_theta_adapt to the
>>     endpoint one it becomes stable ? Anyways all cases are displayed below.
>>
>>     Emil,
>>
>>     What's up with this? Does the endpoint business have a bug or can it
>>     not be used for this problem (the matrix A is a function of t.)
>>
>>     Barry
>>
>>
>>     $ ./ex2 -ts_type cn
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 1
>>     t: 0.03 step: 3 norm-1: 3
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 0
>>     t: 0.03 step: 3 norm-1: 0
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta -ts_theta_theta .5
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 0
>>     t: 0.03 step: 3 norm-1: 0
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 1
>>     t: 0.03 step: 3 norm-1: 3
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>>     -ts_theta_adapt
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 0
>>     t: 0.03 step: 3 norm-1: 0
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>>     -ts_theta_adapt -ts_monitor
>>     0 TS dt 0.01 time 0
>>     t: 0 step: 0 norm-1: 0
>>     0 TS dt 0.01 time 0
>>     1 TS dt 0.1 time 0.01
>>     t: 0.01 step: 1 norm-1: 0
>>     1 TS dt 0.1 time 0.01
>>     2 TS dt 0.1 time 0.02
>>     t: 0.02 step: 2 norm-1: 0
>>     2 TS dt 0.1 time 0.02
>>     3 TS dt 0.1 time 0.03
>>     t: 0.03 step: 3 norm-1: 0
>>     3 TS dt 0.1 time 0.03
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type theta -ts_theta_theta .5 -ts_theta_endpoint
>>     -ts_theta_adapt -ts_monitor -ts_adapt_monitor
>>     0 TS dt 0.01 time 0
>>     t: 0 step: 0 norm-1: 0
>>     0 TS dt 0.01 time 0
>>     TSAdapt 'basic': step 0 accepted t=0 + 1.000e-02 wlte= 0
>>     family='theta' scheme=0:'(null)' dt=1.000e-01
>>     1 TS dt 0.1 time 0.01
>>     t: 0.01 step: 1 norm-1: 0
>>     1 TS dt 0.1 time 0.01
>>     TSAdapt 'basic': step 1 rejected t=0.01 + 1.000e-01 wlte=1.24e+03
>>     family='theta' scheme=0:'(null)' dt=1.000e-02
>>     TSAdapt 'basic': step 1 accepted t=0.01 + 1.000e-02 wlte= 0
>>     family='theta' scheme=0:'(null)' dt=1.000e-01
>>     2 TS dt 0.1 time 0.02
>>     t: 0.02 step: 2 norm-1: 0
>>     2 TS dt 0.1 time 0.02
>>     TSAdapt 'basic': step 2 rejected t=0.02 + 1.000e-01 wlte=1.24e+03
>>     family='theta' scheme=0:'(null)' dt=1.000e-02
>>     TSAdapt 'basic': step 2 accepted t=0.02 + 1.000e-02 wlte= 0
>>     family='theta' scheme=0:'(null)' dt=1.000e-01
>>     3 TS dt 0.1 time 0.03
>>     t: 0.03 step: 3 norm-1: 0
>>     3 TS dt 0.1 time 0.03
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type beuler
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 0
>>     t: 0.03 step: 3 norm-1: 0
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>     $ ./ex2 -ts_type euler
>>     t: 0 step: 0 norm-1: 0
>>     t: 0.01 step: 1 norm-1: 0
>>     t: 0.02 step: 2 norm-1: 0
>>     t: 0.03 step: 3 norm-1: 0
>>     ~/Src/petsc/test-dir (barry/more-tchem-work *=) arch-debug
>>
>>
>>      > On Mar 20, 2015, at 10:18 PM, Andrew Spott
>>     <ansp6066 at colorado.edu> wrote:
>>      >
>>      > here are the data files.
>>      >
>>      > dipole_matrix.dat:
>>      > https://www.dropbox.com/s/2ahkljzt6oo9bdr/dipole_matrix.dat?dl=0
>>      >
>>      > energy_eigenvalues_vector.dat
>>      >
>>     https://www.dropbox.com/s/sb59q38vqvjoypk/energy_eigenvalues_vector.dat?dl=0
>>
>>      >
>>      > -Andrew
>>      >
>>      >
>>      >
>>      > On Fri, Mar 20, 2015 at 7:25 PM, Barry Smith <bsmith at mcs.anl.gov>
>>     wrote:
>>      >
>>      > Data files are needed
>>      >
>>      > PetscViewerBinaryOpen( PETSC_COMM_WORLD,
>>     "hamiltonian/energy_eigenvalues_vector.dat", FILE_MODE_READ, &view );
>>      > VecLoad( H0, view );
>>      > PetscViewerBinaryOpen( PETSC_COMM_WORLD,
>>     "hamiltonian/dipole_matrix.dat", FILE_MODE_READ, &view );
>>      >
>>      > BTW: You do not need to call Mat/VecAssembly on Mats and Vecs
>>     after they have been loaded.
>>      >
>>      > Barry
>>      >
>>      >
>>      > > On Mar 20, 2015, at 6:39 PM, Andrew Spott
>>     <ansp6066 at colorado.edu> wrote:
>>      > >
>>      > > Sorry it took so long, I wanted to create a ?reduced? case
>>     (without all my parameter handling and other stuff?)
>>      > >
>>      > > https://gist.github.com/spott/aea8070f35e79e7249e6
>>      > >
>>      > > The first section does it using the time stepper. The second
>>     section does it by explicitly doing the steps. The output is:
>>      > >
>>      > > //first section, using TimeStepper:
>>      > > t: 0 step: 0 norm-1: 0
>>      > > t: 0.01 step: 1 norm-1: 0
>>      > > t: 0.02 step: 2 norm-1: 0.999995
>>      > > t: 0.03 step: 3 norm-1: 2.99998
>>      > >
>>      > > //Second section, using explicit code.
>>      > > t: 0.01 norm-1: 0
>>      > > t: 0.02 norm-1: 0
>>      > > t: 0.02 norm-1: 2.22045e-16
>>      > >
>>      > >
>>      > >
>>      > > On Fri, Mar 20, 2015 at 4:45 PM, Barry Smith
>>     <bsmith at mcs.anl.gov> wrote:
>>      > >
>>      > > Andrew,
>>      > >
>>      > > Send your entire code. It will be easier and faster than
>>     talking past each other.
>>      > >
>>      > > Barry
>>      > >
>>      > > > On Mar 20, 2015, at 5:00 PM, Andrew Spott
>>     <ansp6066 at colorado.edu> wrote:
>>      > > >
>>      > > > I?m sorry, I?m not trying to be difficult, but I?m not
>>     following.
>>      > > >
>>      > > > The manual states (for my special case):
>>      > > > ? u ? = A(t)u. Use
>>      > > >
>>      > > > TSSetProblemType(ts,TS LINEAR);
>>     TSSetRHSFunction(ts,NULL,TSComputeRHSFunctionLinear,NULL);
>>     TSSetRHSJacobian(ts,A,A,YourComputeRHSJacobian,&appctx);
>>      > > >
>>      > > > where YourComputeRHSJacobian() is a function you provide that
>>     computes A as a function of time. Or use ...
>>      > > > My `func` does this. It is 7 lines:
>>      > > >
>>      > > > context* c = static_cast<context*>( G_u );
>>      > > > PetscScalar e = c->E( t_ );
>>      > > > MatCopy( c->D, A, SAME_NONZERO_PATTERN );
>>      > > > MatShift( A, e );
>>      > > > MatDiagonalSet( A, c->H0, INSERT_VALUES);
>>      > > > MatShift( A, std::complex<double>( 0, -1 ) );
>>      > > > return 0;
>>      > > >
>>      > > > SHOULD `func` touch U? If so, what should `func` do to U? I
>>     thought that the RHSJacobian function was only meant to create A,
>>     since dG/du = A(t) (for this special case).
>>      > > >
>>      > > > -Andrew
>>      > > >
>>      > > >
>>      > > >
>>      > > > On Fri, Mar 20, 2015 at 3:26 PM, Matthew Knepley
>>     <knepley at gmail.com> wrote:
>>      > > >
>>      > > > On Fri, Mar 20, 2015 at 3:09 PM, Andrew Spott
>>     <ansp6066 at colorado.edu> wrote:
>>      > > > So, it doesn?t seem that zeroing the given vector in the
>>     function passed to TSSetRHSJacobian is the problem. When I do that,
>>     it just zeros out the solution.
>>      > > >
>>      > > > I would think you would zero the residual vector (if you add
>>     to it to construct the residual, as in FEM methods), not the solution.
>>      > > >
>>      > > > The function that is passed to TSSetRHSJacobian has only one
>>     responsibility ? to create the jacobian ? correct? In my case this
>>     is A(t). The solution vector is given for when you are solving
>>     nonlinear problems (A(t) also depends on U(t)). In my case, I don?t
>>     even look at the solution vector (because my A(t) doesn?t depend on
>>     it).
>>      > > >
>>      > > > Are you initializing the Jacobian to 0 first?
>>      > > >
>>      > > > Thanks,
>>      > > >
>>      > > > Matt
>>      > > >
>>      > > > Is this the case? or is there some other responsibility of
>>     said function?
>>      > > >
>>      > > > -Andrew
>>      > > >
>>      > > > >Ah ha!
>>      > > > >
>>      > > > >The function passed to TSSetRHSJacobian needs to zero the
>>     solution vector?
>>      > > > >
>>      > > > >As a point, this isn?t mentioned in any documentation that I
>>     can find.
>>      > > > >
>>      > > > >-Andrew
>>      > > >
>>      > > > On Friday, Mar 20, 2015 at 2:17 PM, Matthew Knepley
>>     <knepley at gmail.com>, wrote:
>>      > > > This sounds like a problem in your calculation function where
>>     a Vec or Mat does not get reset to 0, but it does in your by hand code.
>>      > > >
>>      > > > Matt
>>      > > >
>>      > > > On Mar 20, 2015 2:52 PM, "Andrew Spott"
>>     <ansp6066 at colorado.edu> wrote:
>>      > > > I have a fairly simple problem that I?m trying to timestep:
>>      > > >
>>      > > > u? = A(t) u
>>      > > >
>>      > > > I?m using the crank-nicholson method, which I understand (for
>>     this problem) to be:
>>      > > >
>>      > > > u(t + h) = u(t) + h/2[A(t+h)*u(t+h) + A(t)*u(t)]
>>      > > > or
>>      > > > [1 - h/2 * A(t+1)] u(t+1) = [1 + h/2 * A(t)] u(t)
>>      > > >
>>      > > > When I attempt to timestep using PETSc, the norm of `u` blows
>>     up. When I do it directly (using the above), the norm of `u` doesn?t
>>     blow up.
>>      > > >
>>      > > > It is important to note that the solution generated after the
>>     first step is identical for both, but the second step for Petsc has
>>     a norm of ~2, while for the directly calculated version it is ~1.
>>     The third step for petsc has a norm of ~4, while the directly
>>     calculated version it is still ~1.
>>      > > >
>>      > > > I?m not sure what I?m doing wrong.
>>      > > >
>>      > > > PETSc code is taken out of the manual and is pretty simple:
>>      > > >
>>      > > > TSCreate( comm, &ts );
>>      > > > TSSetProblemType( ts, TS_LINEAR);
>>      > > > TSSetType( ts, TSCN );
>>      > > > TSSetInitialTimeStep( ts, 0, 0.01 );
>>      > > > TSSetDuration( ts, 5, 0.03 );
>>      > > > TSSetFromOptions( ts );
>>      > > > TSSetRHSFunction( ts, NULL, TSComputeRHSFunctionLinear, NULL );
>>      > > > TSSetRHSJacobian( ts, A, A, func, &cntx );
>>      > > > TSSolve( ts, psi0 );
>>      > > >
>>      > > > `func` just constructs A(t) at the time given. The same code
>>     for calculating A(t) is used in both calculations, along with the
>>     same initial vector psi0, and the same time steps.
>>      > > >
>>      > > > Let me know what other information is needed. I?m not sure
>>     what could be the problem. `func` doesn?t touch U at all (should it?).
>>      > > >
>>      > > > -Andrew
>>      > > >
>>      > > >
>>      > > >
>>      > > >
>>      > > > --
>>      > > > What most experimenters take for granted before they begin
>>     their experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>      > > > -- Norbert Wiener
>>      > > >
>>      > >
>>      > >
>>      > >
>>      >
>>      >
>>      >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150322/26419ca0/attachment-0001.html>

From dalcinl at gmail.com  Mon Mar 23 04:44:42 2015
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Mar 2015 12:44:42 +0300
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <550D8101.6070006@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
Message-ID: <CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>

On 21 March 2015 at 17:32, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
> When -ts_theta_adapt is used, then it detects the instability as an error
> and reduces the step by a lot! wlte=1.24e+03 which means that the reduction
> should be severe but the controller tries 0.1*dt and that seems to pass but
> it "jig-saws" (take a look at the next attempted step), which means that it
> is likely unstable.

I think -ts_theta_adapt is seriously broken, I cannot make sense of
the way the error estimator is computed.

I have an alternative implementation of adaptivity in PetIGA in custom
Generalized-Alpha implementation. The adaptivity is not based on
embed-RK like for TSCN, but in storing a solution vector from the
previous step then estimate the LTE with backward differences. BTW,
this idea can be generalized to any second-order method at the cost of
tracking an extra solution vector and a bunch of Vec operations.

Emil, if you want to try my code to give it a try, it is very easy to
integrate, you just need to copy a file to your sources
(https://bitbucket.org/dalcinl/petiga/src/tip/src/tsalpha1.c) and add
a TSRegister() call after PetscInitialize(). Then you pass in the
command line -ts_type alpha1 -ts_alpha_adapt. You can also pass
-ts_alpha_radius 0.5 to add some high-frequency dissipation (the
default is 1.0, then the method is roughly equivalent to the midpoint
rule).

PS: I would like to put all this in PETSc, but before that, I need
help from you guys to make sure this method really makes sense. The
fact that this has worked very well for me in some problems is not
enough.

-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From khaipham at utexas.edu  Mon Mar 23 08:41:10 2015
From: khaipham at utexas.edu (Khai Pham)
Date: Mon, 23 Mar 2015 08:41:10 -0500
Subject: [petsc-users] Object is in wrong state,
	Matrix is missing diagonal entry
Message-ID: <CANL+ODpEbH8UGmzOTibybtFroiFYtuLsfy3kcg4DXxsMrwxV-g@mail.gmail.com>

Dear Petsc-Users,

I received the below error message. There could be some error in the matrix
assembly and I try to find it out. Based on the error message, could you
give me any hint where should I focus on for finding the bug ? Thanks

Khai


[2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: Object is in wrong state
[2]PETSC ERROR: Matrix is missing diagonal entry 1
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
[2]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
23 08:24:40 2015
[2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
--download-metis --download-parmetis
[2]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
/home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
[2]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
/home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
[2]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
[2]PETSC ERROR: #4 PCSetUp() line 902 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: #5 KSPSetUp() line 305 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
[2]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #9 KSPSolve() line 418 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
/home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
[2]PETSC ERROR: #11 SNESSolve() line 3743 in
/home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
[2]PETSC ERROR: #12 PF_solver() line 63 in src/PF_solver.cpp
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Object is in wrong state
[1]PETSC ERROR: Matrix is missing diagonal entry 0
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
[1]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
23 08:24:40 2015
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
--download-metis --download-parmetis
[1]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
/home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
[1]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
/home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
[1]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
[1]PETSC ERROR: #4 PCSetUp() line 902 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #5 KSPSetUp() line 305 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
[1]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
/home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #9 KSPSolve() line 418 in
/home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
/home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #11 SNESSolve() line 3743 in
/home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150323/ebcadb69/attachment.html>

From knepley at gmail.com  Mon Mar 23 08:47:08 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 23 Mar 2015 07:47:08 -0600
Subject: [petsc-users] Object is in wrong state,
 Matrix is missing diagonal entry
In-Reply-To: <CANL+ODpEbH8UGmzOTibybtFroiFYtuLsfy3kcg4DXxsMrwxV-g@mail.gmail.com>
References: <CANL+ODpEbH8UGmzOTibybtFroiFYtuLsfy3kcg4DXxsMrwxV-g@mail.gmail.com>
Message-ID: <CAMYG4GnpFe-jAfiuhOUEEOyE9UDyUwX2vpgdfFym9pjq1GiE2Q@mail.gmail.com>

On Mon, Mar 23, 2015 at 7:41 AM, Khai Pham <khaipham at utexas.edu> wrote:

> Dear Petsc-Users,
>
> I received the below error message. There could be some error in the
> matrix assembly and I try to find it out. Based on the error message, could
> you give me any hint where should I focus on for finding the bug ? Thanks
>

If you want to use ILU, we require that a diagonal element exists in the
matrix. If it is 0, you must explicitly
set the 0.

  Thanks,

     Matt


> Khai
>
>
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: Object is in wrong state
> [2]PETSC ERROR: Matrix is missing diagonal entry 1
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [2]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
> [2]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
> 23 08:24:40 2015
> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
> --download-metis --download-parmetis
> [2]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
> /home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
> [2]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
> /home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
> [2]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
> [2]PETSC ERROR: #4 PCSetUp() line 902 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
> [2]PETSC ERROR: #5 KSPSetUp() line 305 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
> [2]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
> [2]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: #9 KSPSolve() line 418 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
> /home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
> [2]PETSC ERROR: #11 SNESSolve() line 3743 in
> /home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
> [2]PETSC ERROR: #12 PF_solver() line 63 in src/PF_solver.cpp
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Object is in wrong state
> [1]PETSC ERROR: Matrix is missing diagonal entry 0
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
> [1]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
> 23 08:24:40 2015
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
> --download-metis --download-parmetis
> [1]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
> /home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
> [1]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
> /home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
> [1]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
> [1]PETSC ERROR: #4 PCSetUp() line 902 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #5 KSPSetUp() line 305 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
> [1]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #9 KSPSolve() line 418 in
> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
> /home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #11 SNESSolve() line 3743 in
> /home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150323/49504ae7/attachment.html>

From khaipham at utexas.edu  Mon Mar 23 09:28:01 2015
From: khaipham at utexas.edu (Khai Pham)
Date: Mon, 23 Mar 2015 09:28:01 -0500
Subject: [petsc-users] Object is in wrong state,
 Matrix is missing diagonal entry
In-Reply-To: <CAMYG4GnpFe-jAfiuhOUEEOyE9UDyUwX2vpgdfFym9pjq1GiE2Q@mail.gmail.com>
References: <CANL+ODpEbH8UGmzOTibybtFroiFYtuLsfy3kcg4DXxsMrwxV-g@mail.gmail.com>
	<CAMYG4GnpFe-jAfiuhOUEEOyE9UDyUwX2vpgdfFym9pjq1GiE2Q@mail.gmail.com>
Message-ID: <CANL+ODqYSYH11qX8+8SO9o=C8m8_5_3E2-c_=LyKN=uxeBg-Og@mail.gmail.com>

Hi Matt,

Thank you for your suggestion. It solves the problem.

Khai

On Mon, Mar 23, 2015 at 8:47 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Mar 23, 2015 at 7:41 AM, Khai Pham <khaipham at utexas.edu> wrote:
>
>> Dear Petsc-Users,
>>
>> I received the below error message. There could be some error in the
>> matrix assembly and I try to find it out. Based on the error message, could
>> you give me any hint where should I focus on for finding the bug ? Thanks
>>
>
> If you want to use ILU, we require that a diagonal element exists in the
> matrix. If it is 0, you must explicitly
> set the 0.
>
>   Thanks,
>
>      Matt
>
>
>> Khai
>>
>>
>> [2]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: Object is in wrong state
>> [2]PETSC ERROR: Matrix is missing diagonal entry 1
>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [2]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
>> [2]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
>> 23 08:24:40 2015
>> [2]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
>> --download-metis --download-parmetis
>> [2]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
>> /home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
>> [2]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
>> /home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
>> [2]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
>> [2]PETSC ERROR: #4 PCSetUp() line 902 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
>> [2]PETSC ERROR: #5 KSPSetUp() line 305 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [2]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
>> [2]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
>> [2]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [2]PETSC ERROR: #9 KSPSolve() line 418 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [2]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
>> /home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
>> [2]PETSC ERROR: #11 SNESSolve() line 3743 in
>> /home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
>> [2]PETSC ERROR: #12 PF_solver() line 63 in src/PF_solver.cpp
>> [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [1]PETSC ERROR: Object is in wrong state
>> [1]PETSC ERROR: Matrix is missing diagonal entry 0
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [1]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014
>> [1]PETSC ERROR: ./3DMMF on a arch-linux2-c-debug named khp by khp Mon Mar
>> 23 08:24:40 2015
>> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5
>> --download-metis --download-parmetis
>> [1]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1738 in
>> /home/khp/bin/petsc-3.5.1/src/mat/impls/aij/seq/aijfact.c
>> [1]PETSC ERROR: #2 MatILUFactorSymbolic() line 6203 in
>> /home/khp/bin/petsc-3.5.1/src/mat/interface/matrix.c
>> [1]PETSC ERROR: #3 PCSetUp_ILU() line 206 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/factor/ilu/ilu.c
>> [1]PETSC ERROR: #4 PCSetUp() line 902 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
>> [1]PETSC ERROR: #5 KSPSetUp() line 305 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Singleblock() line 651 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/impls/bjacobi/bjacobi.c
>> [1]PETSC ERROR: #7 PCSetUpOnBlocks() line 935 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/pc/interface/precon.c
>> [1]PETSC ERROR: #8 KSPSetUpOnBlocks() line 166 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #9 KSPSolve() line 418 in
>> /home/khp/bin/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 232 in
>> /home/khp/bin/petsc-3.5.1/src/snes/impls/ls/ls.c
>> [1]PETSC ERROR: #11 SNESSolve() line 3743 in
>> /home/khp/bin/petsc-3.5.1/src/snes/interface/snes.c
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150323/4e3951ad/attachment-0001.html>

From jed at jedbrown.org  Mon Mar 23 17:34:29 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 23 Mar 2015 16:34:29 -0600
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
Message-ID: <87d23z5one.fsf@jedbrown.org>

Lisandro Dalcin <dalcinl at gmail.com> writes:

> On 21 March 2015 at 17:32, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>> When -ts_theta_adapt is used, then it detects the instability as an error
>> and reduces the step by a lot! wlte=1.24e+03 which means that the reduction
>> should be severe but the controller tries 0.1*dt and that seems to pass but
>> it "jig-saws" (take a look at the next attempted step), which means that it
>> is likely unstable.
>
> I think -ts_theta_adapt is seriously broken, I cannot make sense of
> the way the error estimator is computed.

This was Shri's implementation.  TSEvaluateStep_Theta looks wrong to me,
both using U as an input and assuming that Xdot is set.  I don't even
know what the intended math is for a first-order embedded estimate in
midpoint (theta=0.5).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150323/493e5d85/attachment.pgp>

From emconsta at mcs.anl.gov  Mon Mar 23 20:08:22 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Mon, 23 Mar 2015 20:08:22 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <87d23z5one.fsf@jedbrown.org>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
	<87d23z5one.fsf@jedbrown.org>
Message-ID: <5510B906.3040504@mcs.anl.gov>

On 3/23/15 5:34 PM, Jed Brown wrote:
> Lisandro Dalcin <dalcinl at gmail.com> writes:
>
>> On 21 March 2015 at 17:32, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>>> When -ts_theta_adapt is used, then it detects the instability as an error
>>> and reduces the step by a lot! wlte=1.24e+03 which means that the reduction
>>> should be severe but the controller tries 0.1*dt and that seems to pass but
>>> it "jig-saws" (take a look at the next attempted step), which means that it
>>> is likely unstable.
>>
>> I think -ts_theta_adapt is seriously broken, I cannot make sense of
>> the way the error estimator is computed.
>
> This was Shri's implementation.  TSEvaluateStep_Theta looks wrong to me,
> both using U as an input and assuming that Xdot is set.  I don't even
> know what the intended math is for a first-order embedded estimate in
> midpoint (theta=0.5).

I was off for the last few days and I wasn't able to follow the call 
sequence - I had assumed it was correct. I'll take Debo and look into it.

Emil


From emconsta at mcs.anl.gov  Mon Mar 23 22:15:50 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Mon, 23 Mar 2015 22:15:50 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
Message-ID: <5510D6E6.2060700@mcs.anl.gov>

On 3/23/15 4:44 AM, Lisandro Dalcin wrote:
> I have an alternative implementation of adaptivity in PetIGA in custom
> Generalized-Alpha implementation. The adaptivity is not based on
> embed-RK like for TSCN, but in storing a solution vector from the
> previous step then estimate the LTE with backward differences. BTW,
> this idea can be generalized to any second-order method at the cost of
> tracking an extra solution vector and a bunch of Vec operations.
>
> Emil, if you want to try my code to give it a try, it is very easy to
> integrate, you just need to copy a file to your sources
> (https://bitbucket.org/dalcinl/petiga/src/tip/src/tsalpha1.c) and add
> a TSRegister() call after PetscInitialize(). Then you pass in the
> command line -ts_type alpha1 -ts_alpha_adapt. You can also pass
> -ts_alpha_radius 0.5 to add some high-frequency dissipation (the
> default is 1.0, then the method is roughly equivalent to the midpoint
> rule).
>
> PS: I would like to put all this in PETSc, but before that, I need
> help from you guys to make sure this method really makes sense. The
> fact that this has worked very well for me in some problems is not
> enough.

Lisandro, that's a neat idea. If you are basically moving in the 
multistep realm for error estimation with one-step methods, I think 
there are two considerations:

1. The equations need to be continuous (no restarting, no events) or 
they would have to reset the error estimator somehow.

2. Backward differences are sensitive to time stepping changes; unlike 
one-step methods that can accommodate any step changes, B-D has strict 
limits for stability. While this does not play a crucial role (it is 
just the estimator); if the time step varies wildly (can be easily 
constrained), it may create some problems with the estimator.

Emil

From dalcinl at gmail.com  Tue Mar 24 05:20:44 2015
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Mar 2015 13:20:44 +0300
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <87d23z5one.fsf@jedbrown.org>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
	<87d23z5one.fsf@jedbrown.org>
Message-ID: <CAEcYPwCPMjNeJX0rRrx7eHzcMpURjY_nQwHSpAKQwwoz61NZYw@mail.gmail.com>

On 24 March 2015 at 01:34, Jed Brown <jed at jedbrown.org> wrote:
> Lisandro Dalcin <dalcinl at gmail.com> writes:
>
>> On 21 March 2015 at 17:32, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>>> When -ts_theta_adapt is used, then it detects the instability as an error
>>> and reduces the step by a lot! wlte=1.24e+03 which means that the reduction
>>> should be severe but the controller tries 0.1*dt and that seems to pass but
>>> it "jig-saws" (take a look at the next attempted step), which means that it
>>> is likely unstable.
>>
>> I think -ts_theta_adapt is seriously broken, I cannot make sense of
>> the way the error estimator is computed.
>
> This was Shri's implementation.  TSEvaluateStep_Theta looks wrong to me,
> both using U as an input and assuming that Xdot is set.  I don't even
> know what the intended math is for a first-order embedded estimate in
> midpoint (theta=0.5).

Pease note that -ts_adapt_tetha errors for midpoint. You actually need
-ts_theta_endpoint to use adaptivity. But it is there, in the endpoint
version, that I cannot make sense of the implementation. The
trapezoidal rule is second-order, thus the LTE is O(dt^3). The
computed error estimator should be O(dt^2), but the current code seems
to compute an estimator O(dt), and that's the root of all evil.


-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From dalcinl at gmail.com  Tue Mar 24 05:31:17 2015
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Mar 2015 13:31:17 +0300
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <5510D6E6.2060700@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
	<5510D6E6.2060700@mcs.anl.gov>
Message-ID: <CAEcYPwAPg+3LjtfFEwSFi7rOLDsa2fn1cqDu9dU2HhmkUn1arg@mail.gmail.com>

On 24 March 2015 at 06:15, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
> On 3/23/15 4:44 AM, Lisandro Dalcin wrote:
>
> Lisandro, that's a neat idea. If you are basically moving in the multistep
> realm for error estimation with one-step methods,

I'm wondering if no one ever did it. I could not find any reference.

> I think there are two
> considerations:
>
> 1. The equations need to be continuous (no restarting, no events) or they
> would have to reset the error estimator somehow.
>

Indeed. My generalized-alpha code does not handle events at all. I'll
take a look into this. About restarting (not sure if we are talking
about the same thing), I have some code that is specific to
generalized-alpha (In the initial time step, I solve twice with
backward-Euler using dt/2 and dt) that let me properly initialize
generalized-alpha and also compute an initial error estimator.

> 2. Backward differences are sensitive to time stepping changes; unlike
> one-step methods that can accommodate any step changes, B-D has strict
> limits for stability. While this does not play a crucial role (it is just
> the estimator);

Of course, though my B-D is order 2, and as you said, it is the just
for the estimator.

> if the time step varies wildly (can be easily constrained),
> it may create some problems with the estimator.
>

Indeed, the default clipping 0.1,10 is usually too wide, I usually do
-ts_adapt_basic_clip 0.5,2.0

Emil, is there any chance you can try my code with your problem? I
really need some feedback to push this to PETSc, otherwise

-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From emconsta at mcs.anl.gov  Tue Mar 24 10:40:50 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Tue, 24 Mar 2015 10:40:50 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <CAEcYPwAPg+3LjtfFEwSFi7rOLDsa2fn1cqDu9dU2HhmkUn1arg@mail.gmail.com>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
	<5510D6E6.2060700@mcs.anl.gov>
	<CAEcYPwAPg+3LjtfFEwSFi7rOLDsa2fn1cqDu9dU2HhmkUn1arg@mail.gmail.com>
Message-ID: <55118582.9030500@mcs.anl.gov>

On 3/24/15 5:31 AM, Lisandro Dalcin wrote:
> On 24 March 2015 at 06:15, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>> On 3/23/15 4:44 AM, Lisandro Dalcin wrote:
>>
>> Lisandro, that's a neat idea. If you are basically moving in the multistep
>> realm for error estimation with one-step methods,
>
> I'm wondering if no one ever did it. I could not find any reference.

Not to my knowledge.

>> I think there are two
>> considerations:
>>
>> 1. The equations need to be continuous (no restarting, no events) or they
>> would have to reset the error estimator somehow.
>>
>
> Indeed. My generalized-alpha code does not handle events at all. I'll
> take a look into this. About restarting (not sure if we are talking
> about the same thing), I have some code that is specific to
> generalized-alpha (In the initial time step, I solve twice with
> backward-Euler using dt/2 and dt) that let me properly initialize
> generalized-alpha and also compute an initial error estimator.

Yes, that's sensible.

>> 2. Backward differences are sensitive to time stepping changes; unlike
>> one-step methods that can accommodate any step changes, B-D has strict
>> limits for stability. While this does not play a crucial role (it is just
>> the estimator);
>
> Of course, though my B-D is order 2, and as you said, it is the just
> for the estimator.
>
>> if the time step varies wildly (can be easily constrained),
>> it may create some problems with the estimator.
>>
>
> Indeed, the default clipping 0.1,10 is usually too wide, I usually do
> -ts_adapt_basic_clip 0.5,2.0

The limits are a bit more strict for BDF2, but that's for the worst case 
and no filtering. So that looks good too.

> Emil, is there any chance you can try my code with your problem? I
> really need some feedback to push this to PETSc, otherwise

Yes, we are a little backlogged, but we'll try it asap.

Emil

From dalcinl at gmail.com  Tue Mar 24 11:51:39 2015
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Mar 2015 19:51:39 +0300
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <55118582.9030500@mcs.anl.gov>
References: <A585293F-CB8E-4ACA-B1F1-898F18E5CEA4@mcs.anl.gov>
	<1426907935637.9e8be15c@Nodemailer>
	<1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov>
	<550D8101.6070006@mcs.anl.gov>
	<CAEcYPwBFYuZ22u=YpVyG4UBt+LwzvDmsBctOiKNQvVbvO_tVTQ@mail.gmail.com>
	<5510D6E6.2060700@mcs.anl.gov>
	<CAEcYPwAPg+3LjtfFEwSFi7rOLDsa2fn1cqDu9dU2HhmkUn1arg@mail.gmail.com>
	<55118582.9030500@mcs.anl.gov>
Message-ID: <CAEcYPwDvf1X3oT-0f-EU3pCM2uWaa1gKwA3ru8MeNXdxka2wrw@mail.gmail.com>

On 24 March 2015 at 18:40, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>> Of course, though my B-D is order 2, and as you said, it is the just
>> for the estimator.
>>
>>> if the time step varies wildly (can be easily constrained),
>>> it may create some problems with the estimator.
>>>
>>
>> Indeed, the default clipping 0.1,10 is usually too wide, I usually do
>> -ts_adapt_basic_clip 0.5,2.0
>
>
> The limits are a bit more strict for BDF2, but that's for the worst case and
> no filtering. So that looks good too.

I'm not sure I'm following you 100%. Please note I'm not using BDF2 in
the usual sense, what I do is to compute an A POSTERIORI error
estimator using a BD of order 2 to estimate LTE^{n+1} with the
solution vectors U^{n+1}, U^{n} and U^{n-1}, and this is AFTER
computing U^{n-1}.


-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From mailinglists at xgm.de  Wed Mar 25 10:13:59 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Wed, 25 Mar 2015 16:13:59 +0100
Subject: [petsc-users] petsc4py: Passing command line switches
Message-ID: <1778431.i6UKsYHAes@asaru>

Hello,

I'm using the petsc4py. It works fine after some hurdling with missing documentation. (were not able to find API doc anywhere, I generated it myself now).

I try to use the switches like -ksp_monitor like that:

import petsc4py
from petsc4py import PETSc

petsc4py.init(sys.argv)

but no reaction when calling my script like that:

python prbf.py -ksp_monitor

What am I doing wrong? I also tried calling init before importing PETSc but no changes.

Thanks,
Florian

From knepley at gmail.com  Wed Mar 25 10:17:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Mar 2015 10:17:45 -0500
Subject: [petsc-users] petsc4py: Passing command line switches
In-Reply-To: <1778431.i6UKsYHAes@asaru>
References: <1778431.i6UKsYHAes@asaru>
Message-ID: <CAMYG4Gndtra67QGqfrE094RRQxuKSaVNv7jSUwfEHt_oOnVEcQ@mail.gmail.com>

On Wed, Mar 25, 2015 at 10:13 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Hello,
>
> I'm using the petsc4py. It works fine after some hurdling with missing
> documentation. (were not able to find API doc anywhere, I generated it
> myself now).
>
> I try to use the switches like -ksp_monitor like that:
>
> import petsc4py
> from petsc4py import PETSc
>
> petsc4py.init(sys.argv)
>
> but no reaction when calling my script like that:
>
> python prbf.py -ksp_monitor
>
> What am I doing wrong? I also tried calling init before importing PETSc
> but no changes.
>

Did you remember to call setFromOptions() on the KSP? We do not configure
anything from
the command line unless you tell us to. That way, people can do things
solely from the API
is they want.

How did you generate the documentation?

  Thanks,

    Matt


> Thanks,
> Florian
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/545f22dc/attachment.html>

From mailinglists at xgm.de  Wed Mar 25 10:29:37 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Wed, 25 Mar 2015 16:29:37 +0100
Subject: [petsc-users] petsc4py: Passing command line switches
In-Reply-To: <CAMYG4Gndtra67QGqfrE094RRQxuKSaVNv7jSUwfEHt_oOnVEcQ@mail.gmail.com>
References: <1778431.i6UKsYHAes@asaru>
	<CAMYG4Gndtra67QGqfrE094RRQxuKSaVNv7jSUwfEHt_oOnVEcQ@mail.gmail.com>
Message-ID: <5642467.qj5eEnC8GK@asaru>

Am Mittwoch, 25. M?rz 2015, 10:17:45 schrieb Matthew Knepley:
> On Wed, Mar 25, 2015 at 10:13 AM, Florian Lindner <mailinglists at xgm.de>
> wrote:
> 
> > Hello,
> >
> > I'm using the petsc4py. It works fine after some hurdling with missing
> > documentation. (were not able to find API doc anywhere, I generated it
> > myself now).
> >
> > I try to use the switches like -ksp_monitor like that:
> >
> > import petsc4py
> > from petsc4py import PETSc
> >
> > petsc4py.init(sys.argv)
> >
> > but no reaction when calling my script like that:
> >
> > python prbf.py -ksp_monitor
> >
> > What am I doing wrong? I also tried calling init before importing PETSc
> > but no changes.
> >
> 
> Did you remember to call setFromOptions() on the KSP? We do not configure
> anything from
> the command line unless you tell us to. That way, people can do things
> solely from the API
> is they want.

Double fail from my said.

I works when called like that:

import petsc4py
petsc4py.init(sys.argv)
from petsc4py import PETSc

Importing PETSc probably calls init and subsequent calls have no effect.
 
> How did you generate the documentation?

Second fail. I tried to generate it yesterday after I cloned the git and was unable to generate myself. make html does not brought much sphinx doc. Then I learned that the tarball contains an epydoc. Is that available online somewhere?

The online repo is a bit confusing, it contains an index.rst https://bitbucket.org/petsc/petsc4py/src/9dce29cab0b03f4d75fdebab087575b9635bd206/docs/index.rst?at=master
but this file seems to be competely dysfunctional. It looks like links in the document but nothing clickable (I can't even mark test), the Blame/Embed/Raw also seem dysfunctional.

Thanks,
Florian

> 
>   Thanks,
> 
>     Matt
> 
> 
> > Thanks,
> > Florian
> >
> 
> 
> 
> 

From knepley at gmail.com  Wed Mar 25 10:34:22 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Mar 2015 10:34:22 -0500
Subject: [petsc-users] petsc4py: Passing command line switches
In-Reply-To: <5642467.qj5eEnC8GK@asaru>
References: <1778431.i6UKsYHAes@asaru>
	<CAMYG4Gndtra67QGqfrE094RRQxuKSaVNv7jSUwfEHt_oOnVEcQ@mail.gmail.com>
	<5642467.qj5eEnC8GK@asaru>
Message-ID: <CAMYG4G=zhWMddNZAf5ZcWM4DYw7oRk8AZo0jpCPvLxD7Lx7BcQ@mail.gmail.com>

On Wed, Mar 25, 2015 at 10:29 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Am Mittwoch, 25. M?rz 2015, 10:17:45 schrieb Matthew Knepley:
> > On Wed, Mar 25, 2015 at 10:13 AM, Florian Lindner <mailinglists at xgm.de>
> > wrote:
> >
> > > Hello,
> > >
> > > I'm using the petsc4py. It works fine after some hurdling with missing
> > > documentation. (were not able to find API doc anywhere, I generated it
> > > myself now).
> > >
> > > I try to use the switches like -ksp_monitor like that:
> > >
> > > import petsc4py
> > > from petsc4py import PETSc
> > >
> > > petsc4py.init(sys.argv)
> > >
> > > but no reaction when calling my script like that:
> > >
> > > python prbf.py -ksp_monitor
> > >
> > > What am I doing wrong? I also tried calling init before importing PETSc
> > > but no changes.
> > >
> >
> > Did you remember to call setFromOptions() on the KSP? We do not configure
> > anything from
> > the command line unless you tell us to. That way, people can do things
> > solely from the API
> > is they want.
>
> Double fail from my said.
>
> I works when called like that:
>
> import petsc4py
> petsc4py.init(sys.argv)
> from petsc4py import PETSc
>
> Importing PETSc probably calls init and subsequent calls have no effect.
>
> > How did you generate the documentation?
>
> Second fail. I tried to generate it yesterday after I cloned the git and
> was unable to generate myself. make html does not brought much sphinx doc.
> Then I learned that the tarball contains an epydoc. Is that available
> online somewhere?
>
> The online repo is a bit confusing, it contains an index.rst
> https://bitbucket.org/petsc/petsc4py/src/9dce29cab0b03f4d75fdebab087575b9635bd206/docs/index.rst?at=master
> but this file seems to be competely dysfunctional. It looks like links in
> the document but nothing clickable (I can't even mark test), the
> Blame/Embed/Raw also seem dysfunctional.
>

That file is obviously old and was removed.

Lisandro, are we generating documentation somewhere?

  Thanks,

     Matt


> Thanks,
> Florian
>
> >
> >   Thanks,
> >
> >     Matt
> >
> >
> > > Thanks,
> > > Florian
> > >
> >
> >
> >
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/26cb6ac1/attachment.html>

From haakon at hakostra.net  Wed Mar 25 10:36:28 2015
From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=)
Date: Wed, 25 Mar 2015 16:36:28 +0100
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <550C8ACC.3030102@hakostra.net>
References: <550A94B9.90407@hakostra.net>	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>	<550B07D8.9060903@hakostra.net>	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>	<550BCBD9.80608@hakostra.net>
	<87zj77ivyg.fsf@jedbrown.org>	<550C3642.5000208@hakostra.net>
	<87h9tfaedt.fsf@jedbrown.org>	<550C40A7.5090303@hakostra.net>
	<878uerachd.fsf@jedbrown.org>	<550C4B47.60903@hakostra.net>	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net>
Message-ID: <5512D5FC.7090108@hakostra.net>

Did you come to any conclusion on this issue?

Regards,
H?kon

On 20. mars 2015 22:02, H?kon Strandenes wrote:
> On 20. mars 2015 20:48, Barry Smith wrote:
>> Why is 1 dimension a special case that is not worthy of its own
>> format? The same thing would hold for 2d and 3d. One could then argue
>> that we should have a single six dimensional format for the files for
>> all vectors that PETSc produces. Then a 1d problem has five of the
>> dimensions being 1.
>
> This is a very good point, and support my view.
>
> Let me come with two very simple example cases:
>
>
> Case 1:
> Create a list of grid points in an external preprocessor for the purpose
> of loading this into a Vec later:
>
>    x = np.linspace(0.0, 1.0, num=21)
>    f.create_dataset('/MESH/nodes/x', data=x)
>
> vs.
>
>    x = np.linspace(0.0, 1.0, num=21)
>    x = x.reshape((21,1))
>    f.create_dataset('/MESH/nodes/x', data=x)
>
>
> Case 2:
> Read three Vecs written to disk by PETSc, and calculate total "bounding
> box volume" of the grid:
>
>    g = h5py.File('grid.h5', 'r')
>    x = g['/MESH/nodes/x']
>    y = g['/MESH/nodes/y']
>    z = g['/MESH/nodes/z']
>    Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
>
> vs.
>
>    g = h5py.File('grid.h5', 'r')
>    x = g['/MESH/nodes/x'][:,0]
>    y = g['/MESH/nodes/y'][:,0]
>    z = g['/MESH/nodes/z'][:,0]
>    Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
>
>
> In both cases I think handling this extra, unnecessary dimension makes
> the code less attractive. It's not that either way is difficult,
> problematic or impossible, but it's just that 1D Vecs should intuitively
> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> quite a while until I figured this out, even after having written an
> entire Navier-Stokes DNS solver using the PETSc library for everything
> except time integration and filling these simple 1D coordinate arrays!
>
> Regards,
> H?kon

From Eric.Chamberland at giref.ulaval.ca  Wed Mar 25 13:03:18 2015
From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland)
Date: Wed, 25 Mar 2015 14:03:18 -0400
Subject: [petsc-users] Column #j is wrong in parallel from message
 "Inserting a new nonzero (i, j) into matrix"
Message-ID: <5512F866.5070405@giref.ulaval.ca>

Hi,

while looking for where in the world do I insert the (135,9) entry in my 
matrix, I have discovered that the column # shown is wrong in parallel!

I am using PETsc 3.5.3.

The full error message is:

[0]PETSC ERROR: MatSetValues_MPIAIJ() line 564 in 
/home/mefpp_ericc/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c Inserting a 
new nonzero (135, 9) into matrix

This line code is a call to a #defined macro:

MatSetValues_SeqAIJ_B_Private(row,col,value,addv);

where the "col" parameter is not equal to "in[j]"!!!

in gdb, printing "in[j]" gave me:

print in[j]
$6 = 537

while "col" is:

print col
$7 = 9

So, I expected to have a message telling me that (135,537) and not 
(135,9) is a new entry matrix!!!

Would it be a big work to fix this so that the col # displayed is correct?

Thanks!

Eric

From knepley at gmail.com  Wed Mar 25 13:06:27 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Mar 2015 13:06:27 -0500
Subject: [petsc-users] Column #j is wrong in parallel from message
 "Inserting a new nonzero (i, j) into matrix"
In-Reply-To: <5512F866.5070405@giref.ulaval.ca>
References: <5512F866.5070405@giref.ulaval.ca>
Message-ID: <CAMYG4G=BUFp+tCors7wjKzY11uiQ_+KYWSbgJ-65AH=tKXAZnw@mail.gmail.com>

On Wed, Mar 25, 2015 at 1:03 PM, Eric Chamberland <
Eric.Chamberland at giref.ulaval.ca> wrote:

> Hi,
>
> while looking for where in the world do I insert the (135,9) entry in my
> matrix, I have discovered that the column # shown is wrong in parallel!
>

We have talked about this before. It is certainly not easy. The error is
given by the embedded serial matrix, whereas you want
the parallel column. It would be good if we could restructure to give that
back.

  Thanks,

      Matt


> I am using PETsc 3.5.3.
>
> The full error message is:
>
> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 564 in
> /home/mefpp_ericc/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c Inserting a
> new nonzero (135, 9) into matrix
>
> This line code is a call to a #defined macro:
>
> MatSetValues_SeqAIJ_B_Private(row,col,value,addv);
>
> where the "col" parameter is not equal to "in[j]"!!!
>
> in gdb, printing "in[j]" gave me:
>
> print in[j]
> $6 = 537
>
> while "col" is:
>
> print col
> $7 = 9
>
> So, I expected to have a message telling me that (135,537) and not (135,9)
> is a new entry matrix!!!
>
> Would it be a big work to fix this so that the col # displayed is correct?
>
> Thanks!
>
> Eric
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/ccb10655/attachment.html>

From Eric.Chamberland at giref.ulaval.ca  Wed Mar 25 13:20:37 2015
From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland)
Date: Wed, 25 Mar 2015 14:20:37 -0400
Subject: [petsc-users] Column #j is wrong in parallel from message
 "Inserting a new nonzero (i, j) into matrix"
In-Reply-To: <CAMYG4G=BUFp+tCors7wjKzY11uiQ_+KYWSbgJ-65AH=tKXAZnw@mail.gmail.com>
References: <5512F866.5070405@giref.ulaval.ca>
	<CAMYG4G=BUFp+tCors7wjKzY11uiQ_+KYWSbgJ-65AH=tKXAZnw@mail.gmail.com>
Message-ID: <5512FC75.8010303@giref.ulaval.ca>

On 03/25/2015 02:06 PM, Matthew Knepley wrote:
> On Wed, Mar 25, 2015 at 1:03 PM, Eric Chamberland
> <Eric.Chamberland at giref.ulaval.ca
> <mailto:Eric.Chamberland at giref.ulaval.ca>> wrote:
>
>     Hi,
>
>     while looking for where in the world do I insert the (135,9) entry
>     in my matrix, I have discovered that the column # shown is wrong in
>     parallel!
>
>
> We have talked about this before. It is certainly not easy. The error is
> given by the embedded serial matrix, whereas you want
> the parallel column. It would be good if we could restructure to give
> that back.

Okay!!!  I didn't knew that...

Would it be possible, in parallel, to have the column number marked by a 
****** beside it with a footnote like "this is not the column number" in 
the error message?  How can it help *me* to have this wrong number? 
(just wasted a few hours on this this morning....)

Worded differently: if you take the time to write a message for the end 
user, it is to help the programmer right?  But what does it worth if it 
does not help, worst, if it mislead the programmer???

Ok, that's it for my 0.5cent...

Thanks for the answer,

Eric

>
>    Thanks,
>
>        Matt
>
>     I am using PETsc 3.5.3.
>
>     The full error message is:
>
>     [0]PETSC ERROR: MatSetValues_MPIAIJ() line 564 in
>     /home/mefpp_ericc/petsc-3.5.3/__src/mat/impls/aij/mpi/mpiaij.c
>     Inserting a new nonzero (135, 9) into matrix
>
>     This line code is a call to a #defined macro:
>
>     MatSetValues_SeqAIJ_B_Private(__row,col,value,addv);
>
>     where the "col" parameter is not equal to "in[j]"!!!
>
>     in gdb, printing "in[j]" gave me:
>
>     print in[j]
>     $6 = 537
>
>     while "col" is:
>
>     print col
>     $7 = 9
>
>     So, I expected to have a message telling me that (135,537) and not
>     (135,9) is a new entry matrix!!!
>
>     Would it be a big work to fix this so that the col # displayed is
>     correct?
>
>     Thanks!
>
>     Eric
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener


From bsmith at mcs.anl.gov  Wed Mar 25 14:26:54 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Mar 2015 14:26:54 -0500
Subject: [petsc-users] Column #j is wrong in parallel from message
	"Inserting a new nonzero (i, j) into matrix"
In-Reply-To: <5512F866.5070405@giref.ulaval.ca>
References: <5512F866.5070405@giref.ulaval.ca>
Message-ID: <D5321D39-4D34-40BA-B23E-694E63E14125@mcs.anl.gov>


  Eric,

    This is a good idea. I've fixed it for MatSetValues, MatSetValuesLocal, and MatSetValuesStencil, for AIJ,BAIJ, and SBAIJ matrices in the branch barry/fix-inserting-new-nonzero-column-location. 

    Fixing for the MatSetValuesBlocked family of methods for BAIJ and SBAIJ is tricky since the global information is lost before reaching the location that generates the error message. But you are right the current situation is very confusing and we should find a way to fix it.

  Barry



> On Mar 25, 2015, at 1:03 PM, Eric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
> 
> Hi,
> 
> while looking for where in the world do I insert the (135,9) entry in my matrix, I have discovered that the column # shown is wrong in parallel!
> 
> I am using PETsc 3.5.3.
> 
> The full error message is:
> 
> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 564 in /home/mefpp_ericc/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c Inserting a new nonzero (135, 9) into matrix
> 
> This line code is a call to a #defined macro:
> 
> MatSetValues_SeqAIJ_B_Private(row,col,value,addv);
> 
> where the "col" parameter is not equal to "in[j]"!!!
> 
> in gdb, printing "in[j]" gave me:
> 
> print in[j]
> $6 = 537
> 
> while "col" is:
> 
> print col
> $7 = 9
> 
> So, I expected to have a message telling me that (135,537) and not (135,9) is a new entry matrix!!!
> 
> Would it be a big work to fix this so that the col # displayed is correct?
> 
> Thanks!
> 
> Eric


From harshad.sahasrabudhe at gmail.com  Wed Mar 25 15:29:35 2015
From: harshad.sahasrabudhe at gmail.com (Harshad Sahasrabudhe)
Date: Wed, 25 Mar 2015 16:29:35 -0400
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
Message-ID: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>

Hi,

I'm trying to use the ARPACK interface in SLEPc for calculating smallest
eigenvalues with eigenvectors of a generalized eigenproblem. The matrices
are symmetric.

What are the suggested linear solvers/preconditioners for this type of a
system when using ARPACK? I am using shift and invert with a shift of 0.

I get a lot of spurious eigenvalues when I use GMRES linear solver or MUMPS
for LU factorization. Chebyshev doesn't seem to converge (I don't have a
good guess for the higher end of eigenvalues).

Thanks,
Harshad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/ab9a1657/attachment.html>

From jroman at dsic.upv.es  Wed Mar 25 15:37:45 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 25 Mar 2015 21:37:45 +0100
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
Message-ID: <10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>


El 25/03/2015, a las 21:29, Harshad Sahasrabudhe escribi?:

> Hi,
> 
> I'm trying to use the ARPACK interface in SLEPc for calculating smallest eigenvalues with eigenvectors of a generalized eigenproblem. The matrices are symmetric.
> 
> What are the suggested linear solvers/preconditioners for this type of a system when using ARPACK? I am using shift and invert with a shift of 0.
> 
> I get a lot of spurious eigenvalues when I use GMRES linear solver or MUMPS for LU factorization. Chebyshev doesn't seem to converge (I don't have a good guess for the higher end of eigenvalues). 
> 
> Thanks,
> Harshad

With MUMPS you should not get spurious eigenvalues. Did you try the krylovschur solver? How do you know the eigenvalues are wrong? Are you setting problem type to GHEP?

Jose


From harshad.sahasrabudhe at gmail.com  Wed Mar 25 15:47:39 2015
From: harshad.sahasrabudhe at gmail.com (Harshad Sahasrabudhe)
Date: Wed, 25 Mar 2015 16:47:39 -0400
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
	<10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
Message-ID: <CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>

>
> With MUMPS you should not get spurious eigenvalues.

I get only a few spurious eigenvalues when using MUMPS with ARPACK, but the
eigenvectors are definitely wrong.

Did you try the krylovschur solver?

Yes, Krylov-Schur gives me correct results.

How do you know the eigenvalues are wrong?

I'm testing my implementation of Schrodinger equation solver with a 3D
harmonic oscillator potential. I'm getting correct results using FEAST and
krylovschur solvers.

Are you setting problem type to GHEP?

Yes



On Wed, Mar 25, 2015 at 4:37 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 25/03/2015, a las 21:29, Harshad Sahasrabudhe escribi?:
>
> > Hi,
> >
> > I'm trying to use the ARPACK interface in SLEPc for calculating smallest
> eigenvalues with eigenvectors of a generalized eigenproblem. The matrices
> are symmetric.
> >
> > What are the suggested linear solvers/preconditioners for this type of a
> system when using ARPACK? I am using shift and invert with a shift of 0.
> >
> > I get a lot of spurious eigenvalues when I use GMRES linear solver or
> MUMPS for LU factorization. Chebyshev doesn't seem to converge (I don't
> have a good guess for the higher end of eigenvalues).
> >
> > Thanks,
> > Harshad
>
> With MUMPS you should not get spurious eigenvalues. Did you try the
> krylovschur solver? How do you know the eigenvalues are wrong? Are you
> setting problem type to GHEP?
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/7b3fbc57/attachment-0001.html>

From jroman at dsic.upv.es  Wed Mar 25 16:00:55 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 25 Mar 2015 22:00:55 +0100
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
	<10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
	<CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>
Message-ID: <FE5DC4AA-6C1F-4E18-82E8-2E68F68EB62E@dsic.upv.es>


El 25/03/2015, a las 21:47, Harshad Sahasrabudhe escribi?:

> With MUMPS you should not get spurious eigenvalues.
> I get only a few spurious eigenvalues when using MUMPS with ARPACK, but the eigenvectors are definitely wrong.
> 
> Did you try the krylovschur solver?
> Yes, Krylov-Schur gives me correct results.
> 
> How do you know the eigenvalues are wrong?
> I'm testing my implementation of Schrodinger equation solver with a 3D harmonic oscillator potential. I'm getting correct results using FEAST and krylovschur solvers.
> 
> Are you setting problem type to GHEP?
> Yes 
> 

Did you try the arpack-ng version? Seems that some people have taken over maintainance.

Anyway, why do you insist in using ARPACK when SLEPc's Krylov-Schur work? ARPACK will not give you any further advantage.

Jose


From harshad.sahasrabudhe at gmail.com  Wed Mar 25 16:06:21 2015
From: harshad.sahasrabudhe at gmail.com (Harshad Sahasrabudhe)
Date: Wed, 25 Mar 2015 17:06:21 -0400
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <FE5DC4AA-6C1F-4E18-82E8-2E68F68EB62E@dsic.upv.es>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
	<10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
	<CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>
	<FE5DC4AA-6C1F-4E18-82E8-2E68F68EB62E@dsic.upv.es>
Message-ID: <CAC0ZLSroaH6Yuzmkd-cPX9juiCK=+WrpiBj8ra54s-tv-8F7OA@mail.gmail.com>

>
> Anyway, why do you insist in using ARPACK when SLEPc's Krylov-Schur work?
> ARPACK will not give you any further advantage.


I thought ARPACK was faster when the system size is large and number of
eigenvalues required is small. I will be working with sparse matrices of
size ~60,000. Does SLEPC's Krylov-Schur have about the same performance as
ARPACK for calculating ~50 eigenvalues for such matrices?

On Wed, Mar 25, 2015 at 5:00 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 25/03/2015, a las 21:47, Harshad Sahasrabudhe escribi?:
>
> > With MUMPS you should not get spurious eigenvalues.
> > I get only a few spurious eigenvalues when using MUMPS with ARPACK, but
> the eigenvectors are definitely wrong.
> >
> > Did you try the krylovschur solver?
> > Yes, Krylov-Schur gives me correct results.
> >
> > How do you know the eigenvalues are wrong?
> > I'm testing my implementation of Schrodinger equation solver with a 3D
> harmonic oscillator potential. I'm getting correct results using FEAST and
> krylovschur solvers.
> >
> > Are you setting problem type to GHEP?
> > Yes
> >
>
> Did you try the arpack-ng version? Seems that some people have taken over
> maintainance.
>
> Anyway, why do you insist in using ARPACK when SLEPc's Krylov-Schur work?
> ARPACK will not give you any further advantage.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/519fc54a/attachment.html>

From jroman at dsic.upv.es  Wed Mar 25 16:18:13 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 25 Mar 2015 22:18:13 +0100
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <CAC0ZLSroaH6Yuzmkd-cPX9juiCK=+WrpiBj8ra54s-tv-8F7OA@mail.gmail.com>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
	<10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
	<CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>
	<FE5DC4AA-6C1F-4E18-82E8-2E68F68EB62E@dsic.upv.es>
	<CAC0ZLSroaH6Yuzmkd-cPX9juiCK=+WrpiBj8ra54s-tv-8F7OA@mail.gmail.com>
Message-ID: <E4AF06AF-26AF-4AC9-A418-C75BD320787B@dsic.upv.es>


El 25/03/2015, a las 22:06, Harshad Sahasrabudhe escribi?:

> 
> I thought ARPACK was faster when the system size is large and number of eigenvalues required is small. I will be working with sparse matrices of size ~60,000. Does SLEPC's Krylov-Schur have about the same performance as ARPACK for calculating ~50 eigenvalues for such matrices?
> 

Absolutely. Also, with Krylov-Schur you can adjust the restart parameter (which is hidden in ARPACK); it may help improve convergence in some cases.
http://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSKrylovSchurSetRestart.html

Jose


From harshad.sahasrabudhe at gmail.com  Wed Mar 25 16:21:41 2015
From: harshad.sahasrabudhe at gmail.com (Harshad Sahasrabudhe)
Date: Wed, 25 Mar 2015 17:21:41 -0400
Subject: [petsc-users] SLEPc - Generalized symmetric eigenproblem using
	ARPACK
In-Reply-To: <E4AF06AF-26AF-4AC9-A418-C75BD320787B@dsic.upv.es>
References: <CAC0ZLSrDDBuF_kGARX2Y=Vfvv5V8X_KwBF9KrARuHrex11gQdg@mail.gmail.com>
	<10D71B97-A07D-4744-BFDC-6151C005E09F@dsic.upv.es>
	<CAC0ZLSp=aF4KiZSfK3VUH-7XeyCYho9nforg-G=AXHXJ3Js_kQ@mail.gmail.com>
	<FE5DC4AA-6C1F-4E18-82E8-2E68F68EB62E@dsic.upv.es>
	<CAC0ZLSroaH6Yuzmkd-cPX9juiCK=+WrpiBj8ra54s-tv-8F7OA@mail.gmail.com>
	<E4AF06AF-26AF-4AC9-A418-C75BD320787B@dsic.upv.es>
Message-ID: <CAC0ZLSpg+uFmn+NNZCgkzD01sZyVG4t1Fc05KR1vsx94_SKqdg@mail.gmail.com>

>
> Absolutely. Also, with Krylov-Schur you can adjust the restart parameter
> (which is hidden in ARPACK); it may help improve convergence in some cases.


Awesome. I'll just use Krylov-Schur then. Thanks.

On Wed, Mar 25, 2015 at 5:18 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 25/03/2015, a las 22:06, Harshad Sahasrabudhe escribi?:
>
> >
> > I thought ARPACK was faster when the system size is large and number of
> eigenvalues required is small. I will be working with sparse matrices of
> size ~60,000. Does SLEPC's Krylov-Schur have about the same performance as
> ARPACK for calculating ~50 eigenvalues for such matrices?
> >
>
> Absolutely. Also, with Krylov-Schur you can adjust the restart parameter
> (which is hidden in ARPACK); it may help improve convergence in some cases.
>
> http://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSKrylovSchurSetRestart.html
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/d38c97fc/attachment.html>

From hus003 at ucsd.edu  Wed Mar 25 18:50:30 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Wed, 25 Mar 2015 23:50:30 +0000
Subject: [petsc-users] How do I set PETSc LU reuse factorization
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010F2B31@XMAIL-MBX-BH1.AD.UCSD.EDU>

Hello,

If I have a KSP, whose Mat will be reused for a good number of times, and if I choose to use LU factorization, how do I set the parameter so that the factorization can be reused every time?

Best,
Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150325/9c84e3cd/attachment.html>

From bsmith at mcs.anl.gov  Wed Mar 25 19:38:09 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Mar 2015 19:38:09 -0500
Subject: [petsc-users] How do I set PETSc LU reuse factorization
In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010F2B31@XMAIL-MBX-BH1.AD.UCSD.EDU>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010F2B31@XMAIL-MBX-BH1.AD.UCSD.EDU>
Message-ID: <589FD003-7B64-4A8A-AEE5-26C0325C4AA3@mcs.anl.gov>


> On Mar 25, 2015, at 6:50 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
> 
> Hello, 
> 
> If I have a KSP, whose Mat will be reused for a good number of times, and if I choose to use LU factorization, how do I set the parameter so that the factorization can be reused every time? 

   By default if any of the matrix entries change then it will do a new LU factorization (if the matrix does not change it will continue to use the current factorization. If you want to reuse the factorization even after the matrix changes you call KSPSetReusePreconditioner() and it will use the same factorization until you call KSPSetReusePreconditioner() with PETSC_FALSE.

 Barry

> 
> Best,
> Hui


From hus003 at ucsd.edu  Wed Mar 25 20:00:52 2015
From: hus003 at ucsd.edu (Sun, Hui)
Date: Thu, 26 Mar 2015 01:00:52 +0000
Subject: [petsc-users] How do I set PETSc LU reuse factorization
In-Reply-To: <589FD003-7B64-4A8A-AEE5-26C0325C4AA3@mcs.anl.gov>
References: <7501CC2B7BBCC44A92ECEEC316170ECB010F2B31@XMAIL-MBX-BH1.AD.UCSD.EDU>,
	<589FD003-7B64-4A8A-AEE5-26C0325C4AA3@mcs.anl.gov>
Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010F2B47@XMAIL-MBX-BH1.AD.UCSD.EDU>

Thank you Barry. 

Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Wednesday, March 25, 2015 5:38 PM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] How do I set PETSc LU reuse factorization

> On Mar 25, 2015, at 6:50 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Hello,
>
> If I have a KSP, whose Mat will be reused for a good number of times, and if I choose to use LU factorization, how do I set the parameter so that the factorization can be reused every time?

   By default if any of the matrix entries change then it will do a new LU factorization (if the matrix does not change it will continue to use the current factorization. If you want to reuse the factorization even after the matrix changes you call KSPSetReusePreconditioner() and it will use the same factorization until you call KSPSetReusePreconditioner() with PETSC_FALSE.

 Barry

>
> Best,
> Hui


From bsmith at mcs.anl.gov  Wed Mar 25 22:20:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Mar 2015 22:20:57 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <5512D5FC.7090108@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
Message-ID: <BADDECAA-8992-4D12-9DC5-985167677164@mcs.anl.gov>


> On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> 
> Did you come to any conclusion on this issue?

  No. Different people think a different default is best for different reasons.

  To get the conversation going again I'll propose an approach that allows both; PetscViewerHDF5SetSomething(viewer,PetscBool) where 
Something could be BaseDimension2. And the Load would handle files with either 1 or 2 dimensions as the base as well.

  Barry

> 
> Regards,
> H?kon
> 
> On 20. mars 2015 22:02, H?kon Strandenes wrote:
>> On 20. mars 2015 20:48, Barry Smith wrote:
>>> Why is 1 dimension a special case that is not worthy of its own
>>> format? The same thing would hold for 2d and 3d. One could then argue
>>> that we should have a single six dimensional format for the files for
>>> all vectors that PETSc produces. Then a 1d problem has five of the
>>> dimensions being 1.
>> 
>> This is a very good point, and support my view.
>> 
>> Let me come with two very simple example cases:
>> 
>> 
>> Case 1:
>> Create a list of grid points in an external preprocessor for the purpose
>> of loading this into a Vec later:
>> 
>>   x = np.linspace(0.0, 1.0, num=21)
>>   f.create_dataset('/MESH/nodes/x', data=x)
>> 
>> vs.
>> 
>>   x = np.linspace(0.0, 1.0, num=21)
>>   x = x.reshape((21,1))
>>   f.create_dataset('/MESH/nodes/x', data=x)
>> 
>> 
>> Case 2:
>> Read three Vecs written to disk by PETSc, and calculate total "bounding
>> box volume" of the grid:
>> 
>>   g = h5py.File('grid.h5', 'r')
>>   x = g['/MESH/nodes/x']
>>   y = g['/MESH/nodes/y']
>>   z = g['/MESH/nodes/z']
>>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
>> 
>> vs.
>> 
>>   g = h5py.File('grid.h5', 'r')
>>   x = g['/MESH/nodes/x'][:,0]
>>   y = g['/MESH/nodes/y'][:,0]
>>   z = g['/MESH/nodes/z'][:,0]
>>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
>> 
>> 
>> In both cases I think handling this extra, unnecessary dimension makes
>> the code less attractive. It's not that either way is difficult,
>> problematic or impossible, but it's just that 1D Vecs should intuitively
>> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
>> quite a while until I figured this out, even after having written an
>> entire Navier-Stokes DNS solver using the PETSc library for everything
>> except time integration and filling these simple 1D coordinate arrays!
>> 
>> Regards,
>> H?kon


From Sanjay.Kharche at manchester.ac.uk  Thu Mar 26 02:15:32 2015
From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche)
Date: Thu, 26 Mar 2015 07:15:32 +0000
Subject: [petsc-users] running Petsc programs without a connection.
Message-ID: <E8F644D1675C06428BC49CCCDF77EEEC452F14E7@MBXP03.ds.man.ac.uk>


Dear All

I have a fedora pc which some times cannot have an internet connection. I found that I can run non-Petsc MPI programs when it is not connected to the internet. However, when I try to run my Petsc based program without a connection, I get the following error. By googling a little bit, I found that the mpi launcher requires to log into each of the 4 procs the pc has, and may be using ssh. This or something else may or may not require an internet connection. In my present installation of petsc, I have downloaded and installed all required dependencies (blas, lapack, sundials) including mpich during the configure. I am using a makefile that I took from the ts examples for ex15.c where I changed the ex15 entry to sk2d (my program). I updated the petsc independent mpich to be the same version as the one I got with petsc.

Is there a way of configuring or otherwise to get Petsc programs to run when I do not have an internet connection?

the error:

mpiexec -n 3 ./sk2d
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(498)..............: 
MPID_Init(187).....................: channel initialization failed
MPIDI_CH3_Init(89).................: 
MPID_nem_init(320).................: 
MPID_nem_tcp_init(171).............: 
MPID_nem_tcp_get_business_card(418): 
MPID_nem_tcp_init(377).............: gethostbyname failed, sanjayslaptop.maths.liv.ac.uk (errno 2)


cheers
Sanjay

From dalcinl at gmail.com  Thu Mar 26 05:59:33 2015
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Mar 2015 13:59:33 +0300
Subject: [petsc-users] petsc4py: Passing command line switches
In-Reply-To: <CAMYG4G=zhWMddNZAf5ZcWM4DYw7oRk8AZo0jpCPvLxD7Lx7BcQ@mail.gmail.com>
References: <1778431.i6UKsYHAes@asaru>
	<CAMYG4Gndtra67QGqfrE094RRQxuKSaVNv7jSUwfEHt_oOnVEcQ@mail.gmail.com>
	<5642467.qj5eEnC8GK@asaru>
	<CAMYG4G=zhWMddNZAf5ZcWM4DYw7oRk8AZo0jpCPvLxD7Lx7BcQ@mail.gmail.com>
Message-ID: <CAEcYPwA7LL1J-AW8Rurx0+OCBoRZGC4DbSDrzESNcMYTYXxgRw@mail.gmail.com>

On 25 March 2015 at 18:34, Matthew Knepley <knepley at gmail.com> wrote:
> That file is obviously old and was removed.
>

After installing petsc4py, you can do "make docs" in the top level
source tree, and the docs/ directory will be populated. For this to
work, you need sphinx and epydoc installed.

> Lisandro, are we generating documentation somewhere?

Two places:

+ http://petsc4py.readthedocs.org/ - This does not contain the
epydoc-generated API reference. I'm not sure if setting up readthedocs
to use epydoc is ever possible.

+ https://pythonhosted.org/petsc4py/,  - It is for the last release,
not the in-development version.



-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From mailinglists at xgm.de  Thu Mar 26 07:32:05 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Thu, 26 Mar 2015 13:32:05 +0100
Subject: [petsc-users] petsc4py: Passing command line switches
In-Reply-To: <CAEcYPwA7LL1J-AW8Rurx0+OCBoRZGC4DbSDrzESNcMYTYXxgRw@mail.gmail.com>
References: <1778431.i6UKsYHAes@asaru>
	<CAMYG4G=zhWMddNZAf5ZcWM4DYw7oRk8AZo0jpCPvLxD7Lx7BcQ@mail.gmail.com>
	<CAEcYPwA7LL1J-AW8Rurx0+OCBoRZGC4DbSDrzESNcMYTYXxgRw@mail.gmail.com>
Message-ID: <3708450.0m6gpPsosp@asaru>

Am Donnerstag, 26. M?rz 2015, 13:59:33 schrieb Lisandro Dalcin:
> On 25 March 2015 at 18:34, Matthew Knepley <knepley at gmail.com> wrote:
> > That file is obviously old and was removed.
> >
> 
> After installing petsc4py, you can do "make docs" in the top level
> source tree, and the docs/ directory will be populated. For this to
> work, you need sphinx and epydoc installed.
> 
> > Lisandro, are we generating documentation somewhere?
> 
> Two places:
> 
> + http://petsc4py.readthedocs.org/ - This does not contain the
> epydoc-generated API reference. I'm not sure if setting up readthedocs
> to use epydoc is ever possible.
> 
> + https://pythonhosted.org/petsc4py/,  - It is for the last release,
> not the in-development version.

Great, that is exactly what I was looking for!

> 
> 
> 
> 

From knepley at gmail.com  Thu Mar 26 07:35:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 07:35:19 -0500
Subject: [petsc-users] running Petsc programs without a connection.
In-Reply-To: <E8F644D1675C06428BC49CCCDF77EEEC452F14E7@MBXP03.ds.man.ac.uk>
References: <E8F644D1675C06428BC49CCCDF77EEEC452F14E7@MBXP03.ds.man.ac.uk>
Message-ID: <CAMYG4G=XC9jLDFOcVXBNALGyCs61znQ09Ciiy9zVEUwq_E-w_Q@mail.gmail.com>

On Thu, Mar 26, 2015 at 2:15 AM, Sanjay Kharche <
Sanjay.Kharche at manchester.ac.uk> wrote:

>
> Dear All
>
> I have a fedora pc which some times cannot have an internet connection. I
> found that I can run non-Petsc MPI programs when it is not connected to the
> internet. However, when I try to run my Petsc based program without a
> connection, I get the following error. By googling a little bit, I found
> that the mpi launcher requires to log into each of the 4 procs the pc has,
> and may be using ssh. This or something else may or may not require an
> internet connection. In my present installation of petsc, I have downloaded
> and installed all required dependencies (blas, lapack, sundials) including
> mpich during the configure. I am using a makefile that I took from the ts
> examples for ex15.c where I changed the ex15 entry to sk2d (my program). I
> updated the petsc independent mpich to be the same version as the one I got
> with petsc.
>
> Is there a way of configuring or otherwise to get Petsc programs to run
> when I do not have an internet connection?
>
> the error:
>
> mpiexec -n 3 ./sk2d
> Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(498)..............:
> MPID_Init(187).....................: channel initialization failed
> MPIDI_CH3_Init(89).................:
> MPID_nem_init(320).................:
> MPID_nem_tcp_init(171).............:
> MPID_nem_tcp_get_business_card(418):
> MPID_nem_tcp_init(377).............: gethostbyname failed,
> sanjayslaptop.maths.liv.ac.uk (errno 2)
>

This is not PETSc, but rather the way MPICH is configured. I run using

  -n 3 -hosts localhost,localhost,localhost

and it will run fine disconnected.

Satish, is there a general fix?

  Thanks,

     Matt


>
> cheers
> Sanjay
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/2b015e26/attachment.html>

From mailinglists at xgm.de  Thu Mar 26 08:16:47 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Thu, 26 Mar 2015 14:16:47 +0100
Subject: [petsc-users] Understanding MatCreate bsize parameter
Message-ID: <2259494.rATCdaqx1O@asaru>

Hello,

I'm using petsc with petsc4py.

A matrix is created like that

    MPIrank = MPI.COMM_WORLD.Get_rank()
    MPIsize = MPI.COMM_WORLD.Get_size()
    print("MPI Rank = ", MPIrank)
    print("MPI Size = ", MPIsize)
    parts = partitions()
    
    print("Dimension= ", nSupport + dimension, "bsize = ", len(parts[MPIrank]))

    MPI.COMM_WORLD.Barrier() # Just to keep the output together
    A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport + dimension), bsize = len(parts[MPIrank]) ) # <-- crash here


Output for mpirun -n 2 is like that and it works:

MPI Rank =  0
MPI Size =  2
Dimension=  10 bsize =  5
MPI Rank =  1
MPI Size =  2
Dimension=  10 bsize =  5

But for mpirun -n 3 it crashes:

MPI Rank =  2
MPI Size =  3
Dimension=  10 bsize =  3
MPI Rank =  1
MPI Size =  3
Dimension=  10 bsize =  4
MPI Rank =  0
MPI Size =  3
Dimension=  10 bsize =  3

Error is ValueError: global size 10 not divisible by block size 3.

I tried to dig a bit into the source. Mat_Create uses Mat_Dense to unpack the size and bsize parameters. When there just a scalar n, not a tuple t given as parameter it assumes t = (n, n). (Right?). That should be fine.

When I omit bsize, and mpirun -n 3, A.owner_range returns

(4, 7)
(0, 4)
(7, 10)

Which essentially is the same partioning that I manually set (I need to set partioning manually). Also the sum of bsizes equals the dimension. What is the problem here?

Thanks,
Florian

From knepley at gmail.com  Thu Mar 26 09:08:51 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 09:08:51 -0500
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <2259494.rATCdaqx1O@asaru>
References: <2259494.rATCdaqx1O@asaru>
Message-ID: <CAMYG4Gkt=i2=Yv4nrUQCuDFeR5FBn7OxM4VKbimiKE8oyaKbkg@mail.gmail.com>

On Thu, Mar 26, 2015 at 8:16 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Hello,
>
> I'm using petsc with petsc4py.
>
> A matrix is created like that
>
>     MPIrank = MPI.COMM_WORLD.Get_rank()
>     MPIsize = MPI.COMM_WORLD.Get_size()
>     print("MPI Rank = ", MPIrank)
>     print("MPI Size = ", MPIsize)
>     parts = partitions()
>
>     print("Dimension= ", nSupport + dimension, "bsize = ",
> len(parts[MPIrank]))
>
>     MPI.COMM_WORLD.Barrier() # Just to keep the output together
>     A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport +
> dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
>
>
> Output for mpirun -n 2 is like that and it works:
>
> MPI Rank =  0
> MPI Size =  2
> Dimension=  10 bsize =  5
> MPI Rank =  1
> MPI Size =  2
> Dimension=  10 bsize =  5
>
> But for mpirun -n 3 it crashes:
>
> MPI Rank =  2
> MPI Size =  3
> Dimension=  10 bsize =  3
> MPI Rank =  1
> MPI Size =  3
> Dimension=  10 bsize =  4
> MPI Rank =  0
> MPI Size =  3
> Dimension=  10 bsize =  3
>
> Error is ValueError: global size 10 not divisible by block size 3.
>
> I tried to dig a bit into the source. Mat_Create uses Mat_Dense to unpack
> the size and bsize parameters. When there just a scalar n, not a tuple t
> given as parameter it assumes t = (n, n). (Right?). That should be fine.
>
> When I omit bsize, and mpirun -n 3, A.owner_range returns
>
> (4, 7)
> (0, 4)
> (7, 10)
>
> Which essentially is the same partioning that I manually set (I need to
> set partioning manually). Also the sum of bsizes equals the dimension. What
> is the problem here?
>

This looks to me like the lack of an error check. A global size not
divisible by the block size makes no sense at all.

  Thanks,

     Matt


> Thanks,
> Florian
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/aaf4a327/attachment.html>

From bhatiamanav at gmail.com  Thu Mar 26 09:21:23 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 26 Mar 2015 09:21:23 -0500
Subject: [petsc-users] KSP linear solver returns inf
Message-ID: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>

Hi, 

  I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration. 

  The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result. 

   Is there a reason why the solver would return after the first iteration with an inf? 
 
   I am not sure on where to start debugging this case, so I would appreciate any pointers. 

Thanks,
Manav



From knepley at gmail.com  Thu Mar 26 09:34:27 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 09:34:27 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
Message-ID: <CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>

On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Hi,
>
>   I am using the KSP linear solver for my system of equations, without any
> command line options at this point. I have checked that the L1 norms of my
> system matrix and the force vector are finite values, but the KSP solver is
> returning with an ?inf? residual in the very first iteration.
>
>   The problem has 6.7M dofs and I have tried this on multiple machines
> with different number of nodes with the same result.
>
>    Is there a reason why the solver would return after the first iteration
> with an inf?
>
>    I am not sure on where to start debugging this case, so I would
> appreciate any pointers.
>

For all solver questions, we want to see the output of

  -ksp_view -ksp_monitor_true_residual -ksp_converged_reason

The problem here would be that there is an error, so we would never see the
output
of -ksp_view and know what solver you are using. If you are using something
complex,
can you try using

  -pc_type jacobi

and send the output from the options above? Then we can figure out why the
other solver
gets an inf.

  Thanks,

     Matt


> Thanks,
> Manav
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/dd1c492a/attachment.html>

From bhatiamanav at gmail.com  Thu Mar 26 10:10:40 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 26 Mar 2015 10:10:40 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
Message-ID: <8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>

Thanks, Matt. 

Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason

  0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
Linear solve did not converge due to DIVERGED_NANORINF iterations 0
KSP Object: 12 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1000
  tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 12 MPI processes
  type: bjacobi
    block Jacobi: number of blocks = 12
    Local solve is same for all blocks, in the following KSP and PC objects:
  KSP Object:  (sub_)   1 MPI processes
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
    left preconditioning
    using NONE norm type for convergence test
  PC Object:  (sub_)   1 MPI processes
    type: ilu
      ILU: out-of-place factorization
      0 levels of fill
      tolerance for zero pivot 2.22045e-14
      using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
      matrix ordering: natural
      factor fill ratio given 1, needed 1
        Factored matrix follows:
          Mat Object:           1 MPI processes
            type: seqaij
            rows=667070, cols=667070
            package used to perform factorization: petsc
            total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 133414 nodes, limit used is 5
    linear system matrix = precond matrix:
    Mat Object:    ()     1 MPI processes
      type: seqaij
      rows=667070, cols=667070
      total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
      total number of mallocs used during MatSetValues calls =0
        using I-node routines: found 133414 nodes, limit used is 5
  linear system matrix = precond matrix:
  Mat Object:  ()   12 MPI processes
    type: mpiaij
    rows=6723030, cols=6723030
    total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 133414 nodes, limit used is 5


  Anything jumps out at you as odd? 

-Manav



> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
> Hi,
> 
>   I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
> 
>   The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
> 
>    Is there a reason why the solver would return after the first iteration with an inf?
> 
>    I am not sure on where to start debugging this case, so I would appreciate any pointers.
> 
> For all solver questions, we want to see the output of
> 
>   -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
> 
> The problem here would be that there is an error, so we would never see the output
> of -ksp_view and know what solver you are using. If you are using something complex,
> can you try using
> 
>   -pc_type jacobi
> 
> and send the output from the options above? Then we can figure out why the other solver
> gets an inf.
> 
>   Thanks,
> 
>      Matt
>  
> Thanks,
> Manav
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/fc7027fb/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar 26 10:17:10 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Mar 2015 10:17:10 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
Message-ID: <15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>


 The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 

  Is this a Stokes-like problem?

  Barry

> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Thanks, Matt. 
> 
> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
> 
>   0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
> KSP Object: 12 MPI processes
>   type: gmres
>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=1000
>   tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>   left preconditioning
>   using nonzero initial guess
>   using PRECONDITIONED norm type for convergence test
> PC Object: 12 MPI processes
>   type: bjacobi
>     block Jacobi: number of blocks = 12
>     Local solve is same for all blocks, in the following KSP and PC objects:
>   KSP Object:  (sub_)   1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>     left preconditioning
>     using NONE norm type for convergence test
>   PC Object:  (sub_)   1 MPI processes
>     type: ilu
>       ILU: out-of-place factorization
>       0 levels of fill
>       tolerance for zero pivot 2.22045e-14
>       using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>       matrix ordering: natural
>       factor fill ratio given 1, needed 1
>         Factored matrix follows:
>           Mat Object:           1 MPI processes
>             type: seqaij
>             rows=667070, cols=667070
>             package used to perform factorization: petsc
>             total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>             total number of mallocs used during MatSetValues calls =0
>               using I-node routines: found 133414 nodes, limit used is 5
>     linear system matrix = precond matrix:
>     Mat Object:    ()     1 MPI processes
>       type: seqaij
>       rows=667070, cols=667070
>       total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>       total number of mallocs used during MatSetValues calls =0
>         using I-node routines: found 133414 nodes, limit used is 5
>   linear system matrix = precond matrix:
>   Mat Object:  ()   12 MPI processes
>     type: mpiaij
>     rows=6723030, cols=6723030
>     total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>     total number of mallocs used during MatSetValues calls =0
>       using I-node (on process 0) routines: found 133414 nodes, limit used is 5
> 
> 
>   Anything jumps out at you as odd? 
> 
> -Manav
> 
> 
> 
>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> Hi,
>> 
>>   I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>> 
>>   The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>> 
>>    Is there a reason why the solver would return after the first iteration with an inf?
>> 
>>    I am not sure on where to start debugging this case, so I would appreciate any pointers.
>> 
>> For all solver questions, we want to see the output of
>> 
>>   -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>> 
>> The problem here would be that there is an error, so we would never see the output
>> of -ksp_view and know what solver you are using. If you are using something complex,
>> can you try using
>> 
>>   -pc_type jacobi
>> 
>> and send the output from the options above? Then we can figure out why the other solver
>> gets an inf.
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> Thanks,
>> Manav
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From bhatiamanav at gmail.com  Thu Mar 26 10:19:26 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 26 Mar 2015 10:19:26 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
Message-ID: <259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>

Thanks, Barry. I will try that. 

This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 

-Manav


> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
> 
>  Is this a Stokes-like problem?
> 
>  Barry
> 
>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Thanks, Matt. 
>> 
>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>> 
>>  0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>> KSP Object: 12 MPI processes
>>  type: gmres
>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    GMRES: happy breakdown tolerance 1e-30
>>  maximum iterations=1000
>>  tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>  left preconditioning
>>  using nonzero initial guess
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 12 MPI processes
>>  type: bjacobi
>>    block Jacobi: number of blocks = 12
>>    Local solve is same for all blocks, in the following KSP and PC objects:
>>  KSP Object:  (sub_)   1 MPI processes
>>    type: preonly
>>    maximum iterations=10000, initial guess is zero
>>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>    left preconditioning
>>    using NONE norm type for convergence test
>>  PC Object:  (sub_)   1 MPI processes
>>    type: ilu
>>      ILU: out-of-place factorization
>>      0 levels of fill
>>      tolerance for zero pivot 2.22045e-14
>>      using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>      matrix ordering: natural
>>      factor fill ratio given 1, needed 1
>>        Factored matrix follows:
>>          Mat Object:           1 MPI processes
>>            type: seqaij
>>            rows=667070, cols=667070
>>            package used to perform factorization: petsc
>>            total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>            total number of mallocs used during MatSetValues calls =0
>>              using I-node routines: found 133414 nodes, limit used is 5
>>    linear system matrix = precond matrix:
>>    Mat Object:    ()     1 MPI processes
>>      type: seqaij
>>      rows=667070, cols=667070
>>      total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>      total number of mallocs used during MatSetValues calls =0
>>        using I-node routines: found 133414 nodes, limit used is 5
>>  linear system matrix = precond matrix:
>>  Mat Object:  ()   12 MPI processes
>>    type: mpiaij
>>    rows=6723030, cols=6723030
>>    total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>    total number of mallocs used during MatSetValues calls =0
>>      using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>> 
>> 
>>  Anything jumps out at you as odd? 
>> 
>> -Manav
>> 
>> 
>> 
>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> Hi,
>>> 
>>>  I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>> 
>>>  The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>> 
>>>   Is there a reason why the solver would return after the first iteration with an inf?
>>> 
>>>   I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>> 
>>> For all solver questions, we want to see the output of
>>> 
>>>  -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>> 
>>> The problem here would be that there is an error, so we would never see the output
>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>> can you try using
>>> 
>>>  -pc_type jacobi
>>> 
>>> and send the output from the options above? Then we can figure out why the other solver
>>> gets an inf.
>>> 
>>>  Thanks,
>>> 
>>>     Matt
>>> 
>>> Thanks,
>>> Manav
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 


From bsmith at mcs.anl.gov  Thu Mar 26 10:35:27 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Mar 2015 10:35:27 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
Message-ID: <B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>


> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Thanks, Barry. I will try that. 
> 
> This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 

  Yes, ILU could easily fail on this and really isn't appropriate.  Likely you should be using PCFIELDSPLIT for preconditioning.

  Barry

> 
> -Manav
> 
> 
>> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
>> 
>> Is this a Stokes-like problem?
>> 
>> Barry
>> 
>>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> 
>>> Thanks, Matt. 
>>> 
>>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>> 
>>> 0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>>> KSP Object: 12 MPI processes
>>> type: gmres
>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>   GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=1000
>>> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using nonzero initial guess
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 12 MPI processes
>>> type: bjacobi
>>>   block Jacobi: number of blocks = 12
>>>   Local solve is same for all blocks, in the following KSP and PC objects:
>>> KSP Object:  (sub_)   1 MPI processes
>>>   type: preonly
>>>   maximum iterations=10000, initial guess is zero
>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>   left preconditioning
>>>   using NONE norm type for convergence test
>>> PC Object:  (sub_)   1 MPI processes
>>>   type: ilu
>>>     ILU: out-of-place factorization
>>>     0 levels of fill
>>>     tolerance for zero pivot 2.22045e-14
>>>     using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>     matrix ordering: natural
>>>     factor fill ratio given 1, needed 1
>>>       Factored matrix follows:
>>>         Mat Object:           1 MPI processes
>>>           type: seqaij
>>>           rows=667070, cols=667070
>>>           package used to perform factorization: petsc
>>>           total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>>           total number of mallocs used during MatSetValues calls =0
>>>             using I-node routines: found 133414 nodes, limit used is 5
>>>   linear system matrix = precond matrix:
>>>   Mat Object:    ()     1 MPI processes
>>>     type: seqaij
>>>     rows=667070, cols=667070
>>>     total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>>     total number of mallocs used during MatSetValues calls =0
>>>       using I-node routines: found 133414 nodes, limit used is 5
>>> linear system matrix = precond matrix:
>>> Mat Object:  ()   12 MPI processes
>>>   type: mpiaij
>>>   rows=6723030, cols=6723030
>>>   total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>>   total number of mallocs used during MatSetValues calls =0
>>>     using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>>> 
>>> 
>>> Anything jumps out at you as odd? 
>>> 
>>> -Manav
>>> 
>>> 
>>> 
>>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>> 
>>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>>> 
>>>> The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>>> 
>>>>  Is there a reason why the solver would return after the first iteration with an inf?
>>>> 
>>>>  I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>>> 
>>>> For all solver questions, we want to see the output of
>>>> 
>>>> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>> 
>>>> The problem here would be that there is an error, so we would never see the output
>>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>>> can you try using
>>>> 
>>>> -pc_type jacobi
>>>> 
>>>> and send the output from the options above? Then we can figure out why the other solver
>>>> gets an inf.
>>>> 
>>>> Thanks,
>>>> 
>>>>    Matt
>>>> 
>>>> Thanks,
>>>> Manav
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 


From amesga1 at tigers.lsu.edu  Thu Mar 26 10:36:57 2015
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Thu, 26 Mar 2015 10:36:57 -0500
Subject: [petsc-users] getting a field's Vec from a DMPlex
Message-ID: <CAC+VmGdiUwwqq4YSJs4n5soYNHFF4qEVhasD=tL9VUTZY6B0iw@mail.gmail.com>

Dear all,

I was wondering if someone can tell me how you get a field's Vec (subVec of
the global Vec) from a distributed DMPlex. It seems that DMCreateFieldIS
works for sequential DMPlex but gives incorrect values for a distributed
DMPlex.

Many thanks,
Ata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/96c364d8/attachment.html>

From knepley at gmail.com  Thu Mar 26 10:49:23 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 10:49:23 -0500
Subject: [petsc-users] getting a field's Vec from a DMPlex
In-Reply-To: <CAC+VmGdiUwwqq4YSJs4n5soYNHFF4qEVhasD=tL9VUTZY6B0iw@mail.gmail.com>
References: <CAC+VmGdiUwwqq4YSJs4n5soYNHFF4qEVhasD=tL9VUTZY6B0iw@mail.gmail.com>
Message-ID: <CAMYG4Gm5Ohgkr_E6gWQ2J77WF+dH-bLwesZfD4uuHzJsRCNVag@mail.gmail.com>

On Thu, Mar 26, 2015 at 10:36 AM, Ataollah Mesgarnejad <
amesga1 at tigers.lsu.edu> wrote:

> Dear all,
>
> I was wondering if someone can tell me how you get a field's Vec (subVec
> of the global Vec) from a distributed DMPlex. It seems that DMCreateFieldIS
> works for sequential DMPlex but gives incorrect values for a distributed
> DMPlex.
>

It seems like what you want is:


https://bitbucket.org/petsc/petsc/src/fdac1df83b6d73782209c263cf9d6b0b33600136/src/vec/vec/utils/vsection.c?at=master#cl-229

and also it seems like DMCreateFieldIS() is broken somehow. We should fix
it. It currently passes all the tests for the FieldSplit
solvers, so I am not sure what is going wrong here.

  Thanks,

     Matt


> Many thanks,
> Ata
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/2985ef8c/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar 26 11:14:59 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Mar 2015 11:14:59 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
	<B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>
	<D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
Message-ID: <001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>


> On Mar 26, 2015, at 10:51 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Barry, 
> 
>   On a related note, I have another elasticity problem that I am trying to solver with HEX8 elements. It is an isotropic solid structure. Do you have a recommended preconditioned for this problem? 

   Yes, this one clearly requires PCGAMG. Make sure you read all the docs on PCGMAG, you will need to supply either the coordinates with PCSetCoordiantes() or the near null space of the operator. Unfortunately our documentation for this sucks and everyone refuses to improve it.

  Barry
 


> 
>    The problem has about 770K dofs and the default ILU(0) has not done well. I have also tried ILU(1), and that too has been unhelpful. I am observing stagnation of residuals after a drop of couple of magnitudes. 
> 
>    Any recommendations would be greatly appreciated. 
> 
> Thanks,
> Manav
> 
> 
> 
> 
> 
> 
>> On Mar 26, 2015, at 10:35 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> 
>>> Thanks, Barry. I will try that. 
>>> 
>>> This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 
>> 
>> Yes, ILU could easily fail on this and really isn't appropriate.  Likely you should be using PCFIELDSPLIT for preconditioning.
>> 
>> Barry
>> 
>>> 
>>> -Manav
>>> 
>>> 
>>>> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
>>>> 
>>>> Is this a Stokes-like problem?
>>>> 
>>>> Barry
>>>> 
>>>>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>> 
>>>>> Thanks, Matt. 
>>>>> 
>>>>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>> 
>>>>> 0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>>>>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>>>>> KSP Object: 12 MPI processes
>>>>> type: gmres
>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>> maximum iterations=1000
>>>>> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>>>> left preconditioning
>>>>> using nonzero initial guess
>>>>> using PRECONDITIONED norm type for convergence test
>>>>> PC Object: 12 MPI processes
>>>>> type: bjacobi
>>>>> block Jacobi: number of blocks = 12
>>>>> Local solve is same for all blocks, in the following KSP and PC objects:
>>>>> KSP Object:  (sub_)   1 MPI processes
>>>>> type: preonly
>>>>> maximum iterations=10000, initial guess is zero
>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>> left preconditioning
>>>>> using NONE norm type for convergence test
>>>>> PC Object:  (sub_)   1 MPI processes
>>>>> type: ilu
>>>>>   ILU: out-of-place factorization
>>>>>   0 levels of fill
>>>>>   tolerance for zero pivot 2.22045e-14
>>>>>   using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>>>   matrix ordering: natural
>>>>>   factor fill ratio given 1, needed 1
>>>>>     Factored matrix follows:
>>>>>       Mat Object:           1 MPI processes
>>>>>         type: seqaij
>>>>>         rows=667070, cols=667070
>>>>>         package used to perform factorization: petsc
>>>>>         total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>>>>         total number of mallocs used during MatSetValues calls =0
>>>>>           using I-node routines: found 133414 nodes, limit used is 5
>>>>> linear system matrix = precond matrix:
>>>>> Mat Object:    ()     1 MPI processes
>>>>>   type: seqaij
>>>>>   rows=667070, cols=667070
>>>>>   total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>>>>   total number of mallocs used during MatSetValues calls =0
>>>>>     using I-node routines: found 133414 nodes, limit used is 5
>>>>> linear system matrix = precond matrix:
>>>>> Mat Object:  ()   12 MPI processes
>>>>> type: mpiaij
>>>>> rows=6723030, cols=6723030
>>>>> total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>   using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>>>>> 
>>>>> 
>>>>> Anything jumps out at you as odd? 
>>>>> 
>>>>> -Manav
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>> 
>>>>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>>>>> 
>>>>>> The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>>>>> 
>>>>>> Is there a reason why the solver would return after the first iteration with an inf?
>>>>>> 
>>>>>> I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>>>>> 
>>>>>> For all solver questions, we want to see the output of
>>>>>> 
>>>>>> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>> 
>>>>>> The problem here would be that there is an error, so we would never see the output
>>>>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>>>>> can you try using
>>>>>> 
>>>>>> -pc_type jacobi
>>>>>> 
>>>>>> and send the output from the options above? Then we can figure out why the other solver
>>>>>> gets an inf.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>>  Matt
>>>>>> 
>>>>>> Thanks,
>>>>>> Manav
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>> 
>>> 
>> 
> 


From bhatiamanav at gmail.com  Thu Mar 26 11:20:03 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 26 Mar 2015 11:20:03 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
	<B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>
	<D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
	<001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>
Message-ID: <987432D4-8D31-40AC-A9AA-7C4E4DED7965@gmail.com>

Thanks. 
Quick question (out of ignorance): does it matter that the HEX8 may still be arranged in an unstructured fashion? Meaning, that although I use brick elements, my grid does not have a structured grid appearance.

-Manav


> On Mar 26, 2015, at 11:14 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>> On Mar 26, 2015, at 10:51 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Barry, 
>> 
>>  On a related note, I have another elasticity problem that I am trying to solver with HEX8 elements. It is an isotropic solid structure. Do you have a recommended preconditioned for this problem? 
> 
>   Yes, this one clearly requires PCGAMG. Make sure you read all the docs on PCGMAG, you will need to supply either the coordinates with PCSetCoordiantes() or the near null space of the operator. Unfortunately our documentation for this sucks and everyone refuses to improve it.
> 
>  Barry
> 
> 
> 
>> 
>>   The problem has about 770K dofs and the default ILU(0) has not done well. I have also tried ILU(1), and that too has been unhelpful. I am observing stagnation of residuals after a drop of couple of magnitudes. 
>> 
>>   Any recommendations would be greatly appreciated. 
>> 
>> Thanks,
>> Manav
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Mar 26, 2015, at 10:35 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>> 
>>>> Thanks, Barry. I will try that. 
>>>> 
>>>> This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 
>>> 
>>> Yes, ILU could easily fail on this and really isn't appropriate.  Likely you should be using PCFIELDSPLIT for preconditioning.
>>> 
>>> Barry
>>> 
>>>> 
>>>> -Manav
>>>> 
>>>> 
>>>>> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
>>>>> 
>>>>> Is this a Stokes-like problem?
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>> 
>>>>>> Thanks, Matt. 
>>>>>> 
>>>>>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>> 
>>>>>> 0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>>>>>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>>>>>> KSP Object: 12 MPI processes
>>>>>> type: gmres
>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>> maximum iterations=1000
>>>>>> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>>>>> left preconditioning
>>>>>> using nonzero initial guess
>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>> PC Object: 12 MPI processes
>>>>>> type: bjacobi
>>>>>> block Jacobi: number of blocks = 12
>>>>>> Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>> KSP Object:  (sub_)   1 MPI processes
>>>>>> type: preonly
>>>>>> maximum iterations=10000, initial guess is zero
>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>> left preconditioning
>>>>>> using NONE norm type for convergence test
>>>>>> PC Object:  (sub_)   1 MPI processes
>>>>>> type: ilu
>>>>>>  ILU: out-of-place factorization
>>>>>>  0 levels of fill
>>>>>>  tolerance for zero pivot 2.22045e-14
>>>>>>  using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>>>>  matrix ordering: natural
>>>>>>  factor fill ratio given 1, needed 1
>>>>>>    Factored matrix follows:
>>>>>>      Mat Object:           1 MPI processes
>>>>>>        type: seqaij
>>>>>>        rows=667070, cols=667070
>>>>>>        package used to perform factorization: petsc
>>>>>>        total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>>>>>        total number of mallocs used during MatSetValues calls =0
>>>>>>          using I-node routines: found 133414 nodes, limit used is 5
>>>>>> linear system matrix = precond matrix:
>>>>>> Mat Object:    ()     1 MPI processes
>>>>>>  type: seqaij
>>>>>>  rows=667070, cols=667070
>>>>>>  total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>>>>>  total number of mallocs used during MatSetValues calls =0
>>>>>>    using I-node routines: found 133414 nodes, limit used is 5
>>>>>> linear system matrix = precond matrix:
>>>>>> Mat Object:  ()   12 MPI processes
>>>>>> type: mpiaij
>>>>>> rows=6723030, cols=6723030
>>>>>> total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>  using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>>>>>> 
>>>>>> 
>>>>>> Anything jumps out at you as odd? 
>>>>>> 
>>>>>> -Manav
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>> 
>>>>>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>>>>>> 
>>>>>>> The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>>>>>> 
>>>>>>> Is there a reason why the solver would return after the first iteration with an inf?
>>>>>>> 
>>>>>>> I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>>>>>> 
>>>>>>> For all solver questions, we want to see the output of
>>>>>>> 
>>>>>>> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>> 
>>>>>>> The problem here would be that there is an error, so we would never see the output
>>>>>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>>>>>> can you try using
>>>>>>> 
>>>>>>> -pc_type jacobi
>>>>>>> 
>>>>>>> and send the output from the options above? Then we can figure out why the other solver
>>>>>>> gets an inf.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Matt
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Manav
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Thu Mar 26 11:28:31 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Mar 2015 11:28:31 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <987432D4-8D31-40AC-A9AA-7C4E4DED7965@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
	<B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>
	<D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
	<001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>
	<987432D4-8D31-40AC-A9AA-7C4E4DED7965@gmail.com>
Message-ID: <C725B83B-5E0B-4703-80B8-030707BD4336@mcs.anl.gov>


   This is fine. PCGAMG does algebraic multigrid so the mesh doesn't matter in its use.

  Barry

> On Mar 26, 2015, at 11:20 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Thanks. 
> Quick question (out of ignorance): does it matter that the HEX8 may still be arranged in an unstructured fashion? Meaning, that although I use brick elements, my grid does not have a structured grid appearance.
> 
> -Manav
> 
> 
>> On Mar 26, 2015, at 11:14 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>> On Mar 26, 2015, at 10:51 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> 
>>> Barry, 
>>> 
>>> On a related note, I have another elasticity problem that I am trying to solver with HEX8 elements. It is an isotropic solid structure. Do you have a recommended preconditioned for this problem? 
>> 
>>  Yes, this one clearly requires PCGAMG. Make sure you read all the docs on PCGMAG, you will need to supply either the coordinates with PCSetCoordiantes() or the near null space of the operator. Unfortunately our documentation for this sucks and everyone refuses to improve it.
>> 
>> Barry
>> 
>> 
>> 
>>> 
>>>  The problem has about 770K dofs and the default ILU(0) has not done well. I have also tried ILU(1), and that too has been unhelpful. I am observing stagnation of residuals after a drop of couple of magnitudes. 
>>> 
>>>  Any recommendations would be greatly appreciated. 
>>> 
>>> Thanks,
>>> Manav
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On Mar 26, 2015, at 10:35 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>>> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>> 
>>>>> Thanks, Barry. I will try that. 
>>>>> 
>>>>> This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 
>>>> 
>>>> Yes, ILU could easily fail on this and really isn't appropriate.  Likely you should be using PCFIELDSPLIT for preconditioning.
>>>> 
>>>> Barry
>>>> 
>>>>> 
>>>>> -Manav
>>>>> 
>>>>> 
>>>>>> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> 
>>>>>> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
>>>>>> 
>>>>>> Is this a Stokes-like problem?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>> 
>>>>>>> Thanks, Matt. 
>>>>>>> 
>>>>>>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>> 
>>>>>>> 0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>>>>>>> KSP Object: 12 MPI processes
>>>>>>> type: gmres
>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>> maximum iterations=1000
>>>>>>> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>>>>>> left preconditioning
>>>>>>> using nonzero initial guess
>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>> PC Object: 12 MPI processes
>>>>>>> type: bjacobi
>>>>>>> block Jacobi: number of blocks = 12
>>>>>>> Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>>> KSP Object:  (sub_)   1 MPI processes
>>>>>>> type: preonly
>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>> left preconditioning
>>>>>>> using NONE norm type for convergence test
>>>>>>> PC Object:  (sub_)   1 MPI processes
>>>>>>> type: ilu
>>>>>>> ILU: out-of-place factorization
>>>>>>> 0 levels of fill
>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>>>>> matrix ordering: natural
>>>>>>> factor fill ratio given 1, needed 1
>>>>>>>   Factored matrix follows:
>>>>>>>     Mat Object:           1 MPI processes
>>>>>>>       type: seqaij
>>>>>>>       rows=667070, cols=667070
>>>>>>>       package used to perform factorization: petsc
>>>>>>>       total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>>>>>>       total number of mallocs used during MatSetValues calls =0
>>>>>>>         using I-node routines: found 133414 nodes, limit used is 5
>>>>>>> linear system matrix = precond matrix:
>>>>>>> Mat Object:    ()     1 MPI processes
>>>>>>> type: seqaij
>>>>>>> rows=667070, cols=667070
>>>>>>> total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>   using I-node routines: found 133414 nodes, limit used is 5
>>>>>>> linear system matrix = precond matrix:
>>>>>>> Mat Object:  ()   12 MPI processes
>>>>>>> type: mpiaij
>>>>>>> rows=6723030, cols=6723030
>>>>>>> total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>> using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>>>>>>> 
>>>>>>> 
>>>>>>> Anything jumps out at you as odd? 
>>>>>>> 
>>>>>>> -Manav
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>>>>>>> 
>>>>>>>> The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>>>>>>> 
>>>>>>>> Is there a reason why the solver would return after the first iteration with an inf?
>>>>>>>> 
>>>>>>>> I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>>>>>>> 
>>>>>>>> For all solver questions, we want to see the output of
>>>>>>>> 
>>>>>>>> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>>> 
>>>>>>>> The problem here would be that there is an error, so we would never see the output
>>>>>>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>>>>>>> can you try using
>>>>>>>> 
>>>>>>>> -pc_type jacobi
>>>>>>>> 
>>>>>>>> and send the output from the options above? Then we can figure out why the other solver
>>>>>>>> gets an inf.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Matt
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Manav
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From bhatiamanav at gmail.com  Thu Mar 26 11:31:38 2015
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Thu, 26 Mar 2015 11:31:38 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <C725B83B-5E0B-4703-80B8-030707BD4336@mcs.anl.gov>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
	<B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>
	<D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
	<001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>
	<987432D4-8D31-40AC-A9AA-7C4E4DED7965@gmail.com>
	<C725B83B-5E0B-4703-80B8-030707BD4336@mcs.anl.gov>
Message-ID: <2721655E-6630-45C2-AFCD-E7A829144910@gmail.com>

Ok, so I ran my fluids problem with -pc_type jacobi. This time it did not return with ?inf?, but there was no convergence. 

Here is the first 10 iterations: 
  0 KSP preconditioned resid norm 7.840061446913e+07 true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 7.838673732620e+07 true resid norm 2.709686618724e+06 ||r(i)||/||b|| 1.000222716773e+00
  2 KSP preconditioned resid norm 7.838673500462e+07 true resid norm 2.709685645990e+06 ||r(i)||/||b|| 1.000222357709e+00
  3 KSP preconditioned resid norm 7.838080292706e+07 true resid norm 2.709691690751e+06 ||r(i)||/||b|| 1.000224589003e+00
  4 KSP preconditioned resid norm 7.837871329477e+07 true resid norm 2.709550983943e+06 ||r(i)||/||b|| 1.000172650101e+00
  5 KSP preconditioned resid norm 7.837788814104e+07 true resid norm 2.709528734769e+06 ||r(i)||/||b|| 1.000164437296e+00
  6 KSP preconditioned resid norm 7.837690087318e+07 true resid norm 2.709494763671e+06 ||r(i)||/||b|| 1.000151897594e+00
  7 KSP preconditioned resid norm 7.836061163366e+07 true resid norm 2.709622438419e+06 ||r(i)||/||b|| 1.000199025989e+00
  8 KSP preconditioned resid norm 7.835981946243e+07 true resid norm 2.709587889985e+06 ||r(i)||/||b|| 1.000186273176e+00
  9 KSP preconditioned resid norm 7.828072351639e+07 true resid norm 2.710062032107e+06 ||r(i)||/||b|| 1.000361292574e+00
 10 KSP preconditioned resid norm 7.828054329393e+07 true resid norm 2.710056988152e+06 ||r(i)||/||b|| 1.000359430706e+00

and here are the last 10 iterations: 

991 KSP preconditioned resid norm 7.801323853477e+07 true resid norm 2.710182819026e+06 ||r(i)||/||b|| 1.000405878475e+00
992 KSP preconditioned resid norm 7.801323853477e+07 true resid norm 2.710182817684e+06 ||r(i)||/||b|| 1.000405877980e+00
993 KSP preconditioned resid norm 7.801323853476e+07 true resid norm 2.710182817272e+06 ||r(i)||/||b|| 1.000405877828e+00
994 KSP preconditioned resid norm 7.801323853470e+07 true resid norm 2.710182825637e+06 ||r(i)||/||b|| 1.000405880916e+00
995 KSP preconditioned resid norm 7.801323853470e+07 true resid norm 2.710182825705e+06 ||r(i)||/||b|| 1.000405880941e+00
996 KSP preconditioned resid norm 7.801323853470e+07 true resid norm 2.710182826222e+06 ||r(i)||/||b|| 1.000405881131e+00
997 KSP preconditioned resid norm 7.801323853469e+07 true resid norm 2.710182826506e+06 ||r(i)||/||b|| 1.000405881236e+00
998 KSP preconditioned resid norm 7.801323853424e+07 true resid norm 2.710182832779e+06 ||r(i)||/||b|| 1.000405883552e+00
999 KSP preconditioned resid norm 7.801323852813e+07 true resid norm 2.710182815446e+06 ||r(i)||/||b|| 1.000405877154e+00
1000 KSP preconditioned resid norm 7.801323817948e+07 true resid norm 2.710182749096e+06 ||r(i)||/||b|| 1.000405852662e+00

Any recommendations? 

Thanks,
Manav


> On Mar 26, 2015, at 11:28 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> 
>   This is fine. PCGAMG does algebraic multigrid so the mesh doesn't matter in its use.
> 
>  Barry
> 
>> On Mar 26, 2015, at 11:20 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Thanks. 
>> Quick question (out of ignorance): does it matter that the HEX8 may still be arranged in an unstructured fashion? Meaning, that although I use brick elements, my grid does not have a structured grid appearance.
>> 
>> -Manav
>> 
>> 
>>> On Mar 26, 2015, at 11:14 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>>> On Mar 26, 2015, at 10:51 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>> 
>>>> Barry, 
>>>> 
>>>> On a related note, I have another elasticity problem that I am trying to solver with HEX8 elements. It is an isotropic solid structure. Do you have a recommended preconditioned for this problem? 
>>> 
>>> Yes, this one clearly requires PCGAMG. Make sure you read all the docs on PCGMAG, you will need to supply either the coordinates with PCSetCoordiantes() or the near null space of the operator. Unfortunately our documentation for this sucks and everyone refuses to improve it.
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>>> 
>>>> The problem has about 770K dofs and the default ILU(0) has not done well. I have also tried ILU(1), and that too has been unhelpful. I am observing stagnation of residuals after a drop of couple of magnitudes. 
>>>> 
>>>> Any recommendations would be greatly appreciated. 
>>>> 
>>>> Thanks,
>>>> Manav
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Mar 26, 2015, at 10:35 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>>> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>> 
>>>>>> Thanks, Barry. I will try that. 
>>>>>> 
>>>>>> This is Euler flow equations discretized with SUPG. The mesh is made of 4-noded tetrahedra. The flow parameters correspond to transonic flow. 
>>>>> 
>>>>> Yes, ILU could easily fail on this and really isn't appropriate.  Likely you should be using PCFIELDSPLIT for preconditioning.
>>>>> 
>>>>> Barry
>>>>> 
>>>>>> 
>>>>>> -Manav
>>>>>> 
>>>>>> 
>>>>>>> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> The default preconditioner with ILU(0) on each process is not appropriate for your problem and is producing overflow. Try -sub_pc_type lu and see if that produces a different result. 
>>>>>>> 
>>>>>>> Is this a Stokes-like problem?
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>>> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Thanks, Matt. 
>>>>>>>> 
>>>>>>>> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>>> 
>>>>>>>> 0 KSP preconditioned resid norm            inf true resid norm 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>>>>>>>> KSP Object: 12 MPI processes
>>>>>>>> type: gmres
>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>> maximum iterations=1000
>>>>>>>> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
>>>>>>>> left preconditioning
>>>>>>>> using nonzero initial guess
>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>> PC Object: 12 MPI processes
>>>>>>>> type: bjacobi
>>>>>>>> block Jacobi: number of blocks = 12
>>>>>>>> Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>>>> KSP Object:  (sub_)   1 MPI processes
>>>>>>>> type: preonly
>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>> left preconditioning
>>>>>>>> using NONE norm type for convergence test
>>>>>>>> PC Object:  (sub_)   1 MPI processes
>>>>>>>> type: ilu
>>>>>>>> ILU: out-of-place factorization
>>>>>>>> 0 levels of fill
>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>>>>>> matrix ordering: natural
>>>>>>>> factor fill ratio given 1, needed 1
>>>>>>>>  Factored matrix follows:
>>>>>>>>    Mat Object:           1 MPI processes
>>>>>>>>      type: seqaij
>>>>>>>>      rows=667070, cols=667070
>>>>>>>>      package used to perform factorization: petsc
>>>>>>>>      total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>>>>>>>>      total number of mallocs used during MatSetValues calls =0
>>>>>>>>        using I-node routines: found 133414 nodes, limit used is 5
>>>>>>>> linear system matrix = precond matrix:
>>>>>>>> Mat Object:    ()     1 MPI processes
>>>>>>>> type: seqaij
>>>>>>>> rows=667070, cols=667070
>>>>>>>> total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>  using I-node routines: found 133414 nodes, limit used is 5
>>>>>>>> linear system matrix = precond matrix:
>>>>>>>> Mat Object:  ()   12 MPI processes
>>>>>>>> type: mpiaij
>>>>>>>> rows=6723030, cols=6723030
>>>>>>>> total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>> using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Anything jumps out at you as odd? 
>>>>>>>> 
>>>>>>>> -Manav
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am using the KSP linear solver for my system of equations, without any command line options at this point. I have checked that the L1 norms of my system matrix and the force vector are finite values, but the KSP solver is returning with an ?inf? residual in the very first iteration.
>>>>>>>>> 
>>>>>>>>> The problem has 6.7M dofs and I have tried this on multiple machines with different number of nodes with the same result.
>>>>>>>>> 
>>>>>>>>> Is there a reason why the solver would return after the first iteration with an inf?
>>>>>>>>> 
>>>>>>>>> I am not sure on where to start debugging this case, so I would appreciate any pointers.
>>>>>>>>> 
>>>>>>>>> For all solver questions, we want to see the output of
>>>>>>>>> 
>>>>>>>>> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>>>> 
>>>>>>>>> The problem here would be that there is an error, so we would never see the output
>>>>>>>>> of -ksp_view and know what solver you are using. If you are using something complex,
>>>>>>>>> can you try using
>>>>>>>>> 
>>>>>>>>> -pc_type jacobi
>>>>>>>>> 
>>>>>>>>> and send the output from the options above? Then we can figure out why the other solver
>>>>>>>>> gets an inf.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Matt
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Manav
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/6395524a/attachment-0001.html>

From knepley at gmail.com  Thu Mar 26 11:40:53 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 11:40:53 -0500
Subject: [petsc-users] KSP linear solver returns inf
In-Reply-To: <2721655E-6630-45C2-AFCD-E7A829144910@gmail.com>
References: <B2A2A4D3-B4B3-4573-91AA-2733839A6A1D@gmail.com>
	<CAMYG4G=dJGtS1Nw8XK_cdZtrX1Jb4P2YYxoG4Oj-HyHOScjWiQ@mail.gmail.com>
	<8AA9D37E-F9D3-4D4D-AB82-5BCDE9C49E1D@gmail.com>
	<15F1A953-E8F5-45E2-B14A-EA65F2915C8D@mcs.anl.gov>
	<259844BB-08A2-40A4-A209-1C3F0C114463@gmail.com>
	<B2530DD8-614A-4AAF-9C2B-EF1378EA5DE1@mcs.anl.gov>
	<D1A8FB0F-AEEC-4B26-9C7A-D66D72E7275B@gmail.com>
	<001998A0-72E7-493F-9525-43552BC6244A@mcs.anl.gov>
	<987432D4-8D31-40AC-A9AA-7C4E4DED7965@gmail.com>
	<C725B83B-5E0B-4703-80B8-030707BD4336@mcs.anl.gov>
	<2721655E-6630-45C2-AFCD-E7A829144910@gmail.com>
Message-ID: <CAMYG4Gn8EsyZnMfLqVXVXp11qfxkNPeVjr3epU62EAadK1ohEA@mail.gmail.com>

On Thu, Mar 26, 2015 at 11:31 AM, Manav Bhatia <bhatiamanav at gmail.com>
wrote:

> Ok, so I ran my fluids problem with -pc_type jacobi. This time it did not
> return with ?inf?, but there was no convergence.
>

Good. This seems to confirm that ILU(0) was the problem.

It is not a surprise that Jacobi is not an effective preconditioner. The
second step that we always recommend is to look in
the literature and find some preconditioners that people say work for your
problem, or for something like it.

With PETSc, it is simple to reproduce preconditioners from the literature,
for example SIMPLE from Patankar, and then
optimize them for your problem.

  Thanks,

     Matt


> Here is the first 10 iterations:
>   0 KSP preconditioned resid norm 7.840061446913e+07 true resid norm
> 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.838673732620e+07 true resid norm
> 2.709686618724e+06 ||r(i)||/||b|| 1.000222716773e+00
>   2 KSP preconditioned resid norm 7.838673500462e+07 true resid norm
> 2.709685645990e+06 ||r(i)||/||b|| 1.000222357709e+00
>   3 KSP preconditioned resid norm 7.838080292706e+07 true resid norm
> 2.709691690751e+06 ||r(i)||/||b|| 1.000224589003e+00
>   4 KSP preconditioned resid norm 7.837871329477e+07 true resid norm
> 2.709550983943e+06 ||r(i)||/||b|| 1.000172650101e+00
>   5 KSP preconditioned resid norm 7.837788814104e+07 true resid norm
> 2.709528734769e+06 ||r(i)||/||b|| 1.000164437296e+00
>   6 KSP preconditioned resid norm 7.837690087318e+07 true resid norm
> 2.709494763671e+06 ||r(i)||/||b|| 1.000151897594e+00
>   7 KSP preconditioned resid norm 7.836061163366e+07 true resid norm
> 2.709622438419e+06 ||r(i)||/||b|| 1.000199025989e+00
>   8 KSP preconditioned resid norm 7.835981946243e+07 true resid norm
> 2.709587889985e+06 ||r(i)||/||b|| 1.000186273176e+00
>   9 KSP preconditioned resid norm 7.828072351639e+07 true resid norm
> 2.710062032107e+06 ||r(i)||/||b|| 1.000361292574e+00
>  10 KSP preconditioned resid norm 7.828054329393e+07 true resid norm
> 2.710056988152e+06 ||r(i)||/||b|| 1.000359430706e+00
>
> and here are the last 10 iterations:
>
> 991 KSP preconditioned resid norm 7.801323853477e+07 true resid norm
> 2.710182819026e+06 ||r(i)||/||b|| 1.000405878475e+00
> 992 KSP preconditioned resid norm 7.801323853477e+07 true resid norm
> 2.710182817684e+06 ||r(i)||/||b|| 1.000405877980e+00
> 993 KSP preconditioned resid norm 7.801323853476e+07 true resid norm
> 2.710182817272e+06 ||r(i)||/||b|| 1.000405877828e+00
> 994 KSP preconditioned resid norm 7.801323853470e+07 true resid norm
> 2.710182825637e+06 ||r(i)||/||b|| 1.000405880916e+00
> 995 KSP preconditioned resid norm 7.801323853470e+07 true resid norm
> 2.710182825705e+06 ||r(i)||/||b|| 1.000405880941e+00
> 996 KSP preconditioned resid norm 7.801323853470e+07 true resid norm
> 2.710182826222e+06 ||r(i)||/||b|| 1.000405881131e+00
> 997 KSP preconditioned resid norm 7.801323853469e+07 true resid norm
> 2.710182826506e+06 ||r(i)||/||b|| 1.000405881236e+00
> 998 KSP preconditioned resid norm 7.801323853424e+07 true resid norm
> 2.710182832779e+06 ||r(i)||/||b|| 1.000405883552e+00
> 999 KSP preconditioned resid norm 7.801323852813e+07 true resid norm
> 2.710182815446e+06 ||r(i)||/||b|| 1.000405877154e+00
> 1000 KSP preconditioned resid norm 7.801323817948e+07 true resid norm
> 2.710182749096e+06 ||r(i)||/||b|| 1.000405852662e+00
>
> Any recommendations?
>
> Thanks,
> Manav
>
>
> On Mar 26, 2015, at 11:28 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
>   This is fine. PCGAMG does algebraic multigrid so the mesh doesn't matter
> in its use.
>
>  Barry
>
> On Mar 26, 2015, at 11:20 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Thanks.
> Quick question (out of ignorance): does it matter that the HEX8 may still
> be arranged in an unstructured fashion? Meaning, that although I use brick
> elements, my grid does not have a structured grid appearance.
>
> -Manav
>
>
> On Mar 26, 2015, at 11:14 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Mar 26, 2015, at 10:51 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Barry,
>
> On a related note, I have another elasticity problem that I am trying to
> solver with HEX8 elements. It is an isotropic solid structure. Do you have
> a recommended preconditioned for this problem?
>
>
> Yes, this one clearly requires PCGAMG. Make sure you read all the docs on
> PCGMAG, you will need to supply either the coordinates with
> PCSetCoordiantes() or the near null space of the operator. Unfortunately
> our documentation for this sucks and everyone refuses to improve it.
>
> Barry
>
>
>
>
> The problem has about 770K dofs and the default ILU(0) has not done well.
> I have also tried ILU(1), and that too has been unhelpful. I am observing
> stagnation of residuals after a drop of couple of magnitudes.
>
> Any recommendations would be greatly appreciated.
>
> Thanks,
> Manav
>
>
>
>
>
>
> On Mar 26, 2015, at 10:35 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> On Mar 26, 2015, at 10:19 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Thanks, Barry. I will try that.
>
> This is Euler flow equations discretized with SUPG. The mesh is made of
> 4-noded tetrahedra. The flow parameters correspond to transonic flow.
>
>
> Yes, ILU could easily fail on this and really isn't appropriate.  Likely
> you should be using PCFIELDSPLIT for preconditioning.
>
> Barry
>
>
> -Manav
>
>
> On Mar 26, 2015, at 10:17 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> The default preconditioner with ILU(0) on each process is not appropriate
> for your problem and is producing overflow. Try -sub_pc_type lu and see if
> that produces a different result.
>
> Is this a Stokes-like problem?
>
> Barry
>
> On Mar 26, 2015, at 10:10 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Thanks, Matt.
>
> Following is the output with: -ksp_monitor_lg_residualnorm -ksp_log
> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>
> 0 KSP preconditioned resid norm            inf true resid norm
> 2.709083260443e+06 ||r(i)||/||b|| 1.000000000000e+00
> Linear solve did not converge due to DIVERGED_NANORINF iterations 0
> KSP Object: 12 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1000
> tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
> left preconditioning
> using nonzero initial guess
> using PRECONDITIONED norm type for convergence test
> PC Object: 12 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 12
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object:  (sub_)   1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object:  (sub_)   1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 2.22045e-14
> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
> matrix ordering: natural
> factor fill ratio given 1, needed 1
>  Factored matrix follows:
>    Mat Object:           1 MPI processes
>      type: seqaij
>      rows=667070, cols=667070
>      package used to perform factorization: petsc
>      total: nonzeros=4.6765e+07, allocated nonzeros=4.6765e+07
>      total number of mallocs used during MatSetValues calls =0
>        using I-node routines: found 133414 nodes, limit used is 5
> linear system matrix = precond matrix:
> Mat Object:    ()     1 MPI processes
> type: seqaij
> rows=667070, cols=667070
> total: nonzeros=4.6765e+07, allocated nonzeros=5.473e+07
> total number of mallocs used during MatSetValues calls =0
>  using I-node routines: found 133414 nodes, limit used is 5
> linear system matrix = precond matrix:
> Mat Object:  ()   12 MPI processes
> type: mpiaij
> rows=6723030, cols=6723030
> total: nonzeros=4.98852e+08, allocated nonzeros=5.38983e+08
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 133414 nodes, limit used is 5
>
>
> Anything jumps out at you as odd?
>
> -Manav
>
>
>
> On Mar 26, 2015, at 9:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Mar 26, 2015 at 9:21 AM, Manav Bhatia <bhatiamanav at gmail.com>
> wrote:
> Hi,
>
> I am using the KSP linear solver for my system of equations, without any
> command line options at this point. I have checked that the L1 norms of my
> system matrix and the force vector are finite values, but the KSP solver is
> returning with an ?inf? residual in the very first iteration.
>
> The problem has 6.7M dofs and I have tried this on multiple machines with
> different number of nodes with the same result.
>
> Is there a reason why the solver would return after the first iteration
> with an inf?
>
> I am not sure on where to start debugging this case, so I would appreciate
> any pointers.
>
> For all solver questions, we want to see the output of
>
> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason
>
> The problem here would be that there is an error, so we would never see
> the output
> of -ksp_view and know what solver you are using. If you are using
> something complex,
> can you try using
>
> -pc_type jacobi
>
> and send the output from the options above? Then we can figure out why the
> other solver
> gets an inf.
>
> Thanks,
>
> Matt
>
> Thanks,
> Manav
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/c23fef3e/attachment.html>

From jed at jedbrown.org  Thu Mar 26 08:34:27 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 26 Mar 2015 07:34:27 -0600
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <2259494.rATCdaqx1O@asaru>
References: <2259494.rATCdaqx1O@asaru>
Message-ID: <87mw2z287w.fsf@jedbrown.org>

Florian Lindner <mailinglists at xgm.de> writes:

> Hello,
>
> I'm using petsc with petsc4py.
>
> A matrix is created like that
>
>     MPIrank = MPI.COMM_WORLD.Get_rank()
>     MPIsize = MPI.COMM_WORLD.Get_size()
>     print("MPI Rank = ", MPIrank)
>     print("MPI Size = ", MPIsize)
>     parts = partitions()
>     
>     print("Dimension= ", nSupport + dimension, "bsize = ", len(parts[MPIrank]))
>
>     MPI.COMM_WORLD.Barrier() # Just to keep the output together
>     A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport + dimension), bsize = len(parts[MPIrank]) ) # <-- crash here

bsize is collective (must be the same on all processes).  It is used for
vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/a429b023/attachment.pgp>

From david.knezevic at akselos.com  Thu Mar 26 13:48:13 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 26 Mar 2015 14:48:13 -0400
Subject: [petsc-users] Error configuring with --download-ml and Intel MKL
Message-ID: <CAJCWK9AfUT_tiS-DUinbN-MvAiYqZ9Ww6N1z10XHY+Zf2xG_sg@mail.gmail.com>

Hi all,

I'm trying to configure PETSc using Intel's MKL and with --download-ml.
Here is my configure line:

./configure
--with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64
--download-ml

I get an error when ML does "checking for dgemm". The configure.log is
attached. I was wondering if anyone has any suggestions about how I can get
this to work?

Thanks,
David


 <http://www.akselos.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/3615afec/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 1640086 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/3615afec/attachment-0001.bin>

From lb2653 at columbia.edu  Thu Mar 26 15:07:38 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Thu, 26 Mar 2015 16:07:38 -0400
Subject: [petsc-users] Mixed matrices MatMatMult
Message-ID: <5514670A.4030708@columbi.edu>

Hi all,
I want to multiply two matrices together, one is MATAIJ and the second 
is MATBAIJ, is there a way to leverage the properties of the blocked 
matrix in the BAIJ format or should I just assemble the BAIJ matrix as AIJ?

-- 
Best,
Luc



From knepley at gmail.com  Thu Mar 26 15:12:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 15:12:45 -0500
Subject: [petsc-users] Mixed matrices MatMatMult
In-Reply-To: <5514670A.4030708@columbi.edu>
References: <5514670A.4030708@columbi.edu>
Message-ID: <CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>

On Thu, Mar 26, 2015 at 3:07 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
wrote:

> Hi all,
> I want to multiply two matrices together, one is MATAIJ and the second is
> MATBAIJ, is there a way to leverage the properties of the blocked matrix in
> the BAIJ format or should I just assemble the BAIJ matrix as AIJ?


I am afraid you are currently stuck with the latter.

  Thanks,

    Matt


>
> --
> Best,
> Luc
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/71ecebc4/attachment.html>

From knepley at gmail.com  Thu Mar 26 15:21:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 15:21:45 -0500
Subject: [petsc-users] Error configuring with --download-ml and Intel MKL
In-Reply-To: <CAJCWK9AfUT_tiS-DUinbN-MvAiYqZ9Ww6N1z10XHY+Zf2xG_sg@mail.gmail.com>
References: <CAJCWK9AfUT_tiS-DUinbN-MvAiYqZ9Ww6N1z10XHY+Zf2xG_sg@mail.gmail.com>
Message-ID: <CAMYG4GnJFvUpCA5Dz+4fXEJ+BbvF0xBkcyptDLOuwUvxugBQFA@mail.gmail.com>

On Thu, Mar 26, 2015 at 1:48 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> Hi all,
>
> I'm trying to configure PETSc using Intel's MKL and with --download-ml.
> Here is my configure line:
>
> ./configure
> --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64
> --download-ml
>
> I get an error when ML does "checking for dgemm". The configure.log is
> attached. I was wondering if anyone has any suggestions about how I can get
> this to work?
>

We will need $PETSC_ARCH/externalpackages/ml-*/config.log to see where
their test failed.

However, MKL is almost never worth it.

  Thanks,

    Matt


> Thanks,
> David
>
>
>  <http://www.akselos.com>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/6fdf422e/attachment.html>

From lb2653 at columbia.edu  Thu Mar 26 16:06:55 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Thu, 26 Mar 2015 17:06:55 -0400
Subject: [petsc-users] Mixed matrices MatMatMult
In-Reply-To: <CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>
References: <5514670A.4030708@columbi.edu>
	<CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>
Message-ID: <551474EF.9020708@columbi.edu>

Ok,
this work is still part of my Schur complement approach using the full 
schur but with a block diagonal A00^-1.
I implemented the computation of A00^-1 by extracting each diagonal 
block and inverting them individually.
This works quite well and does not cost some much, especially since I 
can still use threads to accelerate this process (I might send a 
question about this in the future...).

At the moment the most expensive part of the procedure is inverting S 
(I'm using LU at the moment to make sure that everything is implemented 
correctly) and the second most expensive procedure is MatMatMult. I'm 
doing two of these: A10 * A00^-1 and then a right multiplication by A01.
Decreasing that cost would be nice (I attached the output of 
-log_summary for reference).
I think I need to look for the objects that are not Destroyed too.

Finally I now would like to split the Schur complement into two 
submatrices. I have an IS that tracks location of these sub-matrices in 
the global system:

        [ A00  A01  A02 ]  --> IS(0)
A = [ A10  A11  A12 ]  --> IS(1)
        [ A20  A21  A22 ]  --> IS(2)

How can I use IS(1) and IS(2) to track:

S =   [ A11  A12 ]  _  [ A10] * [A00]^-1 * [ A01 A02 ]  = [ S11  S12 ]  
--> IS(1)'
          [ A21  A22 ]     [ 
A20]                                          = [ S21  S22 ]  --> IS(2)'

or is there a simple way to compute IS(1)' and IS(2)' based on IS(1) and 
IS(2)?

Thanks!

Best,
Luc

On 03/26/2015 04:12 PM, Matthew Knepley wrote:
> On Thu, Mar 26, 2015 at 3:07 PM, Luc Berger-Vergiat 
> <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
>
>     Hi all,
>     I want to multiply two matrices together, one is MATAIJ and the
>     second is MATBAIJ, is there a way to leverage the properties of
>     the blocked matrix in the BAIJ format or should I just assemble
>     the BAIJ matrix as AIJ?
>
>
> I am afraid you are currently stuck with the latter.
>
>   Thanks,
>
>     Matt
>
>
>     -- 
>     Best,
>     Luc
>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/74911791/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: time.log
Type: text/x-log
Size: 11655 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/74911791/attachment-0001.bin>

From knepley at gmail.com  Thu Mar 26 17:05:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 17:05:18 -0500
Subject: [petsc-users] Mixed matrices MatMatMult
In-Reply-To: <551474EF.9020708@columbi.edu>
References: <5514670A.4030708@columbi.edu>
	<CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>
	<551474EF.9020708@columbi.edu>
Message-ID: <CAMYG4Gk+=b9Ta41KEgqaJBsVdHu4=0D8XUEWSHwOSbP-3vnbbg@mail.gmail.com>

On Thu, Mar 26, 2015 at 4:06 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
wrote:

>  Ok,
> this work is still part of my Schur complement approach using the full
> schur but with a block diagonal A00^-1.
> I implemented the computation of A00^-1 by extracting each diagonal block
> and inverting them individually.
> This works quite well and does not cost some much, especially since I can
> still use threads to accelerate this process (I might send a question about
> this in the future...).
>
> At the moment the most expensive part of the procedure is inverting S (I'm
> using LU at the moment to make sure that everything is implemented
> correctly) and the second most expensive procedure is MatMatMult. I'm doing
> two of these: A10 * A00^-1 and then a right multiplication by A01.
> Decreasing that cost would be nice (I attached the output of -log_summary
> for reference).
> I think I need to look for the objects that are not Destroyed too.
>

We will look, but this is a notoriously hard thing to speed up. You might
try the RAP code instead.

Finally I now would like to split the Schur complement into two
> submatrices. I have an IS that tracks location of these sub-matrices in the
> global system:
>
>        [ A00  A01  A02 ]  --> IS(0)
> A = [ A10  A11  A12 ]  --> IS(1)
>        [ A20  A21  A22 ]  --> IS(2)
>
> How can I use IS(1) and IS(2) to track:
>
> S =   [ A11  A12 ]  _  [ A10] * [A00]^-1 * [ A01 A02 ]  = [ S11  S12 ]
> --> IS(1)'
>          [ A21  A22 ]     [ A20]
> = [ S21  S22 ]  --> IS(2)'
>
> or is there a simple way to compute IS(1)' and IS(2)' based on IS(1) and
> IS(2)?
>

Aren't they identical?

  Thanks,

     Matt


> Thanks!
>
> Best,
> Luc
>
> On 03/26/2015 04:12 PM, Matthew Knepley wrote:
>
>  On Thu, Mar 26, 2015 at 3:07 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
> wrote:
>
>> Hi all,
>> I want to multiply two matrices together, one is MATAIJ and the second is
>> MATBAIJ, is there a way to leverage the properties of the blocked matrix in
>> the BAIJ format or should I just assemble the BAIJ matrix as AIJ?
>
>
>  I am afraid you are currently stuck with the latter.
>
>    Thanks,
>
>      Matt
>
>
>>
>> --
>> Best,
>> Luc
>>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/17b1ed5d/attachment.html>

From david.knezevic at akselos.com  Thu Mar 26 17:10:29 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Thu, 26 Mar 2015 18:10:29 -0400
Subject: [petsc-users] Error configuring with --download-ml and Intel MKL
In-Reply-To: <CAMYG4GnJFvUpCA5Dz+4fXEJ+BbvF0xBkcyptDLOuwUvxugBQFA@mail.gmail.com>
References: <CAJCWK9AfUT_tiS-DUinbN-MvAiYqZ9Ww6N1z10XHY+Zf2xG_sg@mail.gmail.com>
	<CAMYG4GnJFvUpCA5Dz+4fXEJ+BbvF0xBkcyptDLOuwUvxugBQFA@mail.gmail.com>
Message-ID: <CAJCWK9CNrvBb5E9uMvAr28P1sdOLtXgbcXxBfHbpSW_fpB5brg@mail.gmail.com>

On Thu, Mar 26, 2015 at 4:21 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Mar 26, 2015 at 1:48 PM, David Knezevic <
> david.knezevic at akselos.com> wrote:
>
>> Hi all,
>>
>> I'm trying to configure PETSc using Intel's MKL and with --download-ml.
>> Here is my configure line:
>>
>> ./configure
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64
>> --download-ml
>>
>> I get an error when ML does "checking for dgemm". The configure.log is
>> attached. I was wondering if anyone has any suggestions about how I can get
>> this to work?
>>
>
> We will need $PETSC_ARCH/externalpackages/ml-*/config.log to see where
> their test failed.
>


I've attached this file.




> However, MKL is almost never worth it.
>
>
OK, interesting, why not? The reason I'm using MKL in this case is because
I found that the direct solvers MUMPS and SuperLU were surprising slow on a
Haswell server. After rebuilding PETSc with MKL, these solvers were more
than 3x faster in some cases.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/49f45c69/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: text/x-log
Size: 82076 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/49f45c69/attachment-0001.bin>

From knepley at gmail.com  Thu Mar 26 17:23:24 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Mar 2015 17:23:24 -0500
Subject: [petsc-users] Error configuring with --download-ml and Intel MKL
In-Reply-To: <CAJCWK9CNrvBb5E9uMvAr28P1sdOLtXgbcXxBfHbpSW_fpB5brg@mail.gmail.com>
References: <CAJCWK9AfUT_tiS-DUinbN-MvAiYqZ9Ww6N1z10XHY+Zf2xG_sg@mail.gmail.com>
	<CAMYG4GnJFvUpCA5Dz+4fXEJ+BbvF0xBkcyptDLOuwUvxugBQFA@mail.gmail.com>
	<CAJCWK9CNrvBb5E9uMvAr28P1sdOLtXgbcXxBfHbpSW_fpB5brg@mail.gmail.com>
Message-ID: <CAMYG4Gm45DiRX6qLkVhqH8Z66qBLSH1kg8HAzaZK7mAyF9Hi3Q@mail.gmail.com>

On Thu, Mar 26, 2015 at 5:10 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> On Thu, Mar 26, 2015 at 4:21 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Thu, Mar 26, 2015 at 1:48 PM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to configure PETSc using Intel's MKL and with --download-ml.
>>> Here is my configure line:
>>>
>>> ./configure
>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64
>>> --download-ml
>>>
>>> I get an error when ML does "checking for dgemm". The configure.log is
>>> attached. I was wondering if anyone has any suggestions about how I can get
>>> this to work?
>>>
>>
>> We will need $PETSC_ARCH/externalpackages/ml-*/config.log to see where
>> their test failed.
>>
>
>
> I've attached this file.
>

I do not understand the error:

configure:10865: mpicxx -o conftest  -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -g -O0   -fPIC
-DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include
-I/usr/lib/openmpi/include/openmpi -g -O2 -I/usr/lib/openmpi/include
-I/usr/lib/openmpi/include/openmpi  conftest.cpp
-Wl,-rpath,/opt/intel/composer_xe_2015/mkl/lib/intel64
-L/opt/intel/composer_xe_2015/mkl/lib/intel64 -lmkl_intel_lp64
-lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib
-L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
-L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm
-Wl,-rpath,/usr/lib/openmpi/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
-Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
-lgfortran -lm -lquadmath -lm   -L/usr//lib -L/usr/lib/openmpi/lib
-L/usr/lib/gcc/x86_64-linux-gnu/4.8
-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu
-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu
-L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib
-L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lmpi_f90 -lmpi_f77 -lmpi -ldl
-lhwloc -lgfortran -lm -lquadmath -lpthread >&5
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `logf'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `atan2'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `sin'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `fabs'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `exp'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `cos'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `sqrt'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_intel_lp64.so: undefined
reference to `log'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `pow'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `log10'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `ceil'
/opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined
reference to `expf'
collect2: error: ld returned 1 exit status

Clearly these symbols should be in -lm, which is in the link line, and it
linked when we tried in our configure. Can
you run this link line manually and figure out what is wrong?


>
>
>> However, MKL is almost never worth it.
>>
>>
> OK, interesting, why not? The reason I'm using MKL in this case is because
> I found that the direct solvers MUMPS and SuperLU were surprising slow on a
> Haswell server. After rebuilding PETSc with MKL, these solvers were more
> than 3x faster in some cases.
>

Yes, direct solvers use BLAS2/3 and can benefit, but then I would just
build GotoBLAS or something easy
rather than the crazy hell that is MKL.

  Thanks,

    Matt


> David
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/7b6fc2f5/attachment.html>

From lb2653 at columbia.edu  Thu Mar 26 18:08:14 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Thu, 26 Mar 2015 19:08:14 -0400
Subject: [petsc-users] Mixed matrices MatMatMult
In-Reply-To: <CAMYG4Gk+=b9Ta41KEgqaJBsVdHu4=0D8XUEWSHwOSbP-3vnbbg@mail.gmail.com>
References: <5514670A.4030708@columbi.edu>	<CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>	<551474EF.9020708@columbi.edu>
	<CAMYG4Gk+=b9Ta41KEgqaJBsVdHu4=0D8XUEWSHwOSbP-3vnbbg@mail.gmail.com>
Message-ID: <5514915E.7040303@columbi.edu>

The difference between IS(1), IS(2) and IS(1)',IS(2)' is that they 
operate on two different matrices. The former operate on A whereas the 
operates on S and these matrices have different sizes so it's not 
obvious to me that they IS would be identical? I guess it depends on the 
initial ordering of the blocks of A, if A00 is actually the bottom right 
block then I can see how the two are the same.

But just to get a confirmation here is a simple example: let's assume I 
solve a 2D problem with momentum and energy conservation.
In that case I get three equations per nodes:

C1*d^2 u_x/dx^2 + C2*d^2 u_y/dxdy + a*dT/dt= fx      ---> IS(0)
C3*d^2 u_x/dxdy + C4*d^2 u_y/dy^2 + a*dT/dt = fy     ---> IS(1)
dT/dt + kappa*(d^2 T/dx^2 + d^2 T/dy^2) = fT            ---> IS(2)

This leads to a block matrix with 3x3 block per nodes. If I extract the 
temperature problem, invert it and compute the schur complement I will 
end up with S with the same size as the elasticity part of the problem 
but will the IS(0) and IS(1) be the same for the jacobian and the Schur 
complement?
If that's the case that's awesome that you programmed it to keep track 
of these things!

Best,
Luc

On 03/26/2015 06:05 PM, Matthew Knepley wrote:
> On Thu, Mar 26, 2015 at 4:06 PM, Luc Berger-Vergiat 
> <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
>
>     Ok,
>     this work is still part of my Schur complement approach using the
>     full schur but with a block diagonal A00^-1.
>     I implemented the computation of A00^-1 by extracting each
>     diagonal block and inverting them individually.
>     This works quite well and does not cost some much, especially
>     since I can still use threads to accelerate this process (I might
>     send a question about this in the future...).
>
>     At the moment the most expensive part of the procedure is
>     inverting S (I'm using LU at the moment to make sure that
>     everything is implemented correctly) and the second most expensive
>     procedure is MatMatMult. I'm doing two of these: A10 * A00^-1 and
>     then a right multiplication by A01.
>     Decreasing that cost would be nice (I attached the output of
>     -log_summary for reference).
>     I think I need to look for the objects that are not Destroyed too.
>
>
> We will look, but this is a notoriously hard thing to speed up. You 
> might try the RAP code instead.
>
>     Finally I now would like to split the Schur complement into two
>     submatrices. I have an IS that tracks location of these
>     sub-matrices in the global system:
>
>            [ A00  A01  A02 ]  --> IS(0)
>     A = [ A10  A11  A12 ]  --> IS(1)
>            [ A20  A21  A22 ]  --> IS(2)
>
>     How can I use IS(1) and IS(2) to track:
>
>     S =   [ A11  A12 ]  _  [ A10] * [A00]^-1 * [ A01 A02 ] = [ S11 
>     S12 ]  --> IS(1)'
>              [ A21  A22 ]     [
>     A20]                                          = [ S21 S22 ]  -->
>     IS(2)'
>
>     or is there a simple way to compute IS(1)' and IS(2)' based on
>     IS(1) and IS(2)?
>
>
> Aren't they identical?
>
>   Thanks,
>
>      Matt
>
>     Thanks!
>
>     Best,
>     Luc
>
>     On 03/26/2015 04:12 PM, Matthew Knepley wrote:
>>     On Thu, Mar 26, 2015 at 3:07 PM, Luc Berger-Vergiat
>>     <lb2653 at columbia.edu <mailto:lb2653 at columbia.edu>> wrote:
>>
>>         Hi all,
>>         I want to multiply two matrices together, one is MATAIJ and
>>         the second is MATBAIJ, is there a way to leverage the
>>         properties of the blocked matrix in the BAIJ format or should
>>         I just assemble the BAIJ matrix as AIJ?
>>
>>
>>     I am afraid you are currently stuck with the latter.
>>
>>       Thanks,
>>
>>         Matt
>>
>>
>>         -- 
>>         Best,
>>         Luc
>>
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150326/d7e94e44/attachment.html>

From bsmith at mcs.anl.gov  Thu Mar 26 20:52:59 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Mar 2015 20:52:59 -0500
Subject: [petsc-users] Column #j is wrong in parallel from message
	"Inserting a new nonzero (i, j) into matrix"
In-Reply-To: <5512F866.5070405@giref.ulaval.ca>
References: <5512F866.5070405@giref.ulaval.ca>
Message-ID: <F5103C6A-3314-4C97-B8A4-F9C2A5C91643@mcs.anl.gov>

 
  Eric,

  I have now updated all the standard MPI matrix types AIJ, BAIJ, SBAIJ to print the correct global indices in the error messages when a new nonzero location is generated thus making debugging this issue easier. In the branches barry/fix-inserting-new-nonzero-column-location, next and the next release.

   Thanks for pushing on this. The previous code was too "developer centric" and not enough "user centric" enough.

   Barry

> On Mar 25, 2015, at 1:03 PM, Eric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
> 
> Hi,
> 
> while looking for where in the world do I insert the (135,9) entry in my matrix, I have discovered that the column # shown is wrong in parallel!
> 
> I am using PETsc 3.5.3.
> 
> The full error message is:
> 
> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 564 in /home/mefpp_ericc/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c Inserting a new nonzero (135, 9) into matrix
> 
> This line code is a call to a #defined macro:
> 
> MatSetValues_SeqAIJ_B_Private(row,col,value,addv);
> 
> where the "col" parameter is not equal to "in[j]"!!!
> 
> in gdb, printing "in[j]" gave me:
> 
> print in[j]
> $6 = 537
> 
> while "col" is:
> 
> print col
> $7 = 9
> 
> So, I expected to have a message telling me that (135,537) and not (135,9) is a new entry matrix!!!
> 
> Would it be a big work to fix this so that the col # displayed is correct?
> 
> Thanks!
> 
> Eric


From mailinglists at xgm.de  Fri Mar 27 04:28:45 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Fri, 27 Mar 2015 10:28:45 +0100
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <87mw2z287w.fsf@jedbrown.org>
References: <2259494.rATCdaqx1O@asaru> <87mw2z287w.fsf@jedbrown.org>
Message-ID: <3375546.tIWFGx0teZ@asaru>

Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> Florian Lindner <mailinglists at xgm.de> writes:
> 
> > Hello,
> >
> > I'm using petsc with petsc4py.
> >
> > A matrix is created like that
> >
> >     MPIrank = MPI.COMM_WORLD.Get_rank()
> >     MPIsize = MPI.COMM_WORLD.Get_size()
> >     print("MPI Rank = ", MPIrank)
> >     print("MPI Size = ", MPIsize)
> >     parts = partitions()
> >     
> >     print("Dimension= ", nSupport + dimension, "bsize = ", len(parts[MPIrank]))
> >
> >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> >     A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport + dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> 
> bsize is collective (must be the same on all processes).  It is used for
> vector-valued problems (like elasticity -- bs=3 in 3 dimensions).

It seems I'm still misunderstanding the bsize parameter.

If I distribute a 10x10 matrix on three ranks I need to have a non-homogenous distribution, and thats what petsc does itself:

A.createDense( (n, n) )

print("Rank = ", rank, "Range    = ", A.owner_range, "Size = ", A.owner_range[1] - A.owner_range[0])
print("Rank = ", rank, "ColRange = ", A.getOwnershipRangeColumn(), "Size = ", A.getOwnershipRangeColumn()[1] - A.getOwnershipRangeColumn()[0])

gives:

Rank =  2 Range    =  (7, 10) Size =  3
Rank =  2 ColRange =  (7, 10) Size =  3
Rank =  0 Range    =  (0, 4)  Size =  4
Rank =  0 ColRange =  (0, 4)  Size =  4
Rank =  1 Range    =  (4, 7)  Size =  3
Rank =  1 ColRange =  (4, 7)  Size =  3


How can I manually set a distribution of rows like above? My approach was to call create with bsize = [3,3,4][rank] but that obviously is not the way...

Thanks,
Florian

From Eric.Chamberland at giref.ulaval.ca  Fri Mar 27 05:55:12 2015
From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland)
Date: Fri, 27 Mar 2015 06:55:12 -0400
Subject: [petsc-users] Column #j is wrong in parallel from message
 "Inserting a new nonzero (i, j) into matrix"
In-Reply-To: <F5103C6A-3314-4C97-B8A4-F9C2A5C91643@mcs.anl.gov>
References: <5512F866.5070405@giref.ulaval.ca>
	<F5103C6A-3314-4C97-B8A4-F9C2A5C91643@mcs.anl.gov>
Message-ID: <55153710.2020203@giref.ulaval.ca>

On 03/26/2015 09:52 PM, Barry Smith wrote:
>   
>    Eric,
>
>    I have now updated all the standard MPI matrix types AIJ, BAIJ, SBAIJ to print the correct global indices in the error messages when a new nonzero location is generated thus making debugging this issue easier. In the branches barry/fix-inserting-new-nonzero-column-location, next and the next release.
Great!  I think it will be very useful to help users like us!
>     Thanks for pushing on this. The previous code was too "developer centric" and not enough "user centric" enough.
oh, I would like to thank you to be sensible and so fast to enhance this 
debugging info!...  continuing to make PETSc a such great tool! :-)

Thanks again! :-)

Eric



From knepley at gmail.com  Fri Mar 27 07:26:11 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Mar 2015 07:26:11 -0500
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <3375546.tIWFGx0teZ@asaru>
References: <2259494.rATCdaqx1O@asaru> <87mw2z287w.fsf@jedbrown.org>
	<3375546.tIWFGx0teZ@asaru>
Message-ID: <CAMYG4G=8Gevj9j5ML2q58tap3hKByZJ-dWvtdbHjL9n+W+-6wA@mail.gmail.com>

On Fri, Mar 27, 2015 at 4:28 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> > Florian Lindner <mailinglists at xgm.de> writes:
> >
> > > Hello,
> > >
> > > I'm using petsc with petsc4py.
> > >
> > > A matrix is created like that
> > >
> > >     MPIrank = MPI.COMM_WORLD.Get_rank()
> > >     MPIsize = MPI.COMM_WORLD.Get_size()
> > >     print("MPI Rank = ", MPIrank)
> > >     print("MPI Size = ", MPIsize)
> > >     parts = partitions()
> > >
> > >     print("Dimension= ", nSupport + dimension, "bsize = ",
> len(parts[MPIrank]))
> > >
> > >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> > >     A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport +
> dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> >
> > bsize is collective (must be the same on all processes).  It is used for
> > vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
>
> It seems I'm still misunderstanding the bsize parameter.
>
> If I distribute a 10x10 matrix on three ranks I need to have a
> non-homogenous distribution, and thats what petsc does itself:
>

blockSize really means the uniform block size of the matrix, thus is HAS to
divide the global size. If it does not,
you do not have a uniform block size, you have a bunch of different sized
blocks.

  Thanks,

     Matt


> A.createDense( (n, n) )
>
> print("Rank = ", rank, "Range    = ", A.owner_range, "Size = ",
> A.owner_range[1] - A.owner_range[0])
> print("Rank = ", rank, "ColRange = ", A.getOwnershipRangeColumn(), "Size =
> ", A.getOwnershipRangeColumn()[1] - A.getOwnershipRangeColumn()[0])
>
> gives:
>
> Rank =  2 Range    =  (7, 10) Size =  3
> Rank =  2 ColRange =  (7, 10) Size =  3
> Rank =  0 Range    =  (0, 4)  Size =  4
> Rank =  0 ColRange =  (0, 4)  Size =  4
> Rank =  1 Range    =  (4, 7)  Size =  3
> Rank =  1 ColRange =  (4, 7)  Size =  3
>
>
> How can I manually set a distribution of rows like above? My approach was
> to call create with bsize = [3,3,4][rank] but that obviously is not the
> way...
>
> Thanks,
> Florian
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/3d937b47/attachment.html>

From mailinglists at xgm.de  Fri Mar 27 07:31:17 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Fri, 27 Mar 2015 13:31:17 +0100
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <CAMYG4G=8Gevj9j5ML2q58tap3hKByZJ-dWvtdbHjL9n+W+-6wA@mail.gmail.com>
References: <2259494.rATCdaqx1O@asaru> <3375546.tIWFGx0teZ@asaru>
	<CAMYG4G=8Gevj9j5ML2q58tap3hKByZJ-dWvtdbHjL9n+W+-6wA@mail.gmail.com>
Message-ID: <7174127.NEH3BeKRx3@asaru>



Am Freitag, 27. M?rz 2015, 07:26:11 schrieb Matthew Knepley:
> On Fri, Mar 27, 2015 at 4:28 AM, Florian Lindner <mailinglists at xgm.de>
> wrote:
> 
> > Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> > > Florian Lindner <mailinglists at xgm.de> writes:
> > >
> > > > Hello,
> > > >
> > > > I'm using petsc with petsc4py.
> > > >
> > > > A matrix is created like that
> > > >
> > > >     MPIrank = MPI.COMM_WORLD.Get_rank()
> > > >     MPIsize = MPI.COMM_WORLD.Get_size()
> > > >     print("MPI Rank = ", MPIrank)
> > > >     print("MPI Size = ", MPIsize)
> > > >     parts = partitions()
> > > >
> > > >     print("Dimension= ", nSupport + dimension, "bsize = ",
> > len(parts[MPIrank]))
> > > >
> > > >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> > > >     A = PETSc.Mat(); A.createDense( (nSupport + dimension, nSupport +
> > dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> > >
> > > bsize is collective (must be the same on all processes).  It is used for
> > > vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
> >
> > It seems I'm still misunderstanding the bsize parameter.
> >
> > If I distribute a 10x10 matrix on three ranks I need to have a
> > non-homogenous distribution, and thats what petsc does itself:
> >
> 
> blockSize really means the uniform block size of the matrix, thus is HAS to
> divide the global size. If it does not,
> you do not have a uniform block size, you have a bunch of different sized
> blocks.

But how can I set a parallel layout when the size of the matrix is not divisable by the number of ranks? When I omit bsize Petsc does that for me, by using block sizes of 4, 3 and 3 on the three different ranks. How can I set such a parallel layout manually?

Thanks,
Florian


> > A.createDense( (n, n) )
> >
> > print("Rank = ", rank, "Range    = ", A.owner_range, "Size = ",
> > A.owner_range[1] - A.owner_range[0])
> > print("Rank = ", rank, "ColRange = ", A.getOwnershipRangeColumn(), "Size =
> > ", A.getOwnershipRangeColumn()[1] - A.getOwnershipRangeColumn()[0])
> >
> > gives:
> >
> > Rank =  2 Range    =  (7, 10) Size =  3
> > Rank =  2 ColRange =  (7, 10) Size =  3
> > Rank =  0 Range    =  (0, 4)  Size =  4
> > Rank =  0 ColRange =  (0, 4)  Size =  4
> > Rank =  1 Range    =  (4, 7)  Size =  3
> > Rank =  1 ColRange =  (4, 7)  Size =  3
> >
> >
> > How can I manually set a distribution of rows like above? My approach was
> > to call create with bsize = [3,3,4][rank] but that obviously is not the
> > way...
> >
> > Thanks,
> > Florian
> >
> 
> 
> 
> 

From knepley at gmail.com  Fri Mar 27 07:34:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Mar 2015 07:34:56 -0500
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <7174127.NEH3BeKRx3@asaru>
References: <2259494.rATCdaqx1O@asaru> <3375546.tIWFGx0teZ@asaru>
	<CAMYG4G=8Gevj9j5ML2q58tap3hKByZJ-dWvtdbHjL9n+W+-6wA@mail.gmail.com>
	<7174127.NEH3BeKRx3@asaru>
Message-ID: <CAMYG4Gms2XcpGGXBY26p5Jejatxfn4jXLOm8YpxouOMuPAMXKg@mail.gmail.com>

On Fri, Mar 27, 2015 at 7:31 AM, Florian Lindner <mailinglists at xgm.de>
wrote:
>
> Am Freitag, 27. M?rz 2015, 07:26:11 schrieb Matthew Knepley:
> > On Fri, Mar 27, 2015 at 4:28 AM, Florian Lindner <mailinglists at xgm.de>
> > wrote:
> >
> > > Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> > > > Florian Lindner <mailinglists at xgm.de> writes:
> > > >
> > > > > Hello,
> > > > >
> > > > > I'm using petsc with petsc4py.
> > > > >
> > > > > A matrix is created like that
> > > > >
> > > > >     MPIrank = MPI.COMM_WORLD.Get_rank()
> > > > >     MPIsize = MPI.COMM_WORLD.Get_size()
> > > > >     print("MPI Rank = ", MPIrank)
> > > > >     print("MPI Size = ", MPIsize)
> > > > >     parts = partitions()
> > > > >
> > > > >     print("Dimension= ", nSupport + dimension, "bsize = ",
> > > len(parts[MPIrank]))
> > > > >
> > > > >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> > > > >     A = PETSc.Mat(); A.createDense( (nSupport + dimension,
> nSupport +
> > > dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> > > >
> > > > bsize is collective (must be the same on all processes).  It is used
> for
> > > > vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
> > >
> > > It seems I'm still misunderstanding the bsize parameter.
> > >
> > > If I distribute a 10x10 matrix on three ranks I need to have a
> > > non-homogenous distribution, and thats what petsc does itself:
> > >
> >
> > blockSize really means the uniform block size of the matrix, thus is HAS
> to
> > divide the global size. If it does not,
> > you do not have a uniform block size, you have a bunch of different sized
> > blocks.
>
> But how can I set a parallel layout when the size of the matrix is not
> divisable by the number of ranks? When I omit bsize Petsc does that for me,
> by using block sizes of 4, 3 and 3 on the three different ranks. How can I
> set such a parallel layout manually?
>

I am going to reply in C because it is my native language:

  MatCreate(comm, &A);
  MatSetSizes(A, m, n, PETSC_DETERMINE, PETSC_DETERMINE);
  MatSetFromOptions(A);
  <Preallocation stuff here>

You have each proc give its local size.

  Thanks,

     Matt


> Thanks,
> Florian
>
>
> > > A.createDense( (n, n) )
> > >
> > > print("Rank = ", rank, "Range    = ", A.owner_range, "Size = ",
> > > A.owner_range[1] - A.owner_range[0])
> > > print("Rank = ", rank, "ColRange = ", A.getOwnershipRangeColumn(),
> "Size =
> > > ", A.getOwnershipRangeColumn()[1] - A.getOwnershipRangeColumn()[0])
> > >
> > > gives:
> > >
> > > Rank =  2 Range    =  (7, 10) Size =  3
> > > Rank =  2 ColRange =  (7, 10) Size =  3
> > > Rank =  0 Range    =  (0, 4)  Size =  4
> > > Rank =  0 ColRange =  (0, 4)  Size =  4
> > > Rank =  1 Range    =  (4, 7)  Size =  3
> > > Rank =  1 ColRange =  (4, 7)  Size =  3
> > >
> > >
> > > How can I manually set a distribution of rows like above? My approach
> was
> > > to call create with bsize = [3,3,4][rank] but that obviously is not the
> > > way...
> > >
> > > Thanks,
> > > Florian
> > >
> >
> >
> >
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/36ce7e7e/attachment.html>

From simpson at math.drexel.edu  Fri Mar 27 08:13:53 2015
From: simpson at math.drexel.edu (Gideon Simpson)
Date: Fri, 27 Mar 2015 09:13:53 -0400
Subject: [petsc-users] installation question
Message-ID: <7F4F5191-15DC-464A-9027-AC4F0B7C024D@math.drexel.edu>

My admin did an install of petsc on ubuntu 14.0 using the package manager.  Things seem to be ok, expect I noticed the following when I compile using the petsc rules:

mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -fPIC    -I/usr/lib/petscdir/3.4.2/include -I/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/include -I/usr/include -I/usr/include/suitesparse -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi    -D__INSDIR__= ex19.c
/usr/bin/python: can't open file '/usr/lib/petscdir/3.4.2/bin/petscnagupgrade.py': [Errno 2] No such file or directory
make: [chk_upgrade] Error 2 (ignored)
mpicxx -Wl,--no-as-needed  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19 ex19.o  -L/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib -L/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib  -lpetsc -L/usr/lib -ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord -lscalapack-openmpi -lHYPRE_utilities -lHYPRE_struct_mv -lHYPRE_struct_ls -lHYPRE_sstruct_mv -lHYPRE_sstruct_ls -lHYPRE_IJ_mv -lHYPRE_parcsr_ls -lcholmod -lumfpack -lamd -llapack -lblas -lX11 -lpthread -lptesmumps -lptscotch -lptscotcherr -L/usr/lib/x86_64-linux-gnu -lfftw3 -lfftw3_mpi -lm -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl 
/bin/rm -f -f ex19.o

I know the file is just a nag to upgrade petsc, but should I just continue to ignore it?  Is there a reason it wasn?t installed by the package manager?

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/fee7fada/attachment-0001.html>

From knepley at gmail.com  Fri Mar 27 09:43:46 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Mar 2015 09:43:46 -0500
Subject: [petsc-users] Mixed matrices MatMatMult
In-Reply-To: <5514915E.7040303@columbi.edu>
References: <5514670A.4030708@columbi.edu>
	<CAMYG4GniTA0jX7+JWT=Rvi5bt+XHHTqcaBaofOz5NgwThz_c7w@mail.gmail.com>
	<551474EF.9020708@columbi.edu>
	<CAMYG4Gk+=b9Ta41KEgqaJBsVdHu4=0D8XUEWSHwOSbP-3vnbbg@mail.gmail.com>
	<5514915E.7040303@columbi.edu>
Message-ID: <CAMYG4Gm0HOSQGjT0Pb4HySONvR+O=f_LFQD-MS3ynA6BTfv7Gg@mail.gmail.com>

On Thu, Mar 26, 2015 at 6:08 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
wrote:

>  The difference between IS(1), IS(2) and IS(1)',IS(2)' is that they
> operate on two different matrices. The former operate on A whereas the
> operates on S and these matrices have different sizes so it's not obvious
> to me that they IS would be identical? I guess it depends on the initial
> ordering of the blocks of A, if A00 is actually the bottom right block then
> I can see how the two are the same.
>
> But just to get a confirmation here is a simple example: let's assume I
> solve a 2D problem with momentum and energy conservation.
> In that case I get three equations per nodes:
>
> C1*d^2 u_x/dx^2 + C2*d^2 u_y/dxdy + a*dT/dt= fx      ---> IS(0)
> C3*d^2 u_x/dxdy + C4*d^2 u_y/dy^2 + a*dT/dt = fy     ---> IS(1)
> dT/dt + kappa*(d^2 T/dx^2 + d^2 T/dy^2) = fT            ---> IS(2)
>
> This leads to a block matrix with 3x3 block per nodes. If I extract the
> temperature problem, invert it and compute the schur complement I will end
> up with S with the same size as the elasticity part of the problem but will
> the IS(0) and IS(1) be the same for the jacobian and the Schur complement?
> If that's the case that's awesome that you programmed it to keep track of
> these things!
>

Nope, I was misunderstanding. However, if you have a DM, you can get the
new ISes the same
way. The Schur complement was formed using DMCreateSubDM(), and you can
call the same
thing for those fields to get the subDM and those ISes. Does this makes
sense?

If you do not have a DM, then its a translation process.

  Thanks,

     Matt


> Best,
> Luc
>
> On 03/26/2015 06:05 PM, Matthew Knepley wrote:
>
>  On Thu, Mar 26, 2015 at 4:06 PM, Luc Berger-Vergiat <lb2653 at columbia.edu>
> wrote:
>
>>  Ok,
>> this work is still part of my Schur complement approach using the full
>> schur but with a block diagonal A00^-1.
>> I implemented the computation of A00^-1 by extracting each diagonal block
>> and inverting them individually.
>> This works quite well and does not cost some much, especially since I can
>> still use threads to accelerate this process (I might send a question about
>> this in the future...).
>>
>> At the moment the most expensive part of the procedure is inverting S
>> (I'm using LU at the moment to make sure that everything is implemented
>> correctly) and the second most expensive procedure is MatMatMult. I'm doing
>> two of these: A10 * A00^-1 and then a right multiplication by A01.
>> Decreasing that cost would be nice (I attached the output of -log_summary
>> for reference).
>> I think I need to look for the objects that are not Destroyed too.
>>
>
>  We will look, but this is a notoriously hard thing to speed up. You
> might try the RAP code instead.
>
>   Finally I now would like to split the Schur complement into two
>> submatrices. I have an IS that tracks location of these sub-matrices in the
>> global system:
>>
>>        [ A00  A01  A02 ]  --> IS(0)
>> A = [ A10  A11  A12 ]  --> IS(1)
>>        [ A20  A21  A22 ]  --> IS(2)
>>
>> How can I use IS(1) and IS(2) to track:
>>
>> S =   [ A11  A12 ]  _  [ A10] * [A00]^-1 * [ A01 A02 ]  = [ S11  S12 ]
>> --> IS(1)'
>>          [ A21  A22 ]     [ A20]
>> = [ S21  S22 ]  --> IS(2)'
>>
>> or is there a simple way to compute IS(1)' and IS(2)' based on IS(1) and
>> IS(2)?
>>
>
>  Aren't they identical?
>
>    Thanks,
>
>       Matt
>
>
>>  Thanks!
>>
>> Best,
>> Luc
>>
>> On 03/26/2015 04:12 PM, Matthew Knepley wrote:
>>
>>  On Thu, Mar 26, 2015 at 3:07 PM, Luc Berger-Vergiat <lb2653 at columbia.edu
>> > wrote:
>>
>>> Hi all,
>>> I want to multiply two matrices together, one is MATAIJ and the second
>>> is MATBAIJ, is there a way to leverage the properties of the blocked matrix
>>> in the BAIJ format or should I just assemble the BAIJ matrix as AIJ?
>>
>>
>>  I am afraid you are currently stuck with the latter.
>>
>>    Thanks,
>>
>>      Matt
>>
>>
>>>
>>>  --
>>> Best,
>>> Luc
>>>
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/88373af0/attachment.html>

From kaiser at igpm.rwth-aachen.de  Fri Mar 27 09:48:07 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Fri, 27 Mar 2015 15:48:07 +0100
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
Message-ID: <55156DA7.6010602@igpm.rwth-aachen.de>

Hallo,

I have a strange behavior in my code concerning the function MatAXPY. I 
create 3 different Matrices

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, 
local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

and want to sum these three matrixes with different factors. First I 
fill the Matrix A with some values, and duplicate the structure of A to 
the other two matrices:

     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);

After this I fill the matrices At and Ah with some other values, which 
are not beside the non zero structure (I also tried with just copying 
the Matrix A). Now after another MatAssembly I want to add these 
Matrices in the form A+c*(Ah+d*At):

     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);

When I run the method with mpi and one core everything works fine. 
Starting the same method with more cores, the sum of the matrices fails. 
It seems like some values are added correctly and many values are 
missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in 
the multi-core case, but is very slow. I checked with a viewer if all 
matrices have the same nonzero structure and this is the case.

Does anyone know why this fails, or do I have made any wrong thoughts?

I'm corrently working with a petsc version (Petsc Release Version 3.3.0, 
Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up 
to the current version and did not find any note about MatAXPY or MatAYPX.


Best and Thanks a lot for your help

Klaus


From knepley at gmail.com  Fri Mar 27 09:59:02 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Mar 2015 09:59:02 -0500
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <55156DA7.6010602@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
Message-ID: <CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>

On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de>
wrote:

> Hallo,
>
> I have a strange behavior in my code concerning the function MatAXPY. I
> create 3 different Matrices
>
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>

These creations are superfluous since you use MatDuplicate() below.


> and want to sum these three matrixes with different factors. First I fill
> the Matrix A with some values, and duplicate the structure of A to the
> other two matrices:
>
>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>


> After this I fill the matrices At and Ah with some other values, which are
> not beside the non zero structure (I also tried with just copying the
> Matrix A). Now after another MatAssembly


I do not understand "no beside the nonzero structure". Do you mean that the
nonzero structure is the same?

Can you first test with MAT_COPY_VALUES?


> I want to add these Matrices in the form A+c*(Ah+d*At):


>     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>
> When I run the method with mpi and one core everything works fine.
> Starting the same method with more cores, the sum of the matrices fails. It
> seems like some values are added


Please send the full output of a failure when you use MAT_COPY_VALUES on 2
procs.

  Thanks,

     Matt


> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE
> leads to the right behavior in the multi-core case, but is very slow. I
> checked with a viewer if all matrices have the same nonzero structure and
> this is the case.
>
> Does anyone know why this fails, or do I have made any wrong thoughts?
>
> I'm corrently working with a petsc version (Petsc Release Version 3.3.0,
> Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to
> the current version and did not find any note about MatAXPY or MatAYPX.
>
>
> Best and Thanks a lot for your help
>
> Klaus
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/bb852698/attachment.html>

From balay at mcs.anl.gov  Fri Mar 27 10:16:36 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 27 Mar 2015 10:16:36 -0500
Subject: [petsc-users] installation question
In-Reply-To: <7F4F5191-15DC-464A-9027-AC4F0B7C024D@math.drexel.edu>
References: <7F4F5191-15DC-464A-9027-AC4F0B7C024D@math.drexel.edu>
Message-ID: <alpine.LFD.2.11.1503271015170.15095@asterix>

On Fri, 27 Mar 2015, Gideon Simpson wrote:

> My admin did an install of petsc on ubuntu 14.0 using the package manager.  Things seem to be ok, expect I noticed the following when I compile using the petsc rules:
> 
> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -fPIC    -I/usr/lib/petscdir/3.4.2/include -I/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/include -I/usr/include -I/usr/include/suitesparse -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi    -D__INSDIR__= ex19.c
> /usr/bin/python: can't open file '/usr/lib/petscdir/3.4.2/bin/petscnagupgrade.py': [Errno 2] No such file or directory
> make: [chk_upgrade] Error 2 (ignored)
> mpicxx -Wl,--no-as-needed  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -o ex19 ex19.o  -L/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib -L/usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib  -lpetsc -L/usr/lib -ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord -lscalapack-openmpi -lHYPRE_utilities -lHYPRE_struct_mv -lHYPRE_struct_ls -lHYPRE_sstruct_mv -lHYPRE_sstruct_ls -lHYPRE_IJ_mv -lHYPRE_parcsr_ls -lcholmod -lumfpack -lamd -llapack -lblas -lX11 -lpthread -lptesmumps -lptscotch -lptscotcherr -L/usr/lib/x86_64-linux-gnu -lfftw3 -lfftw3_mpi -lm -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl 
> /bin/rm -f -f ex19.o
> 
> I know the file is just a nag to upgrade petsc, but should I just continue to ignore it?  Is there a reason it wasn?t installed by the package manager?

Thats a question for the person packaging PETSc [for debian/ubuntu]

You can remove 'chkopts' from your makefile - and then it won't do this check..

Satish

From kaiser at igpm.rwth-aachen.de  Fri Mar 27 10:16:48 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Fri, 27 Mar 2015 16:16:48 +0100
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>
Message-ID: <55157460.4080200@igpm.rwth-aachen.de>

Hallo Matthew,

thanks for your fast response. With "no beside the nonzero structure" I 
meant, that I do not create a different non-zero structure while I'm 
adding values to my matrices.

I also tried MAT_COPY_VALUES and the result was the same.

I do not get a error message but the resulting matrix is wrong. Here I 
have a short example. The first 9 rows and first 3 colomn of the 
matrices Ah, At and Ah+At on the first processor (of 8):

Ah:
0:                   60.3553                -0.249975 2.77556e-17
1:                         0 60.3553                        0
2:                         0 0                  60.3553
3:                   17.6777 0.374962                 0.124987
4:                         0 17.6777                        0
5:                         0 0                  17.6777
6:                  -7.32233 -0.124987                 0.374962
7:                         0 -7.32233                        0
8:                         0 0                 -7.32233


At:
0:                         0 0                        0
1:                      2500 0                        0
2:              -4.54747e-13 0                        0
3:                         0 0                        0
4:                      1250 0                        0
5:                      1250 0                        0
6:                         0 0                        0
7:                      1250 0                        0
8:                      3750 0                        0

Ah+At
0:                   60.3553                -0.249975 2.77556e-17
1:                      2500 60.3553                        0
2:              -4.54747e-13 0                  60.3553
3:                   17.6777 0.374962                 0.124987
4:                         0 17.6777                        0
5:                         0 0                  17.6777
6:                  -7.32233 -0.124987                 0.374962
7:                         0 -7.32233                        0
8:                         0 0                 -7.32233

you can see the first 3 rows of the resulting matrix looks exactly like 
what I would expect, but the last 6 rows only consists of the values of 
Ah. When you would also look on the matrix A and A+Ah+At you would also 
see, that values where both matrices have an nonzero entry are not sum 
correctly.

Best

Klaus

On 03/27/2015 03:59 PM, Matthew Knepley wrote:
> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser 
> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>
>     Hallo,
>
>     I have a strange behavior in my code concerning the function
>     MatAXPY. I create 3 different Matrices
>
>         ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>     local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>         ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>         ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>         ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>
>         ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>     local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>         ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>         ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>         ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>
>         ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>     local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>         ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>         ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>         ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>
>
> These creations are superfluous since you use MatDuplicate() below.
>
>     and want to sum these three matrixes with different factors. First
>     I fill the Matrix A with some values, and duplicate the structure
>     of A to the other two matrices:
>
>         MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>         MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>         MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>         MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>         MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>         MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
>
>     After this I fill the matrices At and Ah with some other values,
>     which are not beside the non zero structure (I also tried with
>     just copying the Matrix A). Now after another MatAssembly
>
>
> I do not understand "no beside the nonzero structure". Do you mean 
> that the nonzero structure is the same?
>
> Can you first test with MAT_COPY_VALUES?
>
>     I want to add these Matrices in the form A+c*(Ah+d*At): 
>
>
>         MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>         MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>
>     When I run the method with mpi and one core everything works fine.
>     Starting the same method with more cores, the sum of the matrices
>     fails. It seems like some values are added
>
>
> Please send the full output of a failure when you use MAT_COPY_VALUES 
> on 2 procs.
>
>   Thanks,
>
>      Matt
>
>     correctly and many values are missed. Using
>     DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the
>     multi-core case, but is very slow. I checked with a viewer if all
>     matrices have the same nonzero structure and this is the case.
>
>     Does anyone know why this fails, or do I have made any wrong thoughts?
>
>     I'm corrently working with a petsc version (Petsc Release Version
>     3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the
>     changelogs up to the current version and did not find any note
>     about MatAXPY or MatAYPX.
>
>
>     Best and Thanks a lot for your help
>
>     Klaus
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/52daa64e/attachment.html>

From knepley at gmail.com  Fri Mar 27 10:26:36 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Mar 2015 10:26:36 -0500
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <55157460.4080200@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>
	<55157460.4080200@igpm.rwth-aachen.de>
Message-ID: <CAMYG4G=OR3=gMHxAf2NMQCkkh7PpPDRBaoFZ+AeamiKaE2xapw@mail.gmail.com>

On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de>
wrote:

>  Hallo Matthew,
>
> thanks for your fast response. With "no beside the nonzero structure" I
> meant, that I do not create a different non-zero structure while I'm adding
> values to my matrices.
>
> I also tried MAT_COPY_VALUES and the result was the same.
>

Please send the small example and we will figure out what is going wrong.

  Thanks,

     Matt


> I do not get a error message but the resulting matrix is wrong. Here I
> have a short example. The first 9 rows and first 3 colomn of the matrices
> Ah, At and Ah+At on the first processor (of 8):
>
> Ah:
> 0:                   60.3553                -0.249975
> 2.77556e-17
> 1:                         0
> 60.3553                        0
> 2:                         0                        0
> 60.3553
> 3:                   17.6777                 0.374962
> 0.124987
> 4:                         0
> 17.6777                        0
> 5:                         0                        0
> 17.6777
> 6:                  -7.32233                -0.124987
> 0.374962
> 7:                         0
> -7.32233                        0
> 8:                         0                        0
> -7.32233
>
>
> At:
> 0:                         0
> 0                        0
> 1:                      2500
> 0                        0
> 2:              -4.54747e-13
> 0                        0
> 3:                         0
> 0                        0
> 4:                      1250
> 0                        0
> 5:                      1250
> 0                        0
> 6:                         0
> 0                        0
> 7:                      1250
> 0                        0
> 8:                      3750
> 0                        0
>
> Ah+At
> 0:                   60.3553                -0.249975
> 2.77556e-17
> 1:                      2500
> 60.3553                        0
> 2:              -4.54747e-13                        0
> 60.3553
> 3:                   17.6777                 0.374962
> 0.124987
> 4:                         0
> 17.6777                        0
> 5:                         0                        0
> 17.6777
> 6:                  -7.32233                -0.124987
> 0.374962
> 7:                         0
> -7.32233                        0
> 8:                         0                        0
> -7.32233
>
> you can see the first 3 rows of the resulting matrix looks exactly like
> what I would expect, but the last 6 rows only consists of the values of Ah.
> When you would also look on the matrix A and A+Ah+At you would also see,
> that values where both matrices have an nonzero entry are not sum correctly.
>
> Best
>
> Klaus
>
> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>
>  On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de
> > wrote:
>
>> Hallo,
>>
>> I have a strange behavior in my code concerning the function MatAXPY. I
>> create 3 different Matrices
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>
>  These creations are superfluous since you use MatDuplicate() below.
>
>
>> and want to sum these three matrixes with different factors. First I fill
>> the Matrix A with some values, and duplicate the structure of A to the
>> other two matrices:
>>
>>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>
>
>
>> After this I fill the matrices At and Ah with some other values, which
>> are not beside the non zero structure (I also tried with just copying the
>> Matrix A). Now after another MatAssembly
>
>
>  I do not understand "no beside the nonzero structure". Do you mean that
> the nonzero structure is the same?
>
>  Can you first test with MAT_COPY_VALUES?
>
>
>> I want to add these Matrices in the form A+c*(Ah+d*At):
>
>
>>     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>
>> When I run the method with mpi and one core everything works fine.
>> Starting the same method with more cores, the sum of the matrices fails. It
>> seems like some values are added
>
>
>  Please send the full output of a failure when you use MAT_COPY_VALUES on
> 2 procs.
>
>    Thanks,
>
>       Matt
>
>
>> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE
>> leads to the right behavior in the multi-core case, but is very slow. I
>> checked with a viewer if all matrices have the same nonzero structure and
>> this is the case.
>>
>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>
>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0,
>> Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to
>> the current version and did not find any note about MatAXPY or MatAYPX.
>>
>>
>> Best and Thanks a lot for your help
>>
>> Klaus
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/63a15bf7/attachment-0001.html>

From balay at mcs.anl.gov  Fri Mar 27 10:27:46 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 27 Mar 2015 10:27:46 -0500
Subject: [petsc-users] running Petsc programs without a connection.
In-Reply-To: <CAMYG4G=XC9jLDFOcVXBNALGyCs61znQ09Ciiy9zVEUwq_E-w_Q@mail.gmail.com>
References: <E8F644D1675C06428BC49CCCDF77EEEC452F14E7@MBXP03.ds.man.ac.uk>
	<CAMYG4G=XC9jLDFOcVXBNALGyCs61znQ09Ciiy9zVEUwq_E-w_Q@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1503271024020.15095@asterix>

On Thu, 26 Mar 2015, Matthew Knepley wrote:

> On Thu, Mar 26, 2015 at 2:15 AM, Sanjay Kharche <
> Sanjay.Kharche at manchester.ac.uk> wrote:
> 
> >
> > Dear All
> >
> > I have a fedora pc which some times cannot have an internet connection. I
> > found that I can run non-Petsc MPI programs when it is not connected to the
> > internet. However, when I try to run my Petsc based program without a
> > connection, I get the following error. By googling a little bit, I found
> > that the mpi launcher requires to log into each of the 4 procs the pc has,
> > and may be using ssh. This or something else may or may not require an
> > internet connection. In my present installation of petsc, I have downloaded
> > and installed all required dependencies (blas, lapack, sundials) including
> > mpich during the configure. I am using a makefile that I took from the ts
> > examples for ex15.c where I changed the ex15 entry to sk2d (my program). I
> > updated the petsc independent mpich to be the same version as the one I got
> > with petsc.
> >
> > Is there a way of configuring or otherwise to get Petsc programs to run
> > when I do not have an internet connection?
> >
> > the error:
> >
> > mpiexec -n 3 ./sk2d
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > MPIR_Init_thread(498)..............:
> > MPID_Init(187).....................: channel initialization failed
> > MPIDI_CH3_Init(89).................:
> > MPID_nem_init(320).................:
> > MPID_nem_tcp_init(171).............:
> > MPID_nem_tcp_get_business_card(418):
> > MPID_nem_tcp_init(377).............: gethostbyname failed,
> > sanjayslaptop.maths.liv.ac.uk (errno 2)
> >
> 
> This is not PETSc, but rather the way MPICH is configured. I run using
> 
>   -n 3 -hosts localhost,localhost,localhost
> 
> and it will run fine disconnected.
> 
> Satish, is there a general fix?

Presumably non-petsc MPI programs are using a different install of MPI [openmpi?] that doesn't
have this problem - so one could install PETSc with that.

The other alternative is to configure the laptop/machine to use 'name' that works
with gethostbyname() all the time. I  do the following:

1. make sure the hostname is in /etc/hostname
$ cat /etc/hostname
asterix

[one is supporsed to use hostnamectl - but I don't really understand this command.
soemtimes the hostname is not saved across reboots]

2. Add an entry to /etc/hosts to resolve the set hostname. I have the following entry:

127.0.0.1   asterix

Satish


> 
>   Thanks,
> 
>      Matt
> 
> 
> >
> > cheers
> > Sanjay
> >
> 
> 
> 
> 


From bsmith at mcs.anl.gov  Fri Mar 27 10:57:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 27 Mar 2015 10:57:17 -0500
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <55156DA7.6010602@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
Message-ID: <0213B63B-0ACD-4074-A93C-6577163F4A99@mcs.anl.gov>


  Klaus,

    You would really benefit by upgrading to PETSc 3.5.3, we added much faster symbolic MatAXPY() for the DIFFERENT_NONZERO_STRUCTURE flag. Plus it is much easier for us to support the newest version.

  Barry

> On Mar 27, 2015, at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
> 
> Hallo,
> 
> I have a strange behavior in my code concerning the function MatAXPY. I create 3 different Matrices
> 
>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>    ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>    ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>    ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> 
>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>    ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>    ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>    ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> 
>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>    ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>    ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>    ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
> 
> and want to sum these three matrixes with different factors. First I fill the Matrix A with some values, and duplicate the structure of A to the other two matrices:
> 
>    MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>    MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>    MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>    MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>    MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>    MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>    MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>    MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
> 
> After this I fill the matrices At and Ah with some other values, which are not beside the non zero structure (I also tried with just copying the Matrix A). Now after another MatAssembly I want to add these Matrices in the form A+c*(Ah+d*At):
> 
>    MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>    MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
> 
> When I run the method with mpi and one core everything works fine. Starting the same method with more cores, the sum of the matrices fails. It seems like some values are added correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the multi-core case, but is very slow. I checked with a viewer if all matrices have the same nonzero structure and this is the case.
> 
> Does anyone know why this fails, or do I have made any wrong thoughts?
> 
> I'm corrently working with a petsc version (Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to the current version and did not find any note about MatAXPY or MatAYPX.
> 
> 
> Best and Thanks a lot for your help
> 
> Klaus
> 


From kaiser at igpm.rwth-aachen.de  Fri Mar 27 11:07:55 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Fri, 27 Mar 2015 17:07:55 +0100
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <CAMYG4G=OR3=gMHxAf2NMQCkkh7PpPDRBaoFZ+AeamiKaE2xapw@mail.gmail.com>
References: <55156DA7.6010602@igpm.rwth-aachen.de>	<CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>	<55157460.4080200@igpm.rwth-aachen.de>
	<CAMYG4G=OR3=gMHxAf2NMQCkkh7PpPDRBaoFZ+AeamiKaE2xapw@mail.gmail.com>
Message-ID: <5515805B.3060406@igpm.rwth-aachen.de>

Hallo Matthew,

here is a short example I used for 8 cores:

     int d_nnz[3];
     int o_nnz[3];
     d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
     o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
0, o_nnz,&A);
     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
0, o_nnz,&Ah);
     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
0, o_nnz,&At);

     std::vector<double> insert(3*3*3*3, 1.0);
     for(int i=0;i<8;++i)
     {
         int rows[3] = {i,i+1,i+3};
         int cols[3] = {i,i+1,i+3};

         MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES);
     }

     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
     MatDuplicate(A,MAT_COPY_VALUES,&Ah);
     MatDuplicate(A,MAT_COPY_VALUES,&At);
     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);

     MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
     MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);

     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);

The result is, that only some values are computed correctly.

Best and Thank you a lot

Klaus



On 03/27/2015 04:26 PM, Matthew Knepley wrote:
> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser 
> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>
>     Hallo Matthew,
>
>     thanks for your fast response. With "no beside the nonzero
>     structure" I meant, that I do not create a different non-zero
>     structure while I'm adding values to my matrices.
>
>     I also tried MAT_COPY_VALUES and the result was the same.
>
>
> Please send the small example and we will figure out what is going wrong.
>
>   Thanks,
>
>      Matt
>
>     I do not get a error message but the resulting matrix is wrong.
>     Here I have a short example. The first 9 rows and first 3 colomn
>     of the matrices Ah, At and Ah+At on the first processor (of 8):
>
>     Ah:
>     0:                   60.3553 -0.249975              2.77556e-17
>     1:                         0 60.3553                        0
>     2:                         0 0                  60.3553
>     3:                   17.6777 0.374962                 0.124987
>     4:                         0 17.6777                        0
>     5:                         0 0                  17.6777
>     6:                  -7.32233 -0.124987                 0.374962
>     7:                         0 -7.32233                        0
>     8:                         0 0                 -7.32233
>
>
>     At:
>     0:                         0 0                        0
>     1:                      2500 0                        0
>     2:              -4.54747e-13 0                        0
>     3:                         0 0                        0
>     4:                      1250 0                        0
>     5:                      1250 0                        0
>     6:                         0 0                        0
>     7:                      1250 0                        0
>     8:                      3750 0                        0
>
>     Ah+At
>     0:                   60.3553 -0.249975              2.77556e-17
>     1:                      2500 60.3553                        0
>     2:              -4.54747e-13 0                  60.3553
>     3:                   17.6777 0.374962                 0.124987
>     4:                         0 17.6777                        0
>     5:                         0 0                  17.6777
>     6:                  -7.32233 -0.124987                 0.374962
>     7:                         0 -7.32233                        0
>     8:                         0 0                 -7.32233
>
>     you can see the first 3 rows of the resulting matrix looks exactly
>     like what I would expect, but the last 6 rows only consists of the
>     values of Ah. When you would also look on the matrix A and A+Ah+At
>     you would also see, that values where both matrices have an
>     nonzero entry are not sum correctly.
>
>     Best
>
>     Klaus
>
>     On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>     On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser
>>     <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>>
>>     wrote:
>>
>>         Hallo,
>>
>>         I have a strange behavior in my code concerning the function
>>         MatAXPY. I create 3 different Matrices
>>
>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>         0, o_nnz,&A);
>>             ierr =
>>         MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>             ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>             ierr =
>>         MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>         0, o_nnz,&At);
>>             ierr =
>>         MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>             ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>             ierr =
>>         MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>         0, o_nnz,&Ah);
>>             ierr =
>>         MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>             ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>             ierr =
>>         MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>
>>     These creations are superfluous since you use MatDuplicate() below.
>>
>>         and want to sum these three matrixes with different factors.
>>         First I fill the Matrix A with some values, and duplicate the
>>         structure of A to the other two matrices:
>>
>>             MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>             MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>             MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>             MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>             MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>             MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>
>>
>>         After this I fill the matrices At and Ah with some other
>>         values, which are not beside the non zero structure (I also
>>         tried with just copying the Matrix A). Now after another
>>         MatAssembly
>>
>>
>>     I do not understand "no beside the nonzero structure". Do you
>>     mean that the nonzero structure is the same?
>>
>>     Can you first test with MAT_COPY_VALUES?
>>
>>         I want to add these Matrices in the form A+c*(Ah+d*At): 
>>
>>
>>             MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>             MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>
>>         When I run the method with mpi and one core everything works
>>         fine. Starting the same method with more cores, the sum of
>>         the matrices fails. It seems like some values are added
>>
>>
>>     Please send the full output of a failure when you use
>>     MAT_COPY_VALUES on 2 procs.
>>
>>       Thanks,
>>
>>          Matt
>>
>>         correctly and many values are missed. Using
>>         DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in
>>         the multi-core case, but is very slow. I checked with a
>>         viewer if all matrices have the same nonzero structure and
>>         this is the case.
>>
>>         Does anyone know why this fails, or do I have made any wrong
>>         thoughts?
>>
>>         I'm corrently working with a petsc version (Petsc Release
>>         Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I
>>         looked into the changelogs up to the current version and did
>>         not find any note about MatAXPY or MatAYPX.
>>
>>
>>         Best and Thanks a lot for your help
>>
>>         Klaus
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150327/fb4027e6/attachment-0001.html>

From kaiser at igpm.rwth-aachen.de  Fri Mar 27 11:14:32 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Fri, 27 Mar 2015 17:14:32 +0100
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <0213B63B-0ACD-4074-A93C-6577163F4A99@mcs.anl.gov>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<0213B63B-0ACD-4074-A93C-6577163F4A99@mcs.anl.gov>
Message-ID: <551581E8.4080109@igpm.rwth-aachen.de>

Hallo Barry,

I think you're right, but this is the setup I used for more than a year, 
so I first want to check if there is an error in my code and then switch 
with an correct code to a newer version when I do have the time for the 
installation.

Is the faster symbolic MatAXPY for the DIFFERENT_NONZERO_STRUCTURE flag 
faster than using SAME_NONZERO_STRUCTURE?

Best and Thanks a lot

Klaus

On 03/27/2015 04:57 PM, Barry Smith wrote:
>    Klaus,
>
>      You would really benefit by upgrading to PETSc 3.5.3, we added much faster symbolic MatAXPY() for the DIFFERENT_NONZERO_STRUCTURE flag. Plus it is much easier for us to support the newest version.
>
>    Barry
>
>> On Mar 27, 2015, at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
>>
>> Hallo,
>>
>> I have a strange behavior in my code concerning the function MatAXPY. I create 3 different Matrices
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>
>> and want to sum these three matrixes with different factors. First I fill the Matrix A with some values, and duplicate the structure of A to the other two matrices:
>>
>>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>
>> After this I fill the matrices At and Ah with some other values, which are not beside the non zero structure (I also tried with just copying the Matrix A). Now after another MatAssembly I want to add these Matrices in the form A+c*(Ah+d*At):
>>
>>     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>
>> When I run the method with mpi and one core everything works fine. Starting the same method with more cores, the sum of the matrices fails. It seems like some values are added correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the multi-core case, but is very slow. I checked with a viewer if all matrices have the same nonzero structure and this is the case.
>>
>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>
>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to the current version and did not find any note about MatAXPY or MatAYPX.
>>
>>
>> Best and Thanks a lot for your help
>>
>> Klaus
>>


From bsmith at mcs.anl.gov  Fri Mar 27 12:32:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 27 Mar 2015 12:32:57 -0500
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <551581E8.4080109@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<0213B63B-0ACD-4074-A93C-6577163F4A99@mcs.anl.gov>
	<551581E8.4080109@igpm.rwth-aachen.de>
Message-ID: <FF765675-6957-4597-892A-D87BC6507914@mcs.anl.gov>


> On Mar 27, 2015, at 11:14 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
> 
> Hallo Barry,
> 
> I think you're right, but this is the setup I used for more than a year, so I first want to check if there is an error in my code and then switch with an correct code to a newer version when I do have the time for the installation.
> 
> Is the faster symbolic MatAXPY for the DIFFERENT_NONZERO_STRUCTURE flag faster than using SAME_NONZERO_STRUCTURE?

  No
> 
> Best and Thanks a lot
> 
> Klaus
> 
> On 03/27/2015 04:57 PM, Barry Smith wrote:
>>   Klaus,
>> 
>>     You would really benefit by upgrading to PETSc 3.5.3, we added much faster symbolic MatAXPY() for the DIFFERENT_NONZERO_STRUCTURE flag. Plus it is much easier for us to support the newest version.
>> 
>>   Barry
>> 
>>> On Mar 27, 2015, at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
>>> 
>>> Hallo,
>>> 
>>> I have a strange behavior in my code concerning the function MatAXPY. I create 3 different Matrices
>>> 
>>>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>>    ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>    ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>    ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>> 
>>>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>>    ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>    ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>    ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>> 
>>>    ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>>    ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>    ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>    ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>> 
>>> and want to sum these three matrixes with different factors. First I fill the Matrix A with some values, and duplicate the structure of A to the other two matrices:
>>> 
>>>    MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>>    MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>>    MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>>    MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>>    MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>>    MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>>    MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>>    MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>> 
>>> After this I fill the matrices At and Ah with some other values, which are not beside the non zero structure (I also tried with just copying the Matrix A). Now after another MatAssembly I want to add these Matrices in the form A+c*(Ah+d*At):
>>> 
>>>    MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>>    MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>> 
>>> When I run the method with mpi and one core everything works fine. Starting the same method with more cores, the sum of the matrices fails. It seems like some values are added correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the multi-core case, but is very slow. I checked with a viewer if all matrices have the same nonzero structure and this is the case.
>>> 
>>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>> 
>>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to the current version and did not find any note about MatAXPY or MatAYPX.
>>> 
>>> 
>>> Best and Thanks a lot for your help
>>> 
>>> Klaus
>>> 
> 


From ansp6066 at colorado.edu  Sat Mar 28 15:24:31 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Sat, 28 Mar 2015 13:24:31 -0700 (PDT)
Subject: [petsc-users] In place access to eigenvectors of EPS
Message-ID: <1427574270917.56b2d1bd@Nodemailer>

Is there a way to get in place access to the eigenvectors in the BV contained by EPS using the public API?


-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150328/4ce3e825/attachment.html>

From jroman at dsic.upv.es  Sat Mar 28 19:03:26 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Sun, 29 Mar 2015 01:03:26 +0100
Subject: [petsc-users] In place access to eigenvectors of EPS
In-Reply-To: <1427574270917.56b2d1bd@Nodemailer>
References: <1427574270917.56b2d1bd@Nodemailer>
Message-ID: <28B0565F-9B8E-4BDC-91CD-A94C16AF6DD2@dsic.upv.es>


El 28/03/2015, a las 21:24, Andrew Spott escribi?:

> Is there a way to get in place access to the eigenvectors in the BV contained by EPS using the public API?
> 
> -Andrew
> 

You can do EPSGetBV() and then BVGetColumn() for each of the first nconv columns. But this has risks, for instance the BV may contain Schur vectors instead of eigenvectors, or separated real and imaginary parts in case of complex eigenvectors in real arithmetic, an possibly other solver-dependent issues.

What is the use case for this?
Jose


From bsmith at mcs.anl.gov  Sat Mar 28 21:59:03 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 28 Mar 2015 21:59:03 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <5512D5FC.7090108@hakostra.net>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
Message-ID: <CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>


  Hakon,

    I have pushed to the branch barry/feature-hdf5-flexible-initial-dimension and next the change so that Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT have the extra dimension if BS == 1. To add that extra dimension even when bs == 1 with VecView() or to handle a file with that extra dimension with VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or -viewer_hdf5_base_dimension2 true

   Please try it out and let me know if you have any trouble

   Thanks

  Barry

  Matt,

   Scream as much as you want but adding that extra dimension automatically is not intuitive so it cannot be the default. 

> On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> 
> Did you come to any conclusion on this issue?
> 
> Regards,
> H?kon
> 
> On 20. mars 2015 22:02, H?kon Strandenes wrote:
>> On 20. mars 2015 20:48, Barry Smith wrote:
>>> Why is 1 dimension a special case that is not worthy of its own
>>> format? The same thing would hold for 2d and 3d. One could then argue
>>> that we should have a single six dimensional format for the files for
>>> all vectors that PETSc produces. Then a 1d problem has five of the
>>> dimensions being 1.
>> 
>> This is a very good point, and support my view.
>> 
>> Let me come with two very simple example cases:
>> 
>> 
>> Case 1:
>> Create a list of grid points in an external preprocessor for the purpose
>> of loading this into a Vec later:
>> 
>>   x = np.linspace(0.0, 1.0, num=21)
>>   f.create_dataset('/MESH/nodes/x', data=x)
>> 
>> vs.
>> 
>>   x = np.linspace(0.0, 1.0, num=21)
>>   x = x.reshape((21,1))
>>   f.create_dataset('/MESH/nodes/x', data=x)
>> 
>> 
>> Case 2:
>> Read three Vecs written to disk by PETSc, and calculate total "bounding
>> box volume" of the grid:
>> 
>>   g = h5py.File('grid.h5', 'r')
>>   x = g['/MESH/nodes/x']
>>   y = g['/MESH/nodes/y']
>>   z = g['/MESH/nodes/z']
>>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
>> 
>> vs.
>> 
>>   g = h5py.File('grid.h5', 'r')
>>   x = g['/MESH/nodes/x'][:,0]
>>   y = g['/MESH/nodes/y'][:,0]
>>   z = g['/MESH/nodes/z'][:,0]
>>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
>> 
>> 
>> In both cases I think handling this extra, unnecessary dimension makes
>> the code less attractive. It's not that either way is difficult,
>> problematic or impossible, but it's just that 1D Vecs should intuitively
>> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
>> quite a while until I figured this out, even after having written an
>> entire Navier-Stokes DNS solver using the PETSc library for everything
>> except time integration and filling these simple 1D coordinate arrays!
>> 
>> Regards,
>> H?kon


From knepley at gmail.com  Sat Mar 28 22:04:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 28 Mar 2015 22:04:09 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
Message-ID: <CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>

On Sat, Mar 28, 2015 at 9:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Hakon,
>
>     I have pushed to the branch
> barry/feature-hdf5-flexible-initial-dimension and next the change so that
> Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT
> have the extra dimension if BS == 1. To add that extra dimension even when
> bs == 1 with VecView() or to handle a file with that extra dimension with
> VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or
> -viewer_hdf5_base_dimension2 true
>
>    Please try it out and let me know if you have any trouble
>
>    Thanks
>
>   Barry
>
>   Matt,
>
>    Scream as much as you want but adding that extra dimension
> automatically is not intuitive so it cannot be the default.


Interfaces should be intuitive,  file formats should be consistent. This
gets that entirely wrong.

   Matt


> > On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> >
> > Did you come to any conclusion on this issue?
> >
> > Regards,
> > H?kon
> >
> > On 20. mars 2015 22:02, H?kon Strandenes wrote:
> >> On 20. mars 2015 20:48, Barry Smith wrote:
> >>> Why is 1 dimension a special case that is not worthy of its own
> >>> format? The same thing would hold for 2d and 3d. One could then argue
> >>> that we should have a single six dimensional format for the files for
> >>> all vectors that PETSc produces. Then a 1d problem has five of the
> >>> dimensions being 1.
> >>
> >> This is a very good point, and support my view.
> >>
> >> Let me come with two very simple example cases:
> >>
> >>
> >> Case 1:
> >> Create a list of grid points in an external preprocessor for the purpose
> >> of loading this into a Vec later:
> >>
> >>   x = np.linspace(0.0, 1.0, num=21)
> >>   f.create_dataset('/MESH/nodes/x', data=x)
> >>
> >> vs.
> >>
> >>   x = np.linspace(0.0, 1.0, num=21)
> >>   x = x.reshape((21,1))
> >>   f.create_dataset('/MESH/nodes/x', data=x)
> >>
> >>
> >> Case 2:
> >> Read three Vecs written to disk by PETSc, and calculate total "bounding
> >> box volume" of the grid:
> >>
> >>   g = h5py.File('grid.h5', 'r')
> >>   x = g['/MESH/nodes/x']
> >>   y = g['/MESH/nodes/y']
> >>   z = g['/MESH/nodes/z']
> >>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
> >>
> >> vs.
> >>
> >>   g = h5py.File('grid.h5', 'r')
> >>   x = g['/MESH/nodes/x'][:,0]
> >>   y = g['/MESH/nodes/y'][:,0]
> >>   z = g['/MESH/nodes/z'][:,0]
> >>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
> >>
> >>
> >> In both cases I think handling this extra, unnecessary dimension makes
> >> the code less attractive. It's not that either way is difficult,
> >> problematic or impossible, but it's just that 1D Vecs should intuitively
> >> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> >> quite a while until I figured this out, even after having written an
> >> entire Navier-Stokes DNS solver using the PETSc library for everything
> >> except time integration and filling these simple 1D coordinate arrays!
> >>
> >> Regards,
> >> H?kon
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150328/a9137afd/attachment.html>

From bsmith at mcs.anl.gov  Sat Mar 28 22:36:27 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 28 Mar 2015 22:36:27 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
Message-ID: <4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>


> On Mar 28, 2015, at 10:04 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Mar 28, 2015 at 9:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Hakon,
> 
>     I have pushed to the branch barry/feature-hdf5-flexible-initial-dimension and next the change so that Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT have the extra dimension if BS == 1. To add that extra dimension even when bs == 1 with VecView() or to handle a file with that extra dimension with VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or -viewer_hdf5_base_dimension2 true
> 
>    Please try it out and let me know if you have any trouble
> 
>    Thanks
> 
>   Barry
> 
>   Matt,
> 
>    Scream as much as you want but adding that extra dimension automatically is not intuitive so it cannot be the default.
> 
> Interfaces should be intuitive,  file formats should be consistent.

  Hakon sent specific examples where, because of the file format, the user interface is extremely non-intuitive (below). A user loading up a "plain old vector" from a file format expects 1 a one dimension beast and that is what we should deliver to them. Now you could argue that the HDF5 interface sucks, because it bleeds the file format through the interface, that could be true, I don't care, that is what we are stuck with.

  Barry



> This gets that entirely wrong.
> 
>    Matt
>  
> > On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> >
> > Did you come to any conclusion on this issue?
> >
> > Regards,
> > H?kon
> >
> > On 20. mars 2015 22:02, H?kon Strandenes wrote:
> >> On 20. mars 2015 20:48, Barry Smith wrote:
> >>> Why is 1 dimension a special case that is not worthy of its own
> >>> format? The same thing would hold for 2d and 3d. One could then argue
> >>> that we should have a single six dimensional format for the files for
> >>> all vectors that PETSc produces. Then a 1d problem has five of the
> >>> dimensions being 1.
> >>
> >> This is a very good point, and support my view.
> >>
> >> Let me come with two very simple example cases:
> >>
> >>
> >> Case 1:
> >> Create a list of grid points in an external preprocessor for the purpose
> >> of loading this into a Vec later:
> >>
> >>   x = np.linspace(0.0, 1.0, num=21)
> >>   f.create_dataset('/MESH/nodes/x', data=x)
> >>
> >> vs.
> >>
> >>   x = np.linspace(0.0, 1.0, num=21)
> >>   x = x.reshape((21,1))
> >>   f.create_dataset('/MESH/nodes/x', data=x)
> >>
> >>
> >> Case 2:
> >> Read three Vecs written to disk by PETSc, and calculate total "bounding
> >> box volume" of the grid:
> >>
> >>   g = h5py.File('grid.h5', 'r')
> >>   x = g['/MESH/nodes/x']
> >>   y = g['/MESH/nodes/y']
> >>   z = g['/MESH/nodes/z']
> >>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
> >>
> >> vs.
> >>
> >>   g = h5py.File('grid.h5', 'r')
> >>   x = g['/MESH/nodes/x'][:,0]
> >>   y = g['/MESH/nodes/y'][:,0]
> >>   z = g['/MESH/nodes/z'][:,0]
> >>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
> >>
> >>
> >> In both cases I think handling this extra, unnecessary dimension makes
> >> the code less attractive. It's not that either way is difficult,
> >> problematic or impossible, but it's just that 1D Vecs should intuitively
> >> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> >> quite a while until I figured this out, even after having written an
> >> entire Navier-Stokes DNS solver using the PETSc library for everything
> >> except time integration and filling these simple 1D coordinate arrays!
> >>
> >> Regards,
> >> H?kon
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From ansp6066 at colorado.edu  Sat Mar 28 23:02:36 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Sat, 28 Mar 2015 21:02:36 -0700 (PDT)
Subject: [petsc-users] In place access to eigenvectors of EPS
In-Reply-To: <28B0565F-9B8E-4BDC-91CD-A94C16AF6DD2@dsic.upv.es>
References: <28B0565F-9B8E-4BDC-91CD-A94C16AF6DD2@dsic.upv.es>
Message-ID: <1427601755657.004410ca@Nodemailer>

>What is the use case for this?


Orthogonalization between left and right eigenvectors. ?I?ve got two EPS instances, one for the left and one for the right eigenvectors, and I need to orthogonalize them with respect to each other: enforce L^H R = 1. ?Copying every vector out of both BVs seems wasteful. ?Especially because I don?t really need the EPS to do anything with them after this.




I noticed in the source for EPSGetEigenvector that there is a ?permutation? array that might change the order of the vectors in the BV. ?Is there a way to get this using the public API?




It appears that I can call ?EPSGetEigenvector? to make sure that ?EPSComputeVectors? is called, so I get eigenvectors instead of Shur vectors or other things. ?But I?m not sure how to get the permutation back.




-Andrew

On Sat, Mar 28, 2015 at 6:03 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> El 28/03/2015, a las 21:24, Andrew Spott escribi?:
>> Is there a way to get in place access to the eigenvectors in the BV contained by EPS using the public API?
>> 
>> -Andrew
>> 
> You can do EPSGetBV() and then BVGetColumn() for each of the first nconv columns. But this has risks, for instance the BV may contain Schur vectors instead of eigenvectors, or separated real and imaginary parts in case of complex eigenvectors in real arithmetic, an possibly other solver-dependent issues.
> What is the use case for this?
> Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150328/c675d4d3/attachment-0001.html>

From jroman at dsic.upv.es  Sun Mar 29 05:34:11 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Sun, 29 Mar 2015 12:34:11 +0200
Subject: [petsc-users] In place access to eigenvectors of EPS
In-Reply-To: <1427601755657.004410ca@Nodemailer>
References: <28B0565F-9B8E-4BDC-91CD-A94C16AF6DD2@dsic.upv.es>
	<1427601755657.004410ca@Nodemailer>
Message-ID: <A163BBCB-59D3-41AF-BEE6-92113BC1E289@dsic.upv.es>


El 29/03/2015, a las 06:02, Andrew Spott escribi?:

> >What is the use case for this?
> 
> Orthogonalization between left and right eigenvectors.  I?ve got two EPS instances, one for the left and one for the right eigenvectors, and I need to orthogonalize them with respect to each other: enforce L^H R = 1.  Copying every vector out of both BVs seems wasteful.  Especially because I don?t really need the EPS to do anything with them after this.
> 
> I noticed in the source for EPSGetEigenvector that there is a ?permutation? array that might change the order of the vectors in the BV.  Is there a way to get this using the public API?
> 
> It appears that I can call ?EPSGetEigenvector? to make sure that ?EPSComputeVectors? is called, so I get eigenvectors instead of Shur vectors or other things.  But I?m not sure how to get the permutation back.
> 
> -Andrew
> 

No, there is currently no way of getting the permutation out of EPS.

I don't think it is wasteful to copy all eigenvectors out of EPS. This is done only once, and after that you can destroy the EPS.

Jose


From knepley at gmail.com  Sun Mar 29 06:40:05 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 29 Mar 2015 06:40:05 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
	<4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
Message-ID: <CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>

On Sat, Mar 28, 2015 at 10:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 28, 2015, at 10:04 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Sat, Mar 28, 2015 at 9:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Hakon,
> >
> >     I have pushed to the branch
> barry/feature-hdf5-flexible-initial-dimension and next the change so that
> Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT
> have the extra dimension if BS == 1. To add that extra dimension even when
> bs == 1 with VecView() or to handle a file with that extra dimension with
> VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or
> -viewer_hdf5_base_dimension2 true
> >
> >    Please try it out and let me know if you have any trouble
> >
> >    Thanks
> >
> >   Barry
> >
> >   Matt,
> >
> >    Scream as much as you want but adding that extra dimension
> automatically is not intuitive so it cannot be the default.
> >
> > Interfaces should be intuitive,  file formats should be consistent.
>
>   Hakon sent specific examples where, because of the file format, the user
> interface is extremely non-intuitive (below). A user loading up a "plain
> old vector" from a file format expects 1 a one dimension beast and that is
> what we should deliver to them. Now you could argue that the HDF5 interface
> sucks, because it bleeds the file format through the interface, that could
> be true, I don't care, that is what we are stuck with.


But you changed the PETSc output format in response, which is not necessary.

We could put the special case in the reader, so that he could load the 1D
vector he expects, or we could load the 2D vector from the PETSc format.
He gets the intuitive load he wants, but we get a consistent format.

Now he might get what he expects from View(), which I am not sure he was
asking for, but we have to special case all our tools which manipulate
the PETSc format.

   Matt


>
>   Barry
>
>
>
> > This gets that entirely wrong.
> >
> >    Matt
> >
> > > On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> > >
> > > Did you come to any conclusion on this issue?
> > >
> > > Regards,
> > > H?kon
> > >
> > > On 20. mars 2015 22:02, H?kon Strandenes wrote:
> > >> On 20. mars 2015 20:48, Barry Smith wrote:
> > >>> Why is 1 dimension a special case that is not worthy of its own
> > >>> format? The same thing would hold for 2d and 3d. One could then argue
> > >>> that we should have a single six dimensional format for the files for
> > >>> all vectors that PETSc produces. Then a 1d problem has five of the
> > >>> dimensions being 1.
> > >>
> > >> This is a very good point, and support my view.
> > >>
> > >> Let me come with two very simple example cases:
> > >>
> > >>
> > >> Case 1:
> > >> Create a list of grid points in an external preprocessor for the
> purpose
> > >> of loading this into a Vec later:
> > >>
> > >>   x = np.linspace(0.0, 1.0, num=21)
> > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > >>
> > >> vs.
> > >>
> > >>   x = np.linspace(0.0, 1.0, num=21)
> > >>   x = x.reshape((21,1))
> > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > >>
> > >>
> > >> Case 2:
> > >> Read three Vecs written to disk by PETSc, and calculate total
> "bounding
> > >> box volume" of the grid:
> > >>
> > >>   g = h5py.File('grid.h5', 'r')
> > >>   x = g['/MESH/nodes/x']
> > >>   y = g['/MESH/nodes/y']
> > >>   z = g['/MESH/nodes/z']
> > >>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
> > >>
> > >> vs.
> > >>
> > >>   g = h5py.File('grid.h5', 'r')
> > >>   x = g['/MESH/nodes/x'][:,0]
> > >>   y = g['/MESH/nodes/y'][:,0]
> > >>   z = g['/MESH/nodes/z'][:,0]
> > >>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
> > >>
> > >>
> > >> In both cases I think handling this extra, unnecessary dimension makes
> > >> the code less attractive. It's not that either way is difficult,
> > >> problematic or impossible, but it's just that 1D Vecs should
> intuitively
> > >> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> > >> quite a while until I figured this out, even after having written an
> > >> entire Navier-Stokes DNS solver using the PETSc library for everything
> > >> except time integration and filling these simple 1D coordinate arrays!
> > >>
> > >> Regards,
> > >> H?kon
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150329/93117db8/attachment.html>

From haakon at hakostra.net  Sun Mar 29 07:57:58 2015
From: haakon at hakostra.net (=?UTF-8?Q?H=C3=A5kon_Strandenes?=)
Date: Sun, 29 Mar 2015 14:57:58 +0200
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
Message-ID: <a0359623238694d088fee469afa16cec@webmail.domeneshop.no>

Thanks. Unfortuneatly, I am not in my office until april 7th or 8th, and 
am not able to test this out until then.

I have also seen the other replies to this post. I might have some ideas 
on how to solve this in a bit more flexible way, but again, I cannot 
work on this until I am back in office.

I will follow this up as soon as I am back. Thanks for your help so far.

Regards,
H?kon Strandenes


Den 2015-03-29 04:59, skrev Barry Smith:
> Hakon,
> 
>     I have pushed to the branch
> barry/feature-hdf5-flexible-initial-dimension and next the change so
> that Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA
> will NOT have the extra dimension if BS == 1. To add that extra
> dimension even when bs == 1 with VecView() or to handle a file with
> that extra dimension with VecLoad() one must call
> PetscViewerHDF5SetBaseDimension2() or -viewer_hdf5_base_dimension2
> true
> 
>    Please try it out and let me know if you have any trouble
> 
>    Thanks
> 
>   Barry
> 
>   Matt,
> 
>    Scream as much as you want but adding that extra dimension
> automatically is not intuitive so it cannot be the default.
> 
>> On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net> 
>> wrote:
>> 
>> Did you come to any conclusion on this issue?
>> 
>> Regards,
>> H?kon
>> 
>> On 20. mars 2015 22:02, H?kon Strandenes wrote:
>>> On 20. mars 2015 20:48, Barry Smith wrote:
>>>> Why is 1 dimension a special case that is not worthy of its own
>>>> format? The same thing would hold for 2d and 3d. One could then 
>>>> argue
>>>> that we should have a single six dimensional format for the files 
>>>> for
>>>> all vectors that PETSc produces. Then a 1d problem has five of the
>>>> dimensions being 1.
>>> 
>>> This is a very good point, and support my view.
>>> 
>>> Let me come with two very simple example cases:
>>> 
>>> 
>>> Case 1:
>>> Create a list of grid points in an external preprocessor for the 
>>> purpose
>>> of loading this into a Vec later:
>>> 
>>>   x = np.linspace(0.0, 1.0, num=21)
>>>   f.create_dataset('/MESH/nodes/x', data=x)
>>> 
>>> vs.
>>> 
>>>   x = np.linspace(0.0, 1.0, num=21)
>>>   x = x.reshape((21,1))
>>>   f.create_dataset('/MESH/nodes/x', data=x)
>>> 
>>> 
>>> Case 2:
>>> Read three Vecs written to disk by PETSc, and calculate total 
>>> "bounding
>>> box volume" of the grid:
>>> 
>>>   g = h5py.File('grid.h5', 'r')
>>>   x = g['/MESH/nodes/x']
>>>   y = g['/MESH/nodes/y']
>>>   z = g['/MESH/nodes/z']
>>>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
>>> 
>>> vs.
>>> 
>>>   g = h5py.File('grid.h5', 'r')
>>>   x = g['/MESH/nodes/x'][:,0]
>>>   y = g['/MESH/nodes/y'][:,0]
>>>   z = g['/MESH/nodes/z'][:,0]
>>>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
>>> 
>>> 
>>> In both cases I think handling this extra, unnecessary dimension 
>>> makes
>>> the code less attractive. It's not that either way is difficult,
>>> problematic or impossible, but it's just that 1D Vecs should 
>>> intuitively
>>> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
>>> quite a while until I figured this out, even after having written an
>>> entire Navier-Stokes DNS solver using the PETSc library for 
>>> everything
>>> except time integration and filling these simple 1D coordinate 
>>> arrays!
>>> 
>>> Regards,
>>> H?kon

From bsmith at mcs.anl.gov  Sun Mar 29 14:41:53 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 29 Mar 2015 14:41:53 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
	<4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
	<CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>
Message-ID: <5B59CECC-7A7F-4B4C-B13E-98DDAB29C625@mcs.anl.gov>


> On Mar 29, 2015, at 6:40 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Mar 28, 2015 at 10:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Mar 28, 2015, at 10:04 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Sat, Mar 28, 2015 at 9:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Hakon,
> >
> >     I have pushed to the branch barry/feature-hdf5-flexible-initial-dimension and next the change so that Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT have the extra dimension if BS == 1. To add that extra dimension even when bs == 1 with VecView() or to handle a file with that extra dimension with VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or -viewer_hdf5_base_dimension2 true
> >
> >    Please try it out and let me know if you have any trouble
> >
> >    Thanks
> >
> >   Barry
> >
> >   Matt,
> >
> >    Scream as much as you want but adding that extra dimension automatically is not intuitive so it cannot be the default.
> >
> > Interfaces should be intuitive,  file formats should be consistent.
> 
>   Hakon sent specific examples where, because of the file format, the user interface is extremely non-intuitive (below). A user loading up a "plain old vector" from a file format expects 1 a one dimension beast and that is what we should deliver to them. Now you could argue that the HDF5 interface sucks, because it bleeds the file format through the interface, that could be true, I don't care, that is what we are stuck with.
> 
> But you changed the PETSc output format in response, which is not necessary.
> 
> We could put the special case in the reader,

   Matt, 

     The problem is that we are not providing "the reader", nor could we!  "The reader" is whatever of many many tools that the user is using to read from the HDF5 file. It might be Matlab, python, a Fortran program, a Ruby program, whatever. And the "reader code" that user has to write is directly dependent on the format used in the file. In his example

> g = h5py.File('grid.h5', 'r')
> > >>   x = g['/MESH/nodes/x'][:,0]

  How do we provide this code to the user? Or are you saying we have to provide PETSc specific HDF5 readers for all packages and configurations? That is totally unrealistic and even if we did provide them no one would use them. The entire point of HDF5 is you can write your own readers, you are not stuck with using only readers provided by someone who provided the data set. There is not, and should not be a PETSc specific HDF5 format with its own readers and writers.

  Barry

  

> so that he could load the 1D vector he expects, or we could load the 2D vector from the PETSc format.
> He gets the intuitive load he wants, but we get a consistent format.
> 
> Now he might get what he expects from View(), which I am not sure he was asking for, but we have to special case all our tools which manipulate
> the PETSc format.
> 
>    Matt
>  
> 
>   Barry
> 
> 
> 
> > This gets that entirely wrong.
> >
> >    Matt
> >
> > > On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net> wrote:
> > >
> > > Did you come to any conclusion on this issue?
> > >
> > > Regards,
> > > H?kon
> > >
> > > On 20. mars 2015 22:02, H?kon Strandenes wrote:
> > >> On 20. mars 2015 20:48, Barry Smith wrote:
> > >>> Why is 1 dimension a special case that is not worthy of its own
> > >>> format? The same thing would hold for 2d and 3d. One could then argue
> > >>> that we should have a single six dimensional format for the files for
> > >>> all vectors that PETSc produces. Then a 1d problem has five of the
> > >>> dimensions being 1.
> > >>
> > >> This is a very good point, and support my view.
> > >>
> > >> Let me come with two very simple example cases:
> > >>
> > >>
> > >> Case 1:
> > >> Create a list of grid points in an external preprocessor for the purpose
> > >> of loading this into a Vec later:
> > >>
> > >>   x = np.linspace(0.0, 1.0, num=21)
> > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > >>
> > >> vs.
> > >>
> > >>   x = np.linspace(0.0, 1.0, num=21)
> > >>   x = x.reshape((21,1))
> > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > >>
> > >>
> > >> Case 2:
> > >> Read three Vecs written to disk by PETSc, and calculate total "bounding
> > >> box volume" of the grid:
> > >>
> > >>   g = h5py.File('grid.h5', 'r')
> > >>   x = g['/MESH/nodes/x']
> > >>   y = g['/MESH/nodes/y']
> > >>   z = g['/MESH/nodes/z']
> > >>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
> > >>
> > >> vs.
> > >>
> > >>   g = h5py.File('grid.h5', 'r')
> > >>   x = g['/MESH/nodes/x'][:,0]
> > >>   y = g['/MESH/nodes/y'][:,0]
> > >>   z = g['/MESH/nodes/z'][:,0]
> > >>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
> > >>
> > >>
> > >> In both cases I think handling this extra, unnecessary dimension makes
> > >> the code less attractive. It's not that either way is difficult,
> > >> problematic or impossible, but it's just that 1D Vecs should intuitively
> > >> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> > >> quite a while until I figured this out, even after having written an
> > >> entire Navier-Stokes DNS solver using the PETSc library for everything
> > >> except time integration and filling these simple 1D coordinate arrays!
> > >>
> > >> Regards,
> > >> H?kon
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From knepley at gmail.com  Sun Mar 29 14:48:21 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 29 Mar 2015 14:48:21 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <5B59CECC-7A7F-4B4C-B13E-98DDAB29C625@mcs.anl.gov>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
	<4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
	<CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>
	<5B59CECC-7A7F-4B4C-B13E-98DDAB29C625@mcs.anl.gov>
Message-ID: <CAMYG4GkQJ=n=u_jkAzsz_zxmAWR=Fyq6wP-jGGESj1oT4=p8XA@mail.gmail.com>

On Sun, Mar 29, 2015 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 29, 2015, at 6:40 AM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Sat, Mar 28, 2015 at 10:36 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Mar 28, 2015, at 10:04 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > >
> > > On Sat, Mar 28, 2015 at 9:59 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >   Hakon,
> > >
> > >     I have pushed to the branch
> barry/feature-hdf5-flexible-initial-dimension and next the change so that
> Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT
> have the extra dimension if BS == 1. To add that extra dimension even when
> bs == 1 with VecView() or to handle a file with that extra dimension with
> VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or
> -viewer_hdf5_base_dimension2 true
> > >
> > >    Please try it out and let me know if you have any trouble
> > >
> > >    Thanks
> > >
> > >   Barry
> > >
> > >   Matt,
> > >
> > >    Scream as much as you want but adding that extra dimension
> automatically is not intuitive so it cannot be the default.
> > >
> > > Interfaces should be intuitive,  file formats should be consistent.
> >
> >   Hakon sent specific examples where, because of the file format, the
> user interface is extremely non-intuitive (below). A user loading up a
> "plain old vector" from a file format expects 1 a one dimension beast and
> that is what we should deliver to them. Now you could argue that the HDF5
> interface sucks, because it bleeds the file format through the interface,
> that could be true, I don't care, that is what we are stuck with.
> >
> > But you changed the PETSc output format in response, which is not
> necessary.
> >
> > We could put the special case in the reader,
>
>    Matt,
>
>      The problem is that we are not providing "the reader", nor could we!
> "The reader" is whatever of many many tools that the user is using to read
> from the HDF5 file. It might be Matlab, python, a Fortran program, a Ruby
> program, whatever. And the "reader code" that user has to write is directly
> dependent on the format used in the file. In his example
>
> > g = h5py.File('grid.h5', 'r')
> > > >>   x = g['/MESH/nodes/x'][:,0]
>
>   How do we provide this code to the user? Or are you saying we have to
> provide PETSc specific HDF5 readers for all packages and configurations?
> That is totally unrealistic and even if we did provide them no one would
> use them. The entire point of HDF5 is you can write your own readers, you
> are not stuck with using only readers provided by someone who provided the
> data set.




> There is not, and should not be a PETSc specific HDF5 format with its own
> readers and writers.


But of course there already is a PETSc-specific HDF5 format. Someone made
up "/MESH/nodes/x". This is the essence of HDF5,
XML, etc. that they are not actually formats, but rather storage
technologies like the UNIX filesystem. Someone still has to declare
a format, and we do.

However, if you want the data array to be 1D in any old HDF5 tool, except
when complex is used, except when timestepping is used,
then Yes, we need to make an exception for blocksize. I just think we will
end up changing this back when it becomes apparent that
having to put a check in all these HDF5 tools as well is onerous.

   Matt


>
>   Barry
>
>
>
> > so that he could load the 1D vector he expects, or we could load the 2D
> vector from the PETSc format.
> > He gets the intuitive load he wants, but we get a consistent format.
> >
> > Now he might get what he expects from View(), which I am not sure he was
> asking for, but we have to special case all our tools which manipulate
> > the PETSc format.
> >
> >    Matt
> >
> >
> >   Barry
> >
> >
> >
> > > This gets that entirely wrong.
> > >
> > >    Matt
> > >
> > > > On Mar 25, 2015, at 10:36 AM, H?kon Strandenes <haakon at hakostra.net>
> wrote:
> > > >
> > > > Did you come to any conclusion on this issue?
> > > >
> > > > Regards,
> > > > H?kon
> > > >
> > > > On 20. mars 2015 22:02, H?kon Strandenes wrote:
> > > >> On 20. mars 2015 20:48, Barry Smith wrote:
> > > >>> Why is 1 dimension a special case that is not worthy of its own
> > > >>> format? The same thing would hold for 2d and 3d. One could then
> argue
> > > >>> that we should have a single six dimensional format for the files
> for
> > > >>> all vectors that PETSc produces. Then a 1d problem has five of the
> > > >>> dimensions being 1.
> > > >>
> > > >> This is a very good point, and support my view.
> > > >>
> > > >> Let me come with two very simple example cases:
> > > >>
> > > >>
> > > >> Case 1:
> > > >> Create a list of grid points in an external preprocessor for the
> purpose
> > > >> of loading this into a Vec later:
> > > >>
> > > >>   x = np.linspace(0.0, 1.0, num=21)
> > > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > > >>
> > > >> vs.
> > > >>
> > > >>   x = np.linspace(0.0, 1.0, num=21)
> > > >>   x = x.reshape((21,1))
> > > >>   f.create_dataset('/MESH/nodes/x', data=x)
> > > >>
> > > >>
> > > >> Case 2:
> > > >> Read three Vecs written to disk by PETSc, and calculate total
> "bounding
> > > >> box volume" of the grid:
> > > >>
> > > >>   g = h5py.File('grid.h5', 'r')
> > > >>   x = g['/MESH/nodes/x']
> > > >>   y = g['/MESH/nodes/y']
> > > >>   z = g['/MESH/nodes/z']
> > > >>   Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0])
> > > >>
> > > >> vs.
> > > >>
> > > >>   g = h5py.File('grid.h5', 'r')
> > > >>   x = g['/MESH/nodes/x'][:,0]
> > > >>   y = g['/MESH/nodes/y'][:,0]
> > > >>   z = g['/MESH/nodes/z'][:,0]
> > > >>   Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0])
> > > >>
> > > >>
> > > >> In both cases I think handling this extra, unnecessary dimension
> makes
> > > >> the code less attractive. It's not that either way is difficult,
> > > >> problematic or impossible, but it's just that 1D Vecs should
> intuitively
> > > >> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for
> > > >> quite a while until I figured this out, even after having written an
> > > >> entire Navier-Stokes DNS solver using the PETSc library for
> everything
> > > >> except time integration and filling these simple 1D coordinate
> arrays!
> > > >>
> > > >> Regards,
> > > >> H?kon
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150329/3d655193/attachment.html>

From haakon at hakostra.net  Sun Mar 29 16:00:05 2015
From: haakon at hakostra.net (=?UTF-8?Q?H=C3=A5kon_Strandenes?=)
Date: Sun, 29 Mar 2015 23:00:05 +0200
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <CAMYG4GkQJ=n=u_jkAzsz_zxmAWR=Fyq6wP-jGGESj1oT4=p8XA@mail.gmail.com>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
	<4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
	<CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>
	<5B59CECC-7A7F-4B4C-B13E-98DDAB29C625@mcs.anl.gov>
	<CAMYG4GkQJ=n=u_jkAzsz_zxmAWR=Fyq6wP-jGGESj1oT4=p8XA@mail.gmail.com>
Message-ID: <e44144c8e7434e1d91d6380cf56d675b@webmail.domeneshop.no>

Den 2015-03-29 21:48, skrev Matthew Knepley:
> 
> But of course there already is a PETSc-specific HDF5 format. Someone
> made up "/MESH/nodes/x". This is the essence of HDF5,
> XML, etc. that they are not actually formats, but rather storage
> technologies like the UNIX filesystem. Someone still has to declare
> a format, and we do.
> 
> However, if you want the data array to be 1D in any old HDF5 tool,
> except when complex is used, except when timestepping is used,
> then Yes, we need to make an exception for blocksize. I just think we
> will end up changing this back when it becomes apparent that
> having to put a check in all these HDF5 tools as well is onerous.
> 
>    Matt

My problem, as previously written, is that in the HDF5 output, you do 
already disregard dimensions with size one, all the time, except when 
dealing with the block size. Why?? Just think of the following cases:

1: When you create a dof=1 DMDA, that dof is not added as an extra 
dimension in the HDF5 output. Why don't you always create this 
dimension? When you create a dof=2 or dof=3, you suddenly write one 
extra dimension.
2: When you create a 2D DMDA (or 1D), you do not have that third 
dimension in the output. Why isn't a 2D grid just a special case of 3D 
with one dimension where the size is one?
3: When you do not use complex numbers, you do not have that extra 
dimension in the HDF5 output. Why don't you always add one extra 
dimension here in order to be consistent?
4: When you do not use timestepping, this temporal dimension is not 
added in the HDF5 output. Why don't you always add one extra dimension 
for time, just to avoid having to check for it? You coud treat "no 
timestepping" as "timestep 1"?

So the ONLY TIME you always add one dimension of size 1 is when dealing 
with the block size. In more or less all other cases you disregard these 
one-size dimensions and tries to compact the dataset into as few 
dimensions as possible. This is HIGHLY non-consistent and non-intuitive.

Is it really that hard to handle this "last special case", and suddenly 
get a consistent HDF5 output format?

Regards,
H?kon

(I know that I can make a 3D DMDA with one or two dimensions of size 1, 
and then have one or two dimension of size 1 in the HDF5 output, but 
that is really the only case I can think of. This is however not an 
issue, since it is explicitly declared a 3D case, and the user then 
expect 3D output)

From knepley at gmail.com  Sun Mar 29 20:46:14 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 29 Mar 2015 20:46:14 -0500
Subject: [petsc-users] Load Vec from 1D HDF5 dataset MATT READ THIS
	EMAIL!
In-Reply-To: <e44144c8e7434e1d91d6380cf56d675b@webmail.domeneshop.no>
References: <550A94B9.90407@hakostra.net>
	<90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov>
	<550B07D8.9060903@hakostra.net>
	<6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov>
	<CAMYG4GnyUYTvGeUpxbEwGpTn8hyceO1E=cVTv-BadLhHyvjmGQ@mail.gmail.com>
	<FEE35672-B258-4531-A617-087A6930F105@mcs.anl.gov>
	<550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org>
	<550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org>
	<550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org>
	<550C4B47.60903@hakostra.net>
	<01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov>
	<CAMYG4Gn0C0dXDTWpsdDto9qCdaCBhyLUWxBonbBZ3NR6zrSQ9Q@mail.gmail.com>
	<1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov>
	<CAMYG4GkU6F4ZmCVZ22gdDOAg3_+CJ_tbkJMhsg_Kc+m6yM=vxA@mail.gmail.com>
	<6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov>
	<550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net>
	<CA82DF68-CEC1-49EE-8530-04FA73BFB9F8@mcs.anl.gov>
	<CAMYG4GnFs5rW6UwK-AgR6rHrQ1qCJDQGsA9q5KCyvZE0-3+q+Q@mail.gmail.com>
	<4E13FEB2-644D-4226-9D30-30F6A786D24F@mcs.anl.gov>
	<CAMYG4GkixNu2v6yjp+Ja5h48TgZXQK8JxQy8rt8fJoh__x3_Cw@mail.gmail.com>
	<5B59CECC-7A7F-4B4C-B13E-98DDAB29C625@mcs.anl.gov>
	<CAMYG4GkQJ=n=u_jkAzsz_zxmAWR=Fyq6wP-jGGESj1oT4=p8XA@mail.gmail.com>
	<e44144c8e7434e1d91d6380cf56d675b@webmail.domeneshop.no>
Message-ID: <CAMYG4Gm8R2Of5PCnLDdGCEs9vdEs1PaaNEQDgEaL1WTAUJ3QAw@mail.gmail.com>

On Sun, Mar 29, 2015 at 4:00 PM, H?kon Strandenes <haakon at hakostra.net>
wrote:

> Den 2015-03-29 21:48, skrev Matthew Knepley:
>
>>
>> But of course there already is a PETSc-specific HDF5 format. Someone
>> made up "/MESH/nodes/x". This is the essence of HDF5,
>> XML, etc. that they are not actually formats, but rather storage
>> technologies like the UNIX filesystem. Someone still has to declare
>> a format, and we do.
>>
>> However, if you want the data array to be 1D in any old HDF5 tool,
>> except when complex is used, except when timestepping is used,
>> then Yes, we need to make an exception for blocksize. I just think we
>> will end up changing this back when it becomes apparent that
>> having to put a check in all these HDF5 tools as well is onerous.
>>
>>    Matt
>>
>
> My problem, as previously written, is that in the HDF5 output, you do
> already disregard dimensions with size one, all the time, except when
> dealing with the block size. Why?? Just think of the following cases:
>
> 1: When you create a dof=1 DMDA, that dof is not added as an extra
> dimension in the HDF5 output. Why don't you always create this dimension?
> When you create a dof=2 or dof=3, you suddenly write one extra dimension.
>

Yep, I don't like this either


> 2: When you create a 2D DMDA (or 1D), you do not have that third dimension
> in the output. Why isn't a 2D grid just a special case of 3D with one
> dimension where the size is one?
>

True


> 3: When you do not use complex numbers, you do not have that extra
> dimension in the HDF5 output. Why don't you always add one extra dimension
> here in order to be consistent?
>

I really hate this, but type handling is so weak in C and HDF5.


> 4: When you do not use timestepping, this temporal dimension is not added
> in the HDF5 output. Why don't you always add one extra dimension for time,
> just to avoid having to check for it? You coud treat "no timestepping" as
> "timestep 1"?
>

Yep.


> So the ONLY TIME you always add one dimension of size 1 is when dealing
> with the block size. In more or less all other cases you disregard these
> one-size dimensions and tries to compact the dataset into as few dimensions
> as possible. This is HIGHLY non-consistent and non-intuitive.
>
> Is it really that hard to handle this "last special case", and suddenly
> get a consistent HDF5 output format?
>

Okay, I am convinced by the timestepping and dimension. Unstructured grids
turn out to be much more beautiful
since there is none of this messing around.

I guess we should document this somewhere since the PETSc HDF5 format is
not completely obvious.

  Thanks,

     Matt


> Regards,
> H?kon
>
> (I know that I can make a 3D DMDA with one or two dimensions of size 1,
> and then have one or two dimension of size 1 in the HDF5 output, but that
> is really the only case I can think of. This is however not an issue, since
> it is explicitly declared a 3D case, and the user then expect 3D output)
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150329/edbae8d3/attachment-0001.html>

From stm8086 at yahoo.com  Sun Mar 29 21:56:30 2015
From: stm8086 at yahoo.com (Steena M)
Date: Sun, 29 Mar 2015 19:56:30 -0700
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <9B95D6AB-DCFC-483B-84D3-4C04C1021DFE@mcs.anl.gov>
Message-ID: <1427684190.1198.YahooMailBasic@web125404.mail.ne1.yahoo.com>

Hi Barry,

I am trying to partition a 20 row and 20 col sparse matrix between two procs such that proc 0 has 15 rows and 20 cols and proc 1 has 5 rows and 20 cols. The code snippet:


  
	CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); // at runtime: -matload_block_size 1
	
	
	if (rank ==0)
	{
		CHKERRQ( MatSetSizes(A, 15, 20, 20, 20) ); //rank 0 gets 75% of the rows
		CHKERRQ( MatSetType(A, MATMPIBAIJ) );
		CHKERRQ( MatLoad(A,fd) );
	 }
	
	else
	{
		CHKERRQ( MatSetSizes(A, 5, 20, 20, 20) ); //rank 1 gets 25% of the rows
		CHKERRQ( MatSetType(A, MATMPIBAIJ) ); 
		CHKERRQ( MatLoad(A,fd) );	  
      }

This throws the following error (probably from psplit.c):
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Nonconforming object sizes!
[1]PETSC ERROR: Sum of local lengths 40 does not equal global length 20, my local length 20
  likely a call to VecSetSizes() or MatSetSizes() is wrong.
See http://www.mcs.anl.gov/petsc/documentation/faq.html#split!

This error printout doesn't quite make sense to me. I'm trying to specify a total matrix size of 20x20... I haven't yet figured out where the '40' comes from in the error message.

Any thoughts on what might be going wrong?

Thanks in advance,
Steena



--------------------------------------------
On Sun, 3/22/15, Barry Smith <bsmith at mcs.anl.gov> wrote:

 Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
 To: "Steena M" <stm8086 at yahoo.com>
 Cc: petsc-users at mcs.anl.gov
 Date: Sunday, March 22, 2015, 3:58 PM
 
 
 ?
 Steena,
 
 ???I am
 a little unsure of your question.? 
 
 ???1) you can create a MPIBAIJ
 matrix with any distribution of block rows per process you
 want, just set the local row size for each process to be
 what you like.? Use MatCreateVecs() to get correspondingly
 laid out vectors.
 
 ???or 2) if you have a MPIBAIJ
 matrix with "equal" row layout and you want a new
 one with uneven row layout you can simply use
 MatGetSubMatrix() to create that new matrix.
 
 ? Barry
 
 Unless you have another reason to have the
 matrix with an equal number row layout I would just generate
 the matrix with the layout you want.
 
 
 > On Mar 22, 2015, at 5:50 PM, Steena M
 <stm8086 at yahoo.com>
 wrote:
 > 
 > Hello, 
 > 
 > I need to distribute
 a sparse matrix such that each proc owns an unequal number
 of blocked rows before I proceed with MPI MatMult. My
 initial thoughts on doing this:
 > 
 > 1) Use? MatGetSubMatrices() on the test
 MATMPIBAIJ matrix to produce a new matrix where each proc
 has an unequal number of rows.
 > 
 > 2) Provide scatter context for vector X
 (for MatMult )using IS iscol from MatGetSubMatrices() while
 creating the vector X.
 > 
 > 3) Call MatMult()
 > 
 > Will MatMult_MPIBAIJ continue to scatter
 this matrix and vector such that each proc will own an equal
 number of matrix rows and corresponding diagonal vector
 elements? Should I write my own MPIMatMult function to
 retain my redistribution of the matrix and vector?
 > 
 > Thanks in
 advance,
 > Steena
 

From knepley at gmail.com  Sun Mar 29 22:02:49 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 29 Mar 2015 22:02:49 -0500
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <1427684190.1198.YahooMailBasic@web125404.mail.ne1.yahoo.com>
References: <9B95D6AB-DCFC-483B-84D3-4C04C1021DFE@mcs.anl.gov>
	<1427684190.1198.YahooMailBasic@web125404.mail.ne1.yahoo.com>
Message-ID: <CAMYG4GnsMrnHBqyYk9QbtYDH_=TFVstm_hDoNYUph2xX+fVfEQ@mail.gmail.com>

On Sun, Mar 29, 2015 at 9:56 PM, Steena M <stm8086 at yahoo.com> wrote:

> Hi Barry,
>
> I am trying to partition a 20 row and 20 col sparse matrix between two
> procs such that proc 0 has 15 rows and 20 cols and proc 1 has 5 rows and 20
> cols. The code snippet:
>
>
>
>         CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); // at runtime:
> -matload_block_size 1
>
>
>         if (rank ==0)
>         {
>                 CHKERRQ( MatSetSizes(A, 15, 20, 20, 20) ); //rank 0 gets
> 75% of the rows
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ) );
>                 CHKERRQ( MatLoad(A,fd) );
>          }
>
>         else
>         {
>                 CHKERRQ( MatSetSizes(A, 5, 20, 20, 20) ); //rank 1 gets
> 25% of the rows
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ) );
>                 CHKERRQ( MatLoad(A,fd) );
>       }
>
> This throws the following error (probably from psplit.c):
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Nonconforming object sizes!
> [1]PETSC ERROR: Sum of local lengths 40 does not equal global length 20,
> my local length 20
>   likely a call to VecSetSizes() or MatSetSizes() is wrong.
> See http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
>
> This error printout doesn't quite make sense to me. I'm trying to specify
> a total matrix size of 20x20... I haven't yet figured out where the '40'
> comes from in the error message.
>
> Any thoughts on what might be going wrong?
>

Its the column specification. Just use PETSC_DETERMINE for the local
columns since all our sparse matrix
formats are row divisions anyway.

  Thanks,

     Matt


> Thanks in advance,
> Steena
>
>
>
> --------------------------------------------
> On Sun, 3/22/15, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>  Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI
> MatMult
>  To: "Steena M" <stm8086 at yahoo.com>
>  Cc: petsc-users at mcs.anl.gov
>  Date: Sunday, March 22, 2015, 3:58 PM
>
>
>
>  Steena,
>
>     I am
>  a little unsure of your question.
>
>     1) you can create a MPIBAIJ
>  matrix with any distribution of block rows per process you
>  want, just set the local row size for each process to be
>  what you like.  Use MatCreateVecs() to get correspondingly
>  laid out vectors.
>
>     or 2) if you have a MPIBAIJ
>  matrix with "equal" row layout and you want a new
>  one with uneven row layout you can simply use
>  MatGetSubMatrix() to create that new matrix.
>
>    Barry
>
>  Unless you have another reason to have the
>  matrix with an equal number row layout I would just generate
>  the matrix with the layout you want.
>
>
>  > On Mar 22, 2015, at 5:50 PM, Steena M
>  <stm8086 at yahoo.com>
>  wrote:
>  >
>  > Hello,
>  >
>  > I need to distribute
>  a sparse matrix such that each proc owns an unequal number
>  of blocked rows before I proceed with MPI MatMult. My
>  initial thoughts on doing this:
>  >
>  > 1) Use  MatGetSubMatrices() on the test
>  MATMPIBAIJ matrix to produce a new matrix where each proc
>  has an unequal number of rows.
>  >
>  > 2) Provide scatter context for vector X
>  (for MatMult )using IS iscol from MatGetSubMatrices() while
>  creating the vector X.
>  >
>  > 3) Call MatMult()
>  >
>  > Will MatMult_MPIBAIJ continue to scatter
>  this matrix and vector such that each proc will own an equal
>  number of matrix rows and corresponding diagonal vector
>  elements? Should I write my own MPIMatMult function to
>  retain my redistribution of the matrix and vector?
>  >
>  > Thanks in
>  advance,
>  > Steena
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150329/c9afc1ec/attachment.html>

From stm8086 at yahoo.com  Sun Mar 29 23:05:06 2015
From: stm8086 at yahoo.com (Steena M)
Date: Sun, 29 Mar 2015 21:05:06 -0700
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <CAMYG4GnsMrnHBqyYk9QbtYDH_=TFVstm_hDoNYUph2xX+fVfEQ@mail.gmail.com>
Message-ID: <1427688306.55028.YahooMailBasic@web125403.mail.ne1.yahoo.com>

Thanks Matt. I used PETSC_DETERMINE but I'm now getting an allocation-based error:

[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
[0]PETSC ERROR: ------------------------------------------------------------------------

I tried preallocating on each rank for the diagonal and off diagonal section of the matrix as the next step  My current approximations for preallocation

CHKERRQ( MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));  

 are throwing segmentation errors.

[0]PETSC ERROR:  Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range

Any insights into what I'm doing wrong?

Thanks,
Steena



On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com> wrote:

 Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
 To: "Steena M" <stm8086 at yahoo.com>
 Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov
 Date: Sunday, March 29, 2015, 10:02 PM
 
 On Sun, Mar 29, 2015 at
 9:56 PM, Steena M <stm8086 at yahoo.com>
 wrote:
 Hi
 Barry,
 
 
 
 I am trying to partition a 20 row and 20 col sparse matrix
 between two procs such that proc 0 has 15 rows and 20 cols
 and proc 1 has 5 rows and 20 cols. The code snippet:
 
 
 
 
 
 
 
 ? ? ? ? CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); //
 at runtime: -matload_block_size 1
 
 
 
 
 
 ? ? ? ? if (rank ==0)
 
 ? ? ? ? {
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 15, 20, 20,
 20) ); //rank 0 gets 75% of the rows
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ)
 );
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatLoad(A,fd) );
 
 ? ? ? ? ?}
 
 
 
 ? ? ? ? else
 
 ? ? ? ? {
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 5, 20, 20,
 20) ); //rank 1 gets 25% of the rows
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ)
 );
 
 ? ? ? ? ? ? ? ? CHKERRQ( MatLoad(A,fd) );
 
 ? ? ? }
 
 
 
 This throws the following error (probably from psplit.c):
 
 [1]PETSC ERROR: --------------------- Error Message
 ------------------------------------
 
 [1]PETSC ERROR: Nonconforming object sizes!
 
 [1]PETSC ERROR: Sum of local lengths 40 does not equal
 global length 20, my local length 20
 
 ? likely a call to VecSetSizes() or MatSetSizes() is
 wrong.
 
 See http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
 
 
 
 This error printout doesn't quite make sense to me.
 I'm trying to specify a total matrix size of 20x20... I
 haven't yet figured out where the '40' comes
 from in the error message.
 
 
 
 Any thoughts on what might be going wrong?
 
 Its the column specification. Just
 use PETSC_DETERMINE for the local columns since all our
 sparse matrixformats are row divisions
 anyway.
 ?
 Thanks,
 ? ?
 ?Matt?
 Thanks in advance,
 
 Steena
 
 
 
 
 
 
 
 --------------------------------------------
 
 On Sun, 3/22/15, Barry Smith <bsmith at mcs.anl.gov>
 wrote:
 
 
 
 ?Subject: Re: [petsc-users] Unequal sparse matrix row
 distribution for MPI MatMult
 
 ?To: "Steena M" <stm8086 at yahoo.com>
 
 ?Cc: petsc-users at mcs.anl.gov
 
 ?Date: Sunday, March 22, 2015, 3:58 PM
 
 
 
 
 
 ??
 
 ?Steena,
 
 
 
 ????I am
 
 ?a little unsure of your question.?
 
 
 
 ????1) you can create a MPIBAIJ
 
 ?matrix with any distribution of block rows per process
 you
 
 ?want, just set the local row size for each process to
 be
 
 ?what you like.? Use MatCreateVecs() to get
 correspondingly
 
 ?laid out vectors.
 
 
 
 ????or 2) if you have a MPIBAIJ
 
 ?matrix with "equal" row layout and you want a
 new
 
 ?one with uneven row layout you can simply use
 
 ?MatGetSubMatrix() to create that new matrix.
 
 
 
 ?? Barry
 
 
 
 ?Unless you have another reason to have the
 
 ?matrix with an equal number row layout I would just
 generate
 
 ?the matrix with the layout you want.
 
 
 
 
 
 ?> On Mar 22, 2015, at 5:50 PM, Steena M
 
 ?<stm8086 at yahoo.com>
 
 ?wrote:
 
 ?>
 
 ?> Hello,
 
 ?>
 
 ?> I need to distribute
 
 ?a sparse matrix such that each proc owns an unequal
 number
 
 ?of blocked rows before I proceed with MPI MatMult. My
 
 ?initial thoughts on doing this:
 
 ?>
 
 ?> 1) Use? MatGetSubMatrices() on the test
 
 ?MATMPIBAIJ matrix to produce a new matrix where each
 proc
 
 ?has an unequal number of rows.
 
 ?>
 
 ?> 2) Provide scatter context for vector X
 
 ?(for MatMult )using IS iscol from MatGetSubMatrices()
 while
 
 ?creating the vector X.
 
 ?>
 
 ?> 3) Call MatMult()
 
 ?>
 
 ?> Will MatMult_MPIBAIJ continue to scatter
 
 ?this matrix and vector such that each proc will own an
 equal
 
 ?number of matrix rows and corresponding diagonal vector
 
 ?elements? Should I write my own MPIMatMult function to
 
 ?retain my redistribution of the matrix and vector?
 
 ?>
 
 ?> Thanks in
 
 ?advance,
 
 ?> Steena
 
 
 
 
 
 
 -- 
 What most experimenters
 take for granted before they begin their experiments is
 infinitely more interesting than any results to which their
 experiments lead.
 -- Norbert
 Wiener
 

From bsmith at mcs.anl.gov  Sun Mar 29 23:26:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 29 Mar 2015 23:26:57 -0500
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <1427688306.55028.YahooMailBasic@web125403.mail.ne1.yahoo.com>
References: <1427688306.55028.YahooMailBasic@web125403.mail.ne1.yahoo.com>
Message-ID: <5819F2A4-76EF-459A-B0A6-72A98071D093@mcs.anl.gov>


> On Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com> wrote:
> 
> Thanks Matt. I used PETSC_DETERMINE but I'm now getting an allocation-based error:
> 
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> 
> I tried preallocating on each rank for the diagonal and off diagonal section of the matrix as the next step  My current approximations for preallocation
> 
> CHKERRQ( MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));  

   These arguments where you pass PETSC_DEFAULT are expecting a pointer not an integer. You can pass NULL in those locations. Though it is better to provide the correct preallocation rather than some defaults.

  Barry

> 
> are throwing segmentation errors.
> 
> [0]PETSC ERROR:  Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> 
> Any insights into what I'm doing wrong?
> 
> Thanks,
> Steena
> 
> 
> 
> On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com> wrote:
> 
> Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
> To: "Steena M" <stm8086 at yahoo.com>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov
> Date: Sunday, March 29, 2015, 10:02 PM
> 
> On Sun, Mar 29, 2015 at
> 9:56 PM, Steena M <stm8086 at yahoo.com>
> wrote:
> Hi
> Barry,
> 
> 
> 
> I am trying to partition a 20 row and 20 col sparse matrix
> between two procs such that proc 0 has 15 rows and 20 cols
> and proc 1 has 5 rows and 20 cols. The code snippet:
> 
> 
> 
> 
> 
> 
> 
>         CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); //
> at runtime: -matload_block_size 1
> 
> 
> 
> 
> 
>         if (rank ==0)
> 
>         {
> 
>                 CHKERRQ( MatSetSizes(A, 15, 20, 20,
> 20) ); //rank 0 gets 75% of the rows
> 
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
> 
>                 CHKERRQ( MatLoad(A,fd) );
> 
>          }
> 
> 
> 
>         else
> 
>         {
> 
>                 CHKERRQ( MatSetSizes(A, 5, 20, 20,
> 20) ); //rank 1 gets 25% of the rows
> 
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
> 
>                 CHKERRQ( MatLoad(A,fd) );
> 
>       }
> 
> 
> 
> This throws the following error (probably from psplit.c):
> 
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> 
> [1]PETSC ERROR: Nonconforming object sizes!
> 
> [1]PETSC ERROR: Sum of local lengths 40 does not equal
> global length 20, my local length 20
> 
>   likely a call to VecSetSizes() or MatSetSizes() is
> wrong.
> 
> See http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
> 
> 
> 
> This error printout doesn't quite make sense to me.
> I'm trying to specify a total matrix size of 20x20... I
> haven't yet figured out where the '40' comes
> from in the error message.
> 
> 
> 
> Any thoughts on what might be going wrong?
> 
> Its the column specification. Just
> use PETSC_DETERMINE for the local columns since all our
> sparse matrixformats are row divisions
> anyway.
>  
> Thanks,
>    
>  Matt 
> Thanks in advance,
> 
> Steena
> 
> 
> 
> 
> 
> 
> 
> --------------------------------------------
> 
> On Sun, 3/22/15, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> 
> 
> 
>  Subject: Re: [petsc-users] Unequal sparse matrix row
> distribution for MPI MatMult
> 
>  To: "Steena M" <stm8086 at yahoo.com>
> 
>  Cc: petsc-users at mcs.anl.gov
> 
>  Date: Sunday, March 22, 2015, 3:58 PM
> 
> 
> 
> 
> 
>   
> 
>  Steena,
> 
> 
> 
>     I am
> 
>  a little unsure of your question. 
> 
> 
> 
>     1) you can create a MPIBAIJ
> 
>  matrix with any distribution of block rows per process
> you
> 
>  want, just set the local row size for each process to
> be
> 
>  what you like.  Use MatCreateVecs() to get
> correspondingly
> 
>  laid out vectors.
> 
> 
> 
>     or 2) if you have a MPIBAIJ
> 
>  matrix with "equal" row layout and you want a
> new
> 
>  one with uneven row layout you can simply use
> 
>  MatGetSubMatrix() to create that new matrix.
> 
> 
> 
>    Barry
> 
> 
> 
>  Unless you have another reason to have the
> 
>  matrix with an equal number row layout I would just
> generate
> 
>  the matrix with the layout you want.
> 
> 
> 
> 
> 
>  > On Mar 22, 2015, at 5:50 PM, Steena M
> 
>  <stm8086 at yahoo.com>
> 
>  wrote:
> 
>  >
> 
>  > Hello,
> 
>  >
> 
>  > I need to distribute
> 
>  a sparse matrix such that each proc owns an unequal
> number
> 
>  of blocked rows before I proceed with MPI MatMult. My
> 
>  initial thoughts on doing this:
> 
>  >
> 
>  > 1) Use  MatGetSubMatrices() on the test
> 
>  MATMPIBAIJ matrix to produce a new matrix where each
> proc
> 
>  has an unequal number of rows.
> 
>  >
> 
>  > 2) Provide scatter context for vector X
> 
>  (for MatMult )using IS iscol from MatGetSubMatrices()
> while
> 
>  creating the vector X.
> 
>  >
> 
>  > 3) Call MatMult()
> 
>  >
> 
>  > Will MatMult_MPIBAIJ continue to scatter
> 
>  this matrix and vector such that each proc will own an
> equal
> 
>  number of matrix rows and corresponding diagonal vector
> 
>  elements? Should I write my own MPIMatMult function to
> 
>  retain my redistribution of the matrix and vector?
> 
>  >
> 
>  > Thanks in
> 
>  advance,
> 
>  > Steena
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters
> take for granted before they begin their experiments is
> infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert
> Wiener
> 


From kaiser at igpm.rwth-aachen.de  Mon Mar 30 05:18:39 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Mon, 30 Mar 2015 12:18:39 +0200
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <5515805B.3060406@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>	<CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>	<55157460.4080200@igpm.rwth-aachen.de>
	<CAMYG4G=OR3=gMHxAf2NMQCkkh7PpPDRBaoFZ+AeamiKaE2xapw@mail.gmail.com>
	<5515805B.3060406@igpm.rwth-aachen.de>
Message-ID: <551922FF.2020802@igpm.rwth-aachen.de>

Hallo Metthew,

at the weekend I updated petsc to the newest release version 3.5.2 and I 
still got this strange behavior.

Best

Klaus

On 03/27/2015 05:07 PM, Klaus Kaiser wrote:
> Hallo Matthew,
>
> here is a short example I used for 8 cores:
>
>     int d_nnz[3];
>     int o_nnz[3];
>     d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
>     o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
> 0, o_nnz,&A);
>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
> 0, o_nnz,&Ah);
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 
> 0, o_nnz,&At);
>
>     std::vector<double> insert(3*3*3*3, 1.0);
>     for(int i=0;i<8;++i)
>     {
>         int rows[3] = {i,i+1,i+3};
>         int cols[3] = {i,i+1,i+3};
>
>         MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES);
>     }
>
>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>     MatDuplicate(A,MAT_COPY_VALUES,&Ah);
>     MatDuplicate(A,MAT_COPY_VALUES,&At);
>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
>     MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
>     MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>
>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>
> The result is, that only some values are computed correctly.
>
> Best and Thank you a lot
>
> Klaus
>
>
>
> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser 
>> <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>> wrote:
>>
>>     Hallo Matthew,
>>
>>     thanks for your fast response. With "no beside the nonzero
>>     structure" I meant, that I do not create a different non-zero
>>     structure while I'm adding values to my matrices.
>>
>>     I also tried MAT_COPY_VALUES and the result was the same.
>>
>>
>> Please send the small example and we will figure out what is going wrong.
>>
>>   Thanks,
>>
>>      Matt
>>
>>     I do not get a error message but the resulting matrix is wrong.
>>     Here I have a short example. The first 9 rows and first 3 colomn
>>     of the matrices Ah, At and Ah+At on the first processor (of 8):
>>
>>     Ah:
>>     0:                   60.3553 -0.249975              2.77556e-17
>>     1:                         0 60.3553                        0
>>     2:                         0 0                  60.3553
>>     3:                   17.6777 0.374962                 0.124987
>>     4:                         0 17.6777                        0
>>     5:                         0 0                  17.6777
>>     6:                  -7.32233 -0.124987                 0.374962
>>     7:                         0 -7.32233                        0
>>     8:                         0 0                 -7.32233
>>
>>
>>     At:
>>     0:                         0 0                        0
>>     1:                      2500 0                        0
>>     2:              -4.54747e-13 0                        0
>>     3:                         0 0                        0
>>     4:                      1250 0                        0
>>     5:                      1250 0                        0
>>     6:                         0 0                        0
>>     7:                      1250 0                        0
>>     8:                      3750 0                        0
>>
>>     Ah+At
>>     0:                   60.3553 -0.249975              2.77556e-17
>>     1:                      2500 60.3553                        0
>>     2:              -4.54747e-13 0                  60.3553
>>     3:                   17.6777 0.374962                 0.124987
>>     4:                         0 17.6777                        0
>>     5:                         0 0                  17.6777
>>     6:                  -7.32233 -0.124987                 0.374962
>>     7:                         0 -7.32233                        0
>>     8:                         0 0                 -7.32233
>>
>>     you can see the first 3 rows of the resulting matrix looks
>>     exactly like what I would expect, but the last 6 rows only
>>     consists of the values of Ah. When you would also look on the
>>     matrix A and A+Ah+At you would also see, that values where both
>>     matrices have an nonzero entry are not sum correctly.
>>
>>     Best
>>
>>     Klaus
>>
>>     On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>>     On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser
>>>     <kaiser at igpm.rwth-aachen.de <mailto:kaiser at igpm.rwth-aachen.de>>
>>>     wrote:
>>>
>>>         Hallo,
>>>
>>>         I have a strange behavior in my code concerning the function
>>>         MatAXPY. I create 3 different Matrices
>>>
>>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>>         0, o_nnz,&A);
>>>             ierr =
>>>         MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>             ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>             ierr =
>>>         MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>>         0, o_nnz,&At);
>>>             ierr =
>>>         MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>             ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>             ierr =
>>>         MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>             ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size,
>>>         local_size, local_size, system_size, system_size, 0, d_nnz,
>>>         0, o_nnz,&Ah);
>>>             ierr =
>>>         MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>             ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>             ierr =
>>>         MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>
>>>     These creations are superfluous since you use MatDuplicate() below.
>>>
>>>         and want to sum these three matrixes with different factors.
>>>         First I fill the Matrix A with some values, and duplicate
>>>         the structure of A to the other two matrices:
>>>
>>>             MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>>             MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>>         MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>>             MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>>             MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>>             MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>>             MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>
>>>
>>>         After this I fill the matrices At and Ah with some other
>>>         values, which are not beside the non zero structure (I also
>>>         tried with just copying the Matrix A). Now after another
>>>         MatAssembly
>>>
>>>
>>>     I do not understand "no beside the nonzero structure". Do you
>>>     mean that the nonzero structure is the same?
>>>
>>>     Can you first test with MAT_COPY_VALUES?
>>>
>>>         I want to add these Matrices in the form A+c*(Ah+d*At): 
>>>
>>>
>>>             MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>>             MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>
>>>         When I run the method with mpi and one core everything works
>>>         fine. Starting the same method with more cores, the sum of
>>>         the matrices fails. It seems like some values are added
>>>
>>>
>>>     Please send the full output of a failure when you use
>>>     MAT_COPY_VALUES on 2 procs.
>>>
>>>       Thanks,
>>>
>>>          Matt
>>>
>>>         correctly and many values are missed. Using
>>>         DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in
>>>         the multi-core case, but is very slow. I checked with a
>>>         viewer if all matrices have the same nonzero structure and
>>>         this is the case.
>>>
>>>         Does anyone know why this fails, or do I have made any wrong
>>>         thoughts?
>>>
>>>         I'm corrently working with a petsc version (Petsc Release
>>>         Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I
>>>         looked into the changelogs up to the current version and did
>>>         not find any note about MatAXPY or MatAYPX.
>>>
>>>
>>>         Best and Thanks a lot for your help
>>>
>>>         Klaus
>>>
>>>
>>>
>>>
>>>     -- 
>>>     What most experimenters take for granted before they begin their
>>>     experiments is infinitely more interesting than any results to
>>>     which their experiments lead.
>>>     -- Norbert Wiener
>>
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their 
>> experiments is infinitely more interesting than any results to which 
>> their experiments lead.
>> -- Norbert Wiener
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/2b1bdd78/attachment.html>

From kaiser at igpm.rwth-aachen.de  Mon Mar 30 05:19:19 2015
From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser)
Date: Mon, 30 Mar 2015 12:19:19 +0200
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <FF765675-6957-4597-892A-D87BC6507914@mcs.anl.gov>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<0213B63B-0ACD-4074-A93C-6577163F4A99@mcs.anl.gov>
	<551581E8.4080109@igpm.rwth-aachen.de>
	<FF765675-6957-4597-892A-D87BC6507914@mcs.anl.gov>
Message-ID: <55192327.5060905@igpm.rwth-aachen.de>

Hi Barry,

I updated to the newest version but the behavior stays the same.

Best

Klaus



On 03/27/2015 06:32 PM, Barry Smith wrote:
>> On Mar 27, 2015, at 11:14 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
>>
>> Hallo Barry,
>>
>> I think you're right, but this is the setup I used for more than a year, so I first want to check if there is an error in my code and then switch with an correct code to a newer version when I do have the time for the installation.
>>
>> Is the faster symbolic MatAXPY for the DIFFERENT_NONZERO_STRUCTURE flag faster than using SAME_NONZERO_STRUCTURE?
>    No
>> Best and Thanks a lot
>>
>> Klaus
>>
>> On 03/27/2015 04:57 PM, Barry Smith wrote:
>>>    Klaus,
>>>
>>>      You would really benefit by upgrading to PETSc 3.5.3, we added much faster symbolic MatAXPY() for the DIFFERENT_NONZERO_STRUCTURE flag. Plus it is much easier for us to support the newest version.
>>>
>>>    Barry
>>>
>>>> On Mar 27, 2015, at 9:48 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de> wrote:
>>>>
>>>> Hallo,
>>>>
>>>> I have a strange behavior in my code concerning the function MatAXPY. I create 3 different Matrices
>>>>
>>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>>>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>>>     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>>     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>>     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>>>     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>>     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>>     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>>
>>>> and want to sum these three matrixes with different factors. First I fill the Matrix A with some values, and duplicate the structure of A to the other two matrices:
>>>>
>>>>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>>>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>>>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>>>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>>>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>>>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>>
>>>> After this I fill the matrices At and Ah with some other values, which are not beside the non zero structure (I also tried with just copying the Matrix A). Now after another MatAssembly I want to add these Matrices in the form A+c*(Ah+d*At):
>>>>
>>>>     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>>>     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>>
>>>> When I run the method with mpi and one core everything works fine. Starting the same method with more cores, the sum of the matrices fails. It seems like some values are added correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the multi-core case, but is very slow. I checked with a viewer if all matrices have the same nonzero structure and this is the case.
>>>>
>>>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>>>
>>>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to the current version and did not find any note about MatAXPY or MatAYPX.
>>>>
>>>>
>>>> Best and Thanks a lot for your help
>>>>
>>>> Klaus
>>>>


From mailinglists at xgm.de  Mon Mar 30 05:59:54 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Mon, 30 Mar 2015 12:59:54 +0200
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <CAMYG4Gms2XcpGGXBY26p5Jejatxfn4jXLOm8YpxouOMuPAMXKg@mail.gmail.com>
References: <2259494.rATCdaqx1O@asaru> <7174127.NEH3BeKRx3@asaru>
	<CAMYG4Gms2XcpGGXBY26p5Jejatxfn4jXLOm8YpxouOMuPAMXKg@mail.gmail.com>
Message-ID: <62081577.8iVh1dLrgt@asaru>



Am Freitag, 27. M?rz 2015, 07:34:56 schrieb Matthew Knepley:
> On Fri, Mar 27, 2015 at 7:31 AM, Florian Lindner <mailinglists at xgm.de>
> wrote:
> >
> > Am Freitag, 27. M?rz 2015, 07:26:11 schrieb Matthew Knepley:
> > > On Fri, Mar 27, 2015 at 4:28 AM, Florian Lindner <mailinglists at xgm.de>
> > > wrote:
> > >
> > > > Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> > > > > Florian Lindner <mailinglists at xgm.de> writes:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'm using petsc with petsc4py.
> > > > > >
> > > > > > A matrix is created like that
> > > > > >
> > > > > >     MPIrank = MPI.COMM_WORLD.Get_rank()
> > > > > >     MPIsize = MPI.COMM_WORLD.Get_size()
> > > > > >     print("MPI Rank = ", MPIrank)
> > > > > >     print("MPI Size = ", MPIsize)
> > > > > >     parts = partitions()
> > > > > >
> > > > > >     print("Dimension= ", nSupport + dimension, "bsize = ",
> > > > len(parts[MPIrank]))
> > > > > >
> > > > > >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> > > > > >     A = PETSc.Mat(); A.createDense( (nSupport + dimension,
> > nSupport +
> > > > dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> > > > >
> > > > > bsize is collective (must be the same on all processes).  It is used
> > for
> > > > > vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
> > > >
> > > > It seems I'm still misunderstanding the bsize parameter.
> > > >
> > > > If I distribute a 10x10 matrix on three ranks I need to have a
> > > > non-homogenous distribution, and thats what petsc does itself:
> > > >
> > >
> > > blockSize really means the uniform block size of the matrix, thus is HAS
> > to
> > > divide the global size. If it does not,
> > > you do not have a uniform block size, you have a bunch of different sized
> > > blocks.
> >
> > But how can I set a parallel layout when the size of the matrix is not
> > divisable by the number of ranks? When I omit bsize Petsc does that for me,
> > by using block sizes of 4, 3 and 3 on the three different ranks. How can I
> > set such a parallel layout manually?
> >
> 
> I am going to reply in C because it is my native language:
> 
>   MatCreate(comm, &A);
>   MatSetSizes(A, m, n, PETSC_DETERMINE, PETSC_DETERMINE);
>   MatSetFromOptions(A);
>   <Preallocation stuff here>
> 
> You have each proc give its local size.

Ok, that seems to be what I'm looking for...

I've experienced some things where my understanding of petsc still seems to be far off, regarding the parallel layout:

I have this code:

  ierr = MatSetSizes(matrix, 10, 10, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);

  MatGetOwnershipRange(matrix, &ownerBegin, &ownerEnd);
  cout << "Rank = " << MPIrank << " Begin = " << ownerBegin << " End = " << ownerEnd << endl;

Complete test code: http://pastebin.com/xFM1fJnQ

If started with mpirun -n 3 I it prints

Rank = 2 Begin = 0 End = 10
Rank = 1 Begin = 0 End = 10
Rank = 0 Begin = 0 End = 10

The same happens when I manually set the sizes per processor, like you suggested. 

  int sizes[] = {4, 3, 3};
  MatSetSizes(matrix, sizes[MPIrank], sizes[MPIrank], PETSC_DECIDE, PETSC_DECIDE);

I wonder why the range is always starting from 0, I was rather expecting something like 

Rank = 2 Begin = 7 End = 10
Rank = 1 Begin = 4 End = 7
Rank = 0 Begin = 0 End = 4

Petsc4py prints what I expect:

Rank =  1 Range    =  (4, 7) Size =  3
Rank =  2 Range    =  (7, 10) Size =  3
Rank =  0 Range    =  (0, 4) Size =  4


Is this the way it should be?


petsc4py's Mat::setSizes combines MatSetSizes and MatSetBlockSizes. I had some trouble what the correct datatype for size was, but figured it out now:

size = [ (m, M), (n,N) ]
size =  [ (sizes[rank], PETSc.DETERMINE), (sizes[rank], PETSc.DETERMINE) ]

The documentation on the python bindings is rather sparse...

Thanks,
Florian





From knepley at gmail.com  Mon Mar 30 06:41:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Mar 2015 06:41:45 -0500
Subject: [petsc-users] Understanding MatCreate bsize parameter
In-Reply-To: <62081577.8iVh1dLrgt@asaru>
References: <2259494.rATCdaqx1O@asaru> <7174127.NEH3BeKRx3@asaru>
	<CAMYG4Gms2XcpGGXBY26p5Jejatxfn4jXLOm8YpxouOMuPAMXKg@mail.gmail.com>
	<62081577.8iVh1dLrgt@asaru>
Message-ID: <CAMYG4G=wHLG=sbDVJMhAPMxwsCjgHVd5HxeqMcZ1-0YuaKmw0A@mail.gmail.com>

On Mon, Mar 30, 2015 at 5:59 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

>
>
> Am Freitag, 27. M?rz 2015, 07:34:56 schrieb Matthew Knepley:
> > On Fri, Mar 27, 2015 at 7:31 AM, Florian Lindner <mailinglists at xgm.de>
> > wrote:
> > >
> > > Am Freitag, 27. M?rz 2015, 07:26:11 schrieb Matthew Knepley:
> > > > On Fri, Mar 27, 2015 at 4:28 AM, Florian Lindner <
> mailinglists at xgm.de>
> > > > wrote:
> > > >
> > > > > Am Donnerstag, 26. M?rz 2015, 07:34:27 schrieb Jed Brown:
> > > > > > Florian Lindner <mailinglists at xgm.de> writes:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I'm using petsc with petsc4py.
> > > > > > >
> > > > > > > A matrix is created like that
> > > > > > >
> > > > > > >     MPIrank = MPI.COMM_WORLD.Get_rank()
> > > > > > >     MPIsize = MPI.COMM_WORLD.Get_size()
> > > > > > >     print("MPI Rank = ", MPIrank)
> > > > > > >     print("MPI Size = ", MPIsize)
> > > > > > >     parts = partitions()
> > > > > > >
> > > > > > >     print("Dimension= ", nSupport + dimension, "bsize = ",
> > > > > len(parts[MPIrank]))
> > > > > > >
> > > > > > >     MPI.COMM_WORLD.Barrier() # Just to keep the output together
> > > > > > >     A = PETSc.Mat(); A.createDense( (nSupport + dimension,
> > > nSupport +
> > > > > dimension), bsize = len(parts[MPIrank]) ) # <-- crash here
> > > > > >
> > > > > > bsize is collective (must be the same on all processes).  It is
> used
> > > for
> > > > > > vector-valued problems (like elasticity -- bs=3 in 3 dimensions).
> > > > >
> > > > > It seems I'm still misunderstanding the bsize parameter.
> > > > >
> > > > > If I distribute a 10x10 matrix on three ranks I need to have a
> > > > > non-homogenous distribution, and thats what petsc does itself:
> > > > >
> > > >
> > > > blockSize really means the uniform block size of the matrix, thus is
> HAS
> > > to
> > > > divide the global size. If it does not,
> > > > you do not have a uniform block size, you have a bunch of different
> sized
> > > > blocks.
> > >
> > > But how can I set a parallel layout when the size of the matrix is not
> > > divisable by the number of ranks? When I omit bsize Petsc does that
> for me,
> > > by using block sizes of 4, 3 and 3 on the three different ranks. How
> can I
> > > set such a parallel layout manually?
> > >
> >
> > I am going to reply in C because it is my native language:
> >
> >   MatCreate(comm, &A);
> >   MatSetSizes(A, m, n, PETSC_DETERMINE, PETSC_DETERMINE);
> >   MatSetFromOptions(A);
> >   <Preallocation stuff here>
> >
> > You have each proc give its local size.
>
> Ok, that seems to be what I'm looking for...
>
> I've experienced some things where my understanding of petsc still seems
> to be far off, regarding the parallel layout:
>
> I have this code:
>
>   ierr = MatSetSizes(matrix, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
> CHKERRQ(ierr);
>
>   MatGetOwnershipRange(matrix, &ownerBegin, &ownerEnd);
>   cout << "Rank = " << MPIrank << " Begin = " << ownerBegin << " End = "
> << ownerEnd << endl;
>
> Complete test code: http://pastebin.com/xFM1fJnQ
>
> If started with mpirun -n 3 I it prints
>
> Rank = 2 Begin = 0 End = 10
> Rank = 1 Begin = 0 End = 10
> Rank = 0 Begin = 0 End = 10
>

You created three serial matrices since you used PETSC_COMM_SELF in
MatCreate().

  Thanks,

     Matt


> The same happens when I manually set the sizes per processor, like you
> suggested.
>
>   int sizes[] = {4, 3, 3};
>   MatSetSizes(matrix, sizes[MPIrank], sizes[MPIrank], PETSC_DECIDE,
> PETSC_DECIDE);
>
> I wonder why the range is always starting from 0, I was rather expecting
> something like
>
> Rank = 2 Begin = 7 End = 10
> Rank = 1 Begin = 4 End = 7
> Rank = 0 Begin = 0 End = 4
>
> Petsc4py prints what I expect:
>
> Rank =  1 Range    =  (4, 7) Size =  3
> Rank =  2 Range    =  (7, 10) Size =  3
> Rank =  0 Range    =  (0, 4) Size =  4
>
>
> Is this the way it should be?
>
>
> petsc4py's Mat::setSizes combines MatSetSizes and MatSetBlockSizes. I had
> some trouble what the correct datatype for size was, but figured it out now:
>
> size = [ (m, M), (n,N) ]
> size =  [ (sizes[rank], PETSc.DETERMINE), (sizes[rank], PETSc.DETERMINE) ]
>
> The documentation on the python bindings is rather sparse...
>
> Thanks,
> Florian
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/4dcddecc/attachment.html>

From knepley at gmail.com  Mon Mar 30 06:43:30 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Mar 2015 06:43:30 -0500
Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN
In-Reply-To: <551922FF.2020802@igpm.rwth-aachen.de>
References: <55156DA7.6010602@igpm.rwth-aachen.de>
	<CAMYG4Gn9BDVT+SkW6DbXnQby+H7E-hm7W27L1ptXPdd1bN0q-g@mail.gmail.com>
	<55157460.4080200@igpm.rwth-aachen.de>
	<CAMYG4G=OR3=gMHxAf2NMQCkkh7PpPDRBaoFZ+AeamiKaE2xapw@mail.gmail.com>
	<5515805B.3060406@igpm.rwth-aachen.de>
	<551922FF.2020802@igpm.rwth-aachen.de>
Message-ID: <CAMYG4Gm68poAGxHNNQRjFR509YP5hDn+xf2yNKgL0rV3J1xt6w@mail.gmail.com>

On Mon, Mar 30, 2015 at 5:18 AM, Klaus Kaiser <kaiser at igpm.rwth-aachen.de>
wrote:

>  Hallo Metthew,
>
> at the weekend I updated petsc to the newest release version 3.5.2 and I
> still got this strange behavior.
>

Okay, I will check it out. The first thing I am going to do is try with AIJ
matrices instead of BAIJ, which
you can probably do before me. If you get the right result, then its a bug
in the BAIJ implementation,
which I am confident has not been tested as thoroughly.

  Thanks,

     Matt


> Best
>
> Klaus
>
>
> On 03/27/2015 05:07 PM, Klaus Kaiser wrote:
>
> Hallo Matthew,
>
> here is a short example I used for 8 cores:
>
>     int d_nnz[3];
>     int o_nnz[3];
>     d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;
>     o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&A);
>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&Ah);
>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0,
> o_nnz,&At);
>
>     std::vector<double> insert(3*3*3*3, 1.0);
>     for(int i=0;i<8;++i)
>     {
>         int rows[3] = {i,i+1,i+3};
>         int cols[3] = {i,i+1,i+3};
>
>         MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES);
>     }
>
>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>     MatDuplicate(A,MAT_COPY_VALUES,&Ah);
>     MatDuplicate(A,MAT_COPY_VALUES,&At);
>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>
>     MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);
>     MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);
>
>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>
> The result is, that only some values are computed correctly.
>
> Best and Thank you a lot
>
> Klaus
>
>
>
> On 03/27/2015 04:26 PM, Matthew Knepley wrote:
>
>  On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser <
> kaiser at igpm.rwth-aachen.de> wrote:
>
>>  Hallo Matthew,
>>
>> thanks for your fast response. With "no beside the nonzero structure" I
>> meant, that I do not create a different non-zero structure while I'm adding
>> values to my matrices.
>>
>> I also tried MAT_COPY_VALUES and the result was the same.
>>
>
>  Please send the small example and we will figure out what is going wrong.
>
>    Thanks,
>
>       Matt
>
>
>>  I do not get a error message but the resulting matrix is wrong. Here I
>> have a short example. The first 9 rows and first 3 colomn of the matrices
>> Ah, At and Ah+At on the first processor (of 8):
>>
>> Ah:
>> 0:                   60.3553                -0.249975
>> 2.77556e-17
>> 1:                         0
>> 60.3553                        0
>> 2:                         0                        0
>> 60.3553
>> 3:                   17.6777                 0.374962
>> 0.124987
>> 4:                         0
>> 17.6777                        0
>> 5:                         0                        0
>> 17.6777
>> 6:                  -7.32233                -0.124987
>> 0.374962
>> 7:                         0
>> -7.32233                        0
>> 8:                         0                        0
>> -7.32233
>>
>>
>> At:
>> 0:                         0
>> 0                        0
>> 1:                      2500
>> 0                        0
>> 2:              -4.54747e-13
>> 0                        0
>> 3:                         0
>> 0                        0
>> 4:                      1250
>> 0                        0
>> 5:                      1250
>> 0                        0
>> 6:                         0
>> 0                        0
>> 7:                      1250
>> 0                        0
>> 8:                      3750
>> 0                        0
>>
>> Ah+At
>> 0:                   60.3553                -0.249975
>> 2.77556e-17
>> 1:                      2500
>> 60.3553                        0
>> 2:              -4.54747e-13                        0
>> 60.3553
>> 3:                   17.6777                 0.374962
>> 0.124987
>> 4:                         0
>> 17.6777                        0
>> 5:                         0                        0
>> 17.6777
>> 6:                  -7.32233                -0.124987
>> 0.374962
>> 7:                         0
>> -7.32233                        0
>> 8:                         0                        0
>> -7.32233
>>
>> you can see the first 3 rows of the resulting matrix looks exactly like
>> what I would expect, but the last 6 rows only consists of the values of Ah.
>> When you would also look on the matrix A and A+Ah+At you would also see,
>> that values where both matrices have an nonzero entry are not sum correctly.
>>
>> Best
>>
>> Klaus
>>
>> On 03/27/2015 03:59 PM, Matthew Knepley wrote:
>>
>>  On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser <
>> kaiser at igpm.rwth-aachen.de> wrote:
>>
>>> Hallo,
>>>
>>> I have a strange behavior in my code concerning the function MatAXPY. I
>>> create 3 different Matrices
>>>
>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
>>>     ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>     ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>     ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
>>>     ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>     ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>     ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>>     ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
>>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
>>>     ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
>>>     ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
>>>     ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);
>>>
>>
>>  These creations are superfluous since you use MatDuplicate() below.
>>
>>
>>> and want to sum these three matrixes with different factors. First I
>>> fill the Matrix A with some values, and duplicate the structure of A to the
>>> other two matrices:
>>>
>>>     MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
>>>     MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
>>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
>>>     MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
>>>     MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
>>>     MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
>>>     MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
>>>     MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);
>>>
>>
>>
>>> After this I fill the matrices At and Ah with some other values, which
>>> are not beside the non zero structure (I also tried with just copying the
>>> Matrix A). Now after another MatAssembly
>>
>>
>>  I do not understand "no beside the nonzero structure". Do you mean that
>> the nonzero structure is the same?
>>
>>  Can you first test with MAT_COPY_VALUES?
>>
>>
>>> I want to add these Matrices in the form A+c*(Ah+d*At):
>>
>>
>>>     MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
>>>     MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);
>>>
>>> When I run the method with mpi and one core everything works fine.
>>> Starting the same method with more cores, the sum of the matrices fails. It
>>> seems like some values are added
>>
>>
>>  Please send the full output of a failure when you use MAT_COPY_VALUES
>> on 2 procs.
>>
>>    Thanks,
>>
>>       Matt
>>
>>
>>> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE
>>> leads to the right behavior in the multi-core case, but is very slow. I
>>> checked with a viewer if all matrices have the same nonzero structure and
>>> this is the case.
>>>
>>> Does anyone know why this fails, or do I have made any wrong thoughts?
>>>
>>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0,
>>> Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the changelogs up to
>>> the current version and did not find any note about MatAXPY or MatAYPX.
>>>
>>>
>>> Best and Thanks a lot for your help
>>>
>>> Klaus
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/873f72da/attachment-0001.html>

From jchang27 at uh.edu  Mon Mar 30 10:27:19 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Mon, 30 Mar 2015 10:27:19 -0500
Subject: [petsc-users] Documentation for VTU format?
Message-ID: <CAP2=TMixzqshsDUe3qk2SvYD0L_2=DrjHqSF7dEAcG86P++Y4g@mail.gmail.com>

Hi all,

I have a couple questions:

1) Is there any documentation somewhere that talks specifically on how to
output using the *.vtu format? Specifically for transient problems. When I
look at TS ex11 it seems to me the user has to create several additional
routines, but I was wondering if there are any simpler
examples/instructions.

2) Can vtu files be outputted in parallel?

Thanks,

-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/aabc1b29/attachment.html>

From jed at jedbrown.org  Mon Mar 30 10:54:38 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 30 Mar 2015 09:54:38 -0600
Subject: [petsc-users] Documentation for VTU format?
In-Reply-To: <CAP2=TMixzqshsDUe3qk2SvYD0L_2=DrjHqSF7dEAcG86P++Y4g@mail.gmail.com>
References: <CAP2=TMixzqshsDUe3qk2SvYD0L_2=DrjHqSF7dEAcG86P++Y4g@mail.gmail.com>
Message-ID: <87wq1ywke9.fsf@jedbrown.org>

Justin Chang <jchang27 at uh.edu> writes:

> Hi all,
>
> I have a couple questions:
>
> 1) Is there any documentation somewhere that talks specifically on how to
> output using the *.vtu format? Specifically for transient problems. When I
> look at TS ex11 it seems to me the user has to create several additional
> routines, but I was wondering if there are any simpler
> examples/instructions.

    ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"your-file.vtu",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
    ierr = VecView(x,viewer);CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

... assuming x is a vector obtained from a DM capable of writing *.vtu.
Note that the vtu format is not designed for time-series, so you'll need
to write a separate file per time step.

> 2) Can vtu files be outputted in parallel?

Yes, we write one file in parallel (not one file per rank which is a
scalability disaster).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/7d1e0759/attachment.pgp>

From jchang27 at uh.edu  Mon Mar 30 11:51:18 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Mon, 30 Mar 2015 11:51:18 -0500
Subject: [petsc-users] Documentation for VTU format?
In-Reply-To: <87wq1ywke9.fsf@jedbrown.org>
References: <CAP2=TMixzqshsDUe3qk2SvYD0L_2=DrjHqSF7dEAcG86P++Y4g@mail.gmail.com>
	<87wq1ywke9.fsf@jedbrown.org>
Message-ID: <CAP2=TMgAdWVKtHTZSXuqaUPpB=qz0ADWBG9xROCE3-RD8P9sug@mail.gmail.com>

Jed thank you for the response.

1) In my case, I want to output a transient diffusion problem (with
DMPlex). I see that in ex11 there's a TSMonitorSet(). Is this (and its
calling function) be the place I have something along the lines that you
have suggested? That is, if TSMonitorSet() calls MonitorVTK(), then
MonitorVTK() would have PetscViewerVTKOpen(...). Am I along the right track?

2) What I meant by parallel was if each rank would output its local part of
the solution, kind of like the way it's done with FEniCS. That way, if I
had an extremely large solution set (at least over 10 million DOFS) then I
wouldn't be stuck with one giant ASCII file. Unless that's what binary
formats like HDF5 are for?

Thanks,

On Mon, Mar 30, 2015 at 10:54 AM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jchang27 at uh.edu> writes:
>
> > Hi all,
> >
> > I have a couple questions:
> >
> > 1) Is there any documentation somewhere that talks specifically on how to
> > output using the *.vtu format? Specifically for transient problems. When
> I
> > look at TS ex11 it seems to me the user has to create several additional
> > routines, but I was wondering if there are any simpler
> > examples/instructions.
>
>     ierr =
> PetscViewerVTKOpen(PETSC_COMM_WORLD,"your-file.vtu",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
>     ierr = VecView(x,viewer);CHKERRQ(ierr);
>     ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>
> ... assuming x is a vector obtained from a DM capable of writing *.vtu.
> Note that the vtu format is not designed for time-series, so you'll need
> to write a separate file per time step.
>
> > 2) Can vtu files be outputted in parallel?
>
> Yes, we write one file in parallel (not one file per rank which is a
> scalability disaster).
>
>


-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/466dfbe7/attachment.html>

From jed at jedbrown.org  Mon Mar 30 14:00:31 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 30 Mar 2015 13:00:31 -0600
Subject: [petsc-users] Documentation for VTU format?
In-Reply-To: <CAP2=TMgAdWVKtHTZSXuqaUPpB=qz0ADWBG9xROCE3-RD8P9sug@mail.gmail.com>
References: <CAP2=TMixzqshsDUe3qk2SvYD0L_2=DrjHqSF7dEAcG86P++Y4g@mail.gmail.com>
	<87wq1ywke9.fsf@jedbrown.org>
	<CAP2=TMgAdWVKtHTZSXuqaUPpB=qz0ADWBG9xROCE3-RD8P9sug@mail.gmail.com>
Message-ID: <87k2xywbsg.fsf@jedbrown.org>

Justin Chang <jchang27 at uh.edu> writes:

> Jed thank you for the response.
>
> 1) In my case, I want to output a transient diffusion problem (with
> DMPlex). I see that in ex11 there's a TSMonitorSet(). Is this (and its
> calling function) be the place I have something along the lines that you
> have suggested? That is, if TSMonitorSet() calls MonitorVTK(), then
> MonitorVTK() would have PetscViewerVTKOpen(...). Am I along the right track?

Yes.  A simple case can be done automatically:

  -ts_monitor_solution_vtk 'filename-%03D.vtu'

but if you want something more complicated, you should write your own
monitor.

> 2) What I meant by parallel was if each rank would output its local part of
> the solution, kind of like the way it's done with FEniCS. That way, if I
> had an extremely large solution set (at least over 10 million DOFS) then I
> wouldn't be stuck with one giant ASCII file. Unless that's what binary
> formats like HDF5 are for?

* vtu is a binary format (ASCII/XML header followed by binary data).

* Creating thousands of small files is worse.

* 10M DOFs is small.

* HDF5 is a better format, but also a bigger dependency and somewhat
  higher effort to work with.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/d4514af8/attachment.pgp>

From ansp6066 at colorado.edu  Mon Mar 30 16:25:23 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Mon, 30 Mar 2015 14:25:23 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <CAEcYPwDvf1X3oT-0f-EU3pCM2uWaa1gKwA3ru8MeNXdxka2wrw@mail.gmail.com>
References: <CAEcYPwDvf1X3oT-0f-EU3pCM2uWaa1gKwA3ru8MeNXdxka2wrw@mail.gmail.com>
Message-ID: <1427750723534.fc87bf97@Nodemailer>

Emil: ?If I want to fix this myself, do you have any idea where I should start looking? ?I assume in the th->endpoint section of TSStep_Theta in http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?




Thanks,




-Andrew

On Tue, Mar 24, 2015 at 10:52 AM, Lisandro Dalcin <dalcinl at gmail.com>
wrote:

> On 24 March 2015 at 18:40, Emil Constantinescu <emconsta at mcs.anl.gov> wrote:
>>> Of course, though my B-D is order 2, and as you said, it is the just
>>> for the estimator.
>>>
>>>> if the time step varies wildly (can be easily constrained),
>>>> it may create some problems with the estimator.
>>>>
>>>
>>> Indeed, the default clipping 0.1,10 is usually too wide, I usually do
>>> -ts_adapt_basic_clip 0.5,2.0
>>
>>
>> The limits are a bit more strict for BDF2, but that's for the worst case and
>> no filtering. So that looks good too.
> I'm not sure I'm following you 100%. Please note I'm not using BDF2 in
> the usual sense, what I do is to compute an A POSTERIORI error
> estimator using a BD of order 2 to estimate LTE^{n+1} with the
> solution vectors U^{n+1}, U^{n} and U^{n-1}, and this is AFTER
> computing U^{n-1}.
> -- 
> Lisandro Dalcin
> ============
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Numerical Porous Media Center (NumPor)
> King Abdullah University of Science and Technology (KAUST)
> http://numpor.kaust.edu.sa/
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 4332
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> Office Phone: +966 12 808-0459
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/0ae0d919/attachment.html>

From ansp6066 at colorado.edu  Mon Mar 30 16:27:24 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Mon, 30 Mar 2015 14:27:24 -0700 (PDT)
Subject: [petsc-users] HermitianTranspose version of MatCreateTranspose.
In-Reply-To: <1653360C-A2CA-40EA-B48C-763D9BFA3DF9@mcs.anl.gov>
References: <1653360C-A2CA-40EA-B48C-763D9BFA3DF9@mcs.anl.gov>
Message-ID: <1427750843720.47d950ee@Nodemailer>

I submitted a pull request for this: https://bitbucket.org/petsc/petsc/pull-request/291/creates-a-matcreatehermitiantranspose/diff




However, I forgot to pull master before submitting it, and someone added ex180 in the meantime. ?Should I redo the pull request, or just let the merger fix it?




-Andrew

On Wed, Mar 18, 2015 at 3:10 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>> On Mar 18, 2015, at 3:41 PM, Andrew Spott <ansp6066 at colorado.edu> wrote:
>> 
>> So, I?ve got a MatCreateHermitianTranspose function that has close to the same functionality as the MatCreatTranspose version.  So I?m getting ready to send a pull request.
>> 
>   Great
>> A few questions:
>> 
>> What branch should I add my changes to?
> master
>> 
>> Should I create tests for this?
> most definitely
>>  Where should I put them?
> src/mat/examples/tests
>> 
>> On Monday, Feb 23, 2015 at 9:02 PM, Barry Smith <bsmith at mcs.anl.gov>, wrote:
>> 
>> We've had a small amount of debate over the years on how to handle the Hermitian transpose and non-Hermitian transpose that never got fully resolved. 
>> 
>> Approach 1) Each (complex) matrix has a full set of transpose and Hermitian transpose operations (MatTranspose(), MatHermitianTranspose(), MatMultTranspose()), MatMultHermitianTranspose(), MatSolveTranspose(), MatSolveHermitianTranspose(), MatMatMultTranspose(), MatMatMultHermitianTranspose(), MatTranposeMatMult(), MatHermitianTransposeMatMult().......) plus there are two vector "inner" products; VecDot() and VecTDot(). 
>> 
>> Approach 2) Consider a (complex) vector (and hence the associated matrix operators on it) to live in the usual Hermitian inner product space or the non-Hermitian "inner product space". Then one only needs a single VecDot() and MatTranspose(), MatMultTranspose() ... that just "does the right thing" based on what space the user has declared the vectors/matrices to be in. 
>> 
>> Approach 2) seems nicer since it only requires 1/2 the functions :-) and so long as the two vector "spaces" never interact directly (for example what would be the meaning of the "inner" product of a vector in the usual Hermitian inner product space with a vector from the non-Hermitian "inner product space"?) certain seems simpler. Approach 1) might be simpler for some people who like to always see exactly what they are doing. 
>> 
>> I personally wish I had started with Approach 2 (but I did not), but there could be some flaw with it I am not seeing. 
>> 
>> Barry 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> > On Feb 23, 2015, at 6:50 PM, Andrew Spott <ansp6066 at colorado.edu> wrote: 
>> > 
>> > I?m definitely willing to submit it as a pull request. 
>> > 
>> > Also, while I?m at it, I?m going to write a ?duplicate? function for transpose and hermitian_transpose. Just because this seems 1) easy ( MatHermitianTranspose can return a new copy, as well as MatTranspose), and 2) necessary to use these for EPS. 
>> > 
>> > Also, is ?transpose? a good enough MatType? Or does a new one need to be written? 
>> > 
>> > -Andrew 
>> > 
>> > 
>> > 
>> > On Mon, Feb 23, 2015 at 3:12 PM, Jed Brown <jed at jedbrown.org> wrote: 
>> > 
>> > <signature.asc> 
>> > 
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/203a633c/attachment.html>

From j.pogacnik at auckland.ac.nz  Mon Mar 30 16:39:58 2015
From: j.pogacnik at auckland.ac.nz (Justin Pogacnik)
Date: Mon, 30 Mar 2015 21:39:58 +0000
Subject: [petsc-users] FEM 101
Message-ID: <38871A58892BC5409E1B66DFE5843DE791197AC9@uxcn10-1.UoA.auckland.ac.nz>

Hello,

I'm beginning to write a finite element solid mechanics code in F90 and am having some problems getting started with finite element basics within Petsc. I'm using an unstructured grid and the DMPlex class and able to access most of the mesh information required for FE numerical integration. I've used PetscFECreateDefault and have found the quadrature point locations/data. I'm trying to determine how to find the values of standard trilinear basis functions and their derivatives at the element quadrature points and then to perform the integration on the element. Can anyone advise what the proper routines are that I should call? It's possible that I'm way off target. I've seen "PetscFEGetBasisSpace", but I'm unsure what to do with the resulting output space. I've also seen the PetscFEIntegrate routines, but unsure how their intended to be used and don't see any fortran wrappers.

I hope this question is clear. Please let me know if you require more information. I appreciate any help and guidance that anyone can offer.

Many thanks,

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/ddf84196/attachment.html>

From jed at jedbrown.org  Mon Mar 30 17:02:33 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 30 Mar 2015 16:02:33 -0600
Subject: [petsc-users] HermitianTranspose version of MatCreateTranspose.
In-Reply-To: <1427750843720.47d950ee@Nodemailer>
References: <1653360C-A2CA-40EA-B48C-763D9BFA3DF9@mcs.anl.gov>
	<1427750843720.47d950ee@Nodemailer>
Message-ID: <87pp7quosm.fsf@jedbrown.org>

Andrew Spott <ansp6066 at colorado.edu> writes:

> I submitted a pull request for this: https://bitbucket.org/petsc/petsc/pull-request/291/creates-a-matcreatehermitiantranspose/diff
>
>
>
>
> However, I forgot to pull master before submitting it, and someone added ex180 in the meantime. ?Should I redo the pull request, or just let the merger fix it?

The merger can fix this.  Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/9bd60b51/attachment.pgp>

From emconsta at mcs.anl.gov  Mon Mar 30 20:23:55 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Mon, 30 Mar 2015 20:23:55 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1427750723534.fc87bf97@Nodemailer>
References: <CAEcYPwDvf1X3oT-0f-EU3pCM2uWaa1gKwA3ru8MeNXdxka2wrw@mail.gmail.com>
	<1427750723534.fc87bf97@Nodemailer>
Message-ID: <5519F72B.3060308@mcs.anl.gov>

Hi Andrew,

Sorry for the delay; things are going slower with me over the spring 
break but we are working on it.

The problem that you see may not be related to the integrator, which 
runs well (we get expected convergence rates and good results) for other 
problems, in particular, for problems with real variables (and no 
c++11). So the high level th->endpoint implementation seems fine.

For some reason the linear solver is misbehaving when using cn. If you 
use option -snes_mf the problem disappears.

Please do not use -ts_theta_adapt until it is fixed.

In conclusion I don't think that looking at theta.c will be productive. 
We are currently looking at what happens at the lower levels.

Thanks,
Emil

On 3/30/15 4:25 PM, Andrew Spott wrote:
> Emil:  If I want to fix this myself, do you have any idea where I should
> start looking?  I assume in the th->endpoint section of TSStep_Theta in
> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?
>
> Thanks,
>
> -Andrew
>
>
>

From ansp6066 at colorado.edu  Mon Mar 30 21:32:45 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Mon, 30 Mar 2015 19:32:45 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <5519F72B.3060308@mcs.anl.gov>
References: <5519F72B.3060308@mcs.anl.gov>
Message-ID: <1427769165045.62947bec@Nodemailer>

I?m sorry, I didn?t want to imply you were taking too much time. ?I was just interested in trying to solve the problem myself, especially if it was a simpler fix.




I don?t have enough of an understanding of the internals of PETSc though, if it depends on the compiler flags.




Good luck!




-Andrew



?
Andrew

On Mon, Mar 30, 2015 at 7:23 PM, Emil Constantinescu <emconsta at mcs.anl.gov>
wrote:

> Hi Andrew,
> Sorry for the delay; things are going slower with me over the spring 
> break but we are working on it.
> The problem that you see may not be related to the integrator, which 
> runs well (we get expected convergence rates and good results) for other 
> problems, in particular, for problems with real variables (and no 
> c++11). So the high level th->endpoint implementation seems fine.
> For some reason the linear solver is misbehaving when using cn. If you 
> use option -snes_mf the problem disappears.
> Please do not use -ts_theta_adapt until it is fixed.
> In conclusion I don't think that looking at theta.c will be productive. 
> We are currently looking at what happens at the lower levels.
> Thanks,
> Emil
> On 3/30/15 4:25 PM, Andrew Spott wrote:
>> Emil:  If I want to fix this myself, do you have any idea where I should
>> start looking?  I assume in the th->endpoint section of TSStep_Theta in
>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?
>>
>> Thanks,
>>
>> -Andrew
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/0c0cfd9d/attachment.html>

From emconsta at mcs.anl.gov  Mon Mar 30 22:01:40 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Mon, 30 Mar 2015 22:01:40 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1427769165045.62947bec@Nodemailer>
References: <5519F72B.3060308@mcs.anl.gov> <1427769165045.62947bec@Nodemailer>
Message-ID: <551A0E14.1030805@mcs.anl.gov>

On 3/30/15 9:32 PM, Andrew Spott wrote:
> I?m sorry, I didn?t want to imply you were taking too much time.  I was
> just interested in trying to solve the problem myself, especially if it
> was a simpler fix.

Nothing of that sort. I just expected to figure it out sooner, 
especially when you were kind enough to produce a very simple case and 
an external Crank?Nicolson. We welcome any help. However, as far as I 
can tell the issue may not be in the way the theta.c is implemented.

Emil

> I don?t have enough of an understanding of the internals of PETSc
> though, if it depends on the compiler flags.
>
> Good luck!
>
> -Andrew
>
> ?
> Andrew
>
>
> On Mon, Mar 30, 2015 at 7:23 PM, Emil Constantinescu
> <emconsta at mcs.anl.gov <mailto:emconsta at mcs.anl.gov>> wrote:
>
>     Hi Andrew,
>
>     Sorry for the delay; things are going slower with me over the spring
>     break but we are working on it.
>
>     The problem that you see may not be related to the integrator, which
>     runs well (we get expected convergence rates and good results) for
>     other
>     problems, in particular, for problems with real variables (and no
>     c++11). So the high level th->endpoint implementation seems fine.
>
>     For some reason the linear solver is misbehaving when using cn. If you
>     use option -snes_mf the problem disappears.
>
>     Please do not use -ts_theta_adapt until it is fixed.
>
>     In conclusion I don't think that looking at theta.c will be productive.
>     We are currently looking at what happens at the lower levels.
>
>     Thanks,
>     Emil
>
>     On 3/30/15 4:25 PM, Andrew Spott wrote:
>      > Emil: If I want to fix this myself, do you have any idea where I
>     should
>      > start looking? I assume in the th->endpoint section of
>     TSStep_Theta in
>      >
>     http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?
>
>      >
>      > Thanks,
>      >
>      > -Andrew
>      >
>      >
>      >
>
>

From knepley at gmail.com  Mon Mar 30 23:14:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Mar 2015 23:14:18 -0500
Subject: [petsc-users] FEM 101
In-Reply-To: <38871A58892BC5409E1B66DFE5843DE791197AC9@uxcn10-1.UoA.auckland.ac.nz>
References: <38871A58892BC5409E1B66DFE5843DE791197AC9@uxcn10-1.UoA.auckland.ac.nz>
Message-ID: <CAMYG4GkabOonmntUwh=YXE3VnKkUfrgq0Ep0HgC6PVYmeE2_sQ@mail.gmail.com>

On Mon, Mar 30, 2015 at 4:39 PM, Justin Pogacnik <j.pogacnik at auckland.ac.nz>
wrote:

>  Hello,
>
>  I'm beginning to write a finite element solid mechanics code in F90 and
> am having some problems getting started with finite element basics within
> Petsc. I'm using an unstructured grid and the DMPlex class and able to
> access most of the mesh information required for FE numerical integration.
> I've used PetscFECreateDefault and have found the quadrature point
> locations/data. I'm trying to determine how to find the values of standard
> trilinear basis functions and their derivatives at the element quadrature
> points and then to perform the integration on the element. Can anyone
> advise what the proper routines are that I should call? It's possible that
> I'm way off target. I've seen "PetscFEGetBasisSpace", but I'm unsure what
> to do with the resulting output space. I've also seen the PetscFEIntegrate
> routines, but unsure how their intended to be used and don't see any
> fortran wrappers.
>
>  I hope this question is clear. Please let me know if you require more
> information. I appreciate any help and guidance that anyone can offer.
>

Here is the plain vanilla integration routine:


https://bitbucket.org/petsc/petsc/src/1c9d18e7633f5b782021dca6d8727de2838ee261/src/dm/dt/interface/dtfe.c?at=master#cl-3380

which is an implementation of PetscFEIntegrate(). The idea is that this
integrates a bunch of elements, and then there is an outer
loop over the whole mesh, which is currently DMPlexIntegrateResidual().
Unfortunately, this function also does a bunch of other
stuff, and right now I do not see how to simplify it.

Does this make sense?

  Thanks,

     Matt


> Many thanks,
>
>  Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150330/e8846379/attachment.html>

From dmeiser at txcorp.com  Tue Mar 31 10:12:35 2015
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Tue, 31 Mar 2015 09:12:35 -0600
Subject: [petsc-users] DMDA halo exchange
Message-ID: <551AB963.8060003@txcorp.com>

Hi,

I have a DMDA for a structured grid in a 3D FDTD calculation. What is 
the most efficient way to update the ghost regions after each time step? 
Currently I update a global vector and then I scatter to a local vector 
using DMGlobalToLocalBegin/End for the next time step. I also tried 
DMLocalToLocal with the same source and target vector and that appears 
to be slightly faster, perhaps due to better cache usage in the copy of 
the owned region. Ideally I would like to update just the ghost cells 
from the owned regions. Is there a good way to do that? Something 
similar to VecGhostUpdateBegin/End?

Thanks,
Dominic

-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com


From bsmith at mcs.anl.gov  Tue Mar 31 11:24:36 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 31 Mar 2015 11:24:36 -0500
Subject: [petsc-users] DMDA halo exchange
In-Reply-To: <551AB963.8060003@txcorp.com>
References: <551AB963.8060003@txcorp.com>
Message-ID: <C1A96ECA-AC74-451B-B76B-81D7CA9CD968@mcs.anl.gov>


  Dominic,

    DMLocalToLocal is the intended usage. There is code inside the VecScatter to detect unneeded copies, which would result in only touching the ghost cells,  but for some reason that is not currently turned on, I'll take a look at fixing it in master.

  Barry

> On Mar 31, 2015, at 10:12 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:
> 
> Hi,
> 
> I have a DMDA for a structured grid in a 3D FDTD calculation. What is the most efficient way to update the ghost regions after each time step? Currently I update a global vector and then I scatter to a local vector using DMGlobalToLocalBegin/End for the next time step. I also tried DMLocalToLocal with the same source and target vector and that appears to be slightly faster, perhaps due to better cache usage in the copy of the owned region. Ideally I would like to update just the ghost cells from the owned regions. Is there a good way to do that? Something similar to VecGhostUpdateBegin/End?
> 
> Thanks,
> Dominic
> 
> -- 
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
> 


From dmeiser at txcorp.com  Tue Mar 31 12:21:39 2015
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Tue, 31 Mar 2015 11:21:39 -0600
Subject: [petsc-users] DMDA halo exchange
In-Reply-To: <C1A96ECA-AC74-451B-B76B-81D7CA9CD968@mcs.anl.gov>
References: <551AB963.8060003@txcorp.com>
	<C1A96ECA-AC74-451B-B76B-81D7CA9CD968@mcs.anl.gov>
Message-ID: <551AD7A3.8030002@txcorp.com>

Great, thanks very much.
Cheers,
Dominic

On 03/31/2015 10:24 AM, Barry Smith wrote:
>    Dominic,
>
>      DMLocalToLocal is the intended usage. There is code inside the VecScatter to detect unneeded copies, which would result in only touching the ghost cells,  but for some reason that is not currently turned on, I'll take a look at fixing it in master.
>
>    Barry
>
>> On Mar 31, 2015, at 10:12 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:
>>
>> Hi,
>>
>> I have a DMDA for a structured grid in a 3D FDTD calculation. What is the most efficient way to update the ghost regions after each time step? Currently I update a global vector and then I scatter to a local vector using DMGlobalToLocalBegin/End for the next time step. I also tried DMLocalToLocal with the same source and target vector and that appears to be slightly faster, perhaps due to better cache usage in the copy of the owned region. Ideally I would like to update just the ghost cells from the owned regions. Is there a good way to do that? Something similar to VecGhostUpdateBegin/End?
>>
>> Thanks,
>> Dominic
>>
>> -- 
>> Dominic Meiser
>> Tech-X Corporation
>> 5621 Arapahoe Avenue
>> Boulder, CO 80303
>> USA
>> Telephone: 303-996-2036
>> Fax: 303-448-7756
>> www.txcorp.com
>>
>
>


-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com


From Carol.Brickley at awe.co.uk  Tue Mar 31 12:10:44 2015
From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk)
Date: Tue, 31 Mar 2015 17:10:44 +0000
Subject: [petsc-users] Problems trying to use -log_summary
Message-ID: <201503311710.t2VHAnpw002827@msw1.awe.co.uk>

Hi,

I am having problems trying to use -log_summary. I have added PetscLogBegin,PetscLogInitSummary  and compiled with PETSC_USE_LOG. I then run with -log_summary but the code will not accept this flag,

Any ideas?

Carol


Dr Carol Brickley
BSc,PhD,ARCS,DIC,MBCS

Senior Software Engineer
Applied Computer Science
DS+T,
AWE
Aldermaston
Reading
Berkshire
RG7 4PR

Direct: 0118 9855035




___________________________________________________
____________________________

The information in this email and in any attachment(s) is 
commercial in confidence. If you are not the named addressee(s) 
or 
if you receive this email in error then any distribution, copying or 
use of this communication or the information in it is strictly 
prohibited. Please notify us immediately by email at 
admin.internet(at)awe.co.uk, and then delete this message from 
your computer. While attachments are virus checked, AWE plc 
does not accept any liability in respect of any virus which is not 
detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/b803c383/attachment.html>

From knepley at gmail.com  Tue Mar 31 12:18:40 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 12:18:40 -0500
Subject: [petsc-users] Problems trying to use -log_summary
In-Reply-To: <201503311710.t2VHAnpw002827@msw1.awe.co.uk>
References: <201503311710.t2VHAnpw002827@msw1.awe.co.uk>
Message-ID: <CAMYG4GkJoXwrD+8OdVPrve3VbVA=1-XHTmKcdNVVYOXCWJzgNw@mail.gmail.com>

On Tue, Mar 31, 2015 at 12:10 PM, <Carol.Brickley at awe.co.uk> wrote:

>  Hi,
>
> I am having problems trying to use -log_summary. I have added
> PetscLogBegin,PetscLogInitSummary  and compiled with PETSC_USE_LOG. I then
> run with -log_summary but the code will not accept this flag,
>

Use the debugger and find out why its not executing the routine. It could
be that you did not rebuild everything correctly,
or that the option is not being passed correctly, etc. There are so many
variables that we cannot really be helpful yet. Also,
since this is used by almost every PETSc run done, it is very unlikely to
be a bug in the code.

  Thanks,

    Matt


>
> Any ideas?
>
> Carol
>
>
>
> *Dr Carol Brickley *
>
> *BSc,PhD,ARCS,DIC,MBCS*
>
>
>
> *Senior Software Engineer*
>
> *Applied Computer Science*
>
> *DS+T,*
>
> *AWE*
>
> *Aldermaston*
>
>  *Reading*
>
> *Berkshire*
>
> *RG7 4PR*
>
>
>
> *Direct: 0118 9855035*
>
>
>
>
> ___________________________________________________
> ____________________________ The information in this email and in any
> attachment(s) is commercial in confidence. If you are not the named
> addressee(s) or if you receive this email in error then any distribution,
> copying or use of this communication or the information in it is strictly
> prohibited. Please notify us immediately by email at admin.internet(at)
> awe.co.uk, and then delete this message from your computer. While
> attachments are virus checked, AWE plc does not accept any liability in
> respect of any virus which is not detected. AWE Plc Registered in England
> and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/29dbdea4/attachment.html>

From balay at mcs.anl.gov  Tue Mar 31 12:22:03 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 31 Mar 2015 12:22:03 -0500
Subject: [petsc-users] Problems trying to use -log_summary
In-Reply-To: <CAMYG4GkJoXwrD+8OdVPrve3VbVA=1-XHTmKcdNVVYOXCWJzgNw@mail.gmail.com>
References: <201503311710.t2VHAnpw002827@msw1.awe.co.uk>
	<CAMYG4GkJoXwrD+8OdVPrve3VbVA=1-XHTmKcdNVVYOXCWJzgNw@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1503311221280.4080@asterix>

Also check if petsc examples work [with -log_summary]. This will
isolate the issue [petsc build vs application code]

Satish

On Tue, 31 Mar 2015, Matthew Knepley wrote:

> On Tue, Mar 31, 2015 at 12:10 PM, <Carol.Brickley at awe.co.uk> wrote:
> 
> >  Hi,
> >
> > I am having problems trying to use -log_summary. I have added
> > PetscLogBegin,PetscLogInitSummary  and compiled with PETSC_USE_LOG. I then
> > run with -log_summary but the code will not accept this flag,
> >
> 
> Use the debugger and find out why its not executing the routine. It could
> be that you did not rebuild everything correctly,
> or that the option is not being passed correctly, etc. There are so many
> variables that we cannot really be helpful yet. Also,
> since this is used by almost every PETSc run done, it is very unlikely to
> be a bug in the code.
> 
>   Thanks,
> 
>     Matt
> 
> 
> >
> > Any ideas?
> >
> > Carol
> >
> >
> >
> > *Dr Carol Brickley *
> >
> > *BSc,PhD,ARCS,DIC,MBCS*
> >
> >
> >
> > *Senior Software Engineer*
> >
> > *Applied Computer Science*
> >
> > *DS+T,*
> >
> > *AWE*
> >
> > *Aldermaston*
> >
> >  *Reading*
> >
> > *Berkshire*
> >
> > *RG7 4PR*
> >
> >
> >
> > *Direct: 0118 9855035*
> >
> >
> >
> >
> > ___________________________________________________
> > ____________________________ The information in this email and in any
> > attachment(s) is commercial in confidence. If you are not the named
> > addressee(s) or if you receive this email in error then any distribution,
> > copying or use of this communication or the information in it is strictly
> > prohibited. Please notify us immediately by email at admin.internet(at)
> > awe.co.uk, and then delete this message from your computer. While
> > attachments are virus checked, AWE plc does not accept any liability in
> > respect of any virus which is not detected. AWE Plc Registered in England
> > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR
> >
> 
> 
> 
> 


From ling.zou at inl.gov  Tue Mar 31 13:55:36 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Tue, 31 Mar 2015 12:55:36 -0600
Subject: [petsc-users] Two questions on PETSc installation
Message-ID: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>

Hi All,

Following the instructions, I installed PETSc 3.5.2 on my Mac and
everything works great.
Two questions:
1, I suppose the name, PETSC_ARCH=arch-darwin-c-debug, suggests the
installation is a debug version?
2, How can I install a faster (optimized?) version of PETSc ?

Best,

Ling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/7613eb35/attachment.html>

From jed at jedbrown.org  Tue Mar 31 13:59:23 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 31 Mar 2015 12:59:23 -0600
Subject: [petsc-users] Two questions on PETSc installation
In-Reply-To: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
References: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
Message-ID: <87oan92dtg.fsf@jedbrown.org>

"Zou (Non-US), Ling" <ling.zou at inl.gov> writes:

> Hi All,
>
> Following the instructions, I installed PETSc 3.5.2 on my Mac and
> everything works great.
> Two questions:
> 1, I suppose the name, PETSC_ARCH=arch-darwin-c-debug, suggests the
> installation is a debug version?
> 2, How can I install a faster (optimized?) version of PETSc ?

--with-debugging=0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/12e04fed/attachment-0001.pgp>

From ling.zou at inl.gov  Tue Mar 31 14:02:16 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Tue, 31 Mar 2015 13:02:16 -0600
Subject: [petsc-users] Two questions on PETSc installation
In-Reply-To: <87oan92dtg.fsf@jedbrown.org>
References: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
	<87oan92dtg.fsf@jedbrown.org>
Message-ID: <CAJWxZDQrsEYE6Tr9MN2R=zAoYzamZ+kDVHz2Az0go9ZB1kfvHQ@mail.gmail.com>

Thank you Jed.

Ling

On Tue, Mar 31, 2015 at 12:59 PM, Jed Brown <jed at jedbrown.org> wrote:

> "Zou (Non-US), Ling" <ling.zou at inl.gov> writes:
>
> > Hi All,
> >
> > Following the instructions, I installed PETSc 3.5.2 on my Mac and
> > everything works great.
> > Two questions:
> > 1, I suppose the name, PETSC_ARCH=arch-darwin-c-debug, suggests the
> > installation is a debug version?
> > 2, How can I install a faster (optimized?) version of PETSc ?
>
> --with-debugging=0
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/244d3d94/attachment.html>

From bichinhoverde at spwinternet.com.br  Tue Mar 31 14:24:07 2015
From: bichinhoverde at spwinternet.com.br (bichinhoverde)
Date: Tue, 31 Mar 2015 16:24:07 -0300
Subject: [petsc-users] Two questions on PETSc installation
In-Reply-To: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
References: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
Message-ID: <CAD1bcpctDA5gZ43osvtebcitnFwniNWP27BkcgnPSua0fXmahA@mail.gmail.com>

--with-debugging=0 is the configure option to disable debug mode.

I use this configure line, which provides good optimization:

./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
--download-fblaslapack --download-mpich --with-debugging=0
COPTFLAGS='-Ofast -march=native -mtune=native' CXXOPTFLAGS='-Ofast
-march=native -mtune=native' FOPTFLAGS='-Ofast -march=native -mtune=native'




On Tue, Mar 31, 2015 at 3:55 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
wrote:

> Hi All,
>
> Following the instructions, I installed PETSc 3.5.2 on my Mac and
> everything works great.
> Two questions:
> 1, I suppose the name, PETSC_ARCH=arch-darwin-c-debug, suggests the
> installation is a debug version?
> 2, How can I install a faster (optimized?) version of PETSc ?
>
> Best,
>
> Ling
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/5d3901b2/attachment.html>

From j.pogacnik at auckland.ac.nz  Tue Mar 31 14:33:42 2015
From: j.pogacnik at auckland.ac.nz (Justin Pogacnik)
Date: Tue, 31 Mar 2015 19:33:42 +0000
Subject: [petsc-users] FEM 101
In-Reply-To: <CAMYG4GkabOonmntUwh=YXE3VnKkUfrgq0Ep0HgC6PVYmeE2_sQ@mail.gmail.com>
References: <38871A58892BC5409E1B66DFE5843DE791197AC9@uxcn10-1.UoA.auckland.ac.nz>,
	<CAMYG4GkabOonmntUwh=YXE3VnKkUfrgq0Ep0HgC6PVYmeE2_sQ@mail.gmail.com>
Message-ID: <38871A58892BC5409E1B66DFE5843DE791197C0E@uxcn10-1.UoA.auckland.ac.nz>

Yep, thanks for that Matt. I'll start playing around with the PetscFEIntegrate routines in C++. Is there any plan to make those accessible in F90? Also, I'm not seeing DMPlexIntegrateResidual.

Thanks for the help,

Justin
________________________________
From: Matthew Knepley [knepley at gmail.com]
Sent: Tuesday, March 31, 2015 5:14 PM
To: Justin Pogacnik
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] FEM 101

On Mon, Mar 30, 2015 at 4:39 PM, Justin Pogacnik <j.pogacnik at auckland.ac.nz<mailto:j.pogacnik at auckland.ac.nz>> wrote:
Hello,

I'm beginning to write a finite element solid mechanics code in F90 and am having some problems getting started with finite element basics within Petsc. I'm using an unstructured grid and the DMPlex class and able to access most of the mesh information required for FE numerical integration. I've used PetscFECreateDefault and have found the quadrature point locations/data. I'm trying to determine how to find the values of standard trilinear basis functions and their derivatives at the element quadrature points and then to perform the integration on the element. Can anyone advise what the proper routines are that I should call? It's possible that I'm way off target. I've seen "PetscFEGetBasisSpace", but I'm unsure what to do with the resulting output space. I've also seen the PetscFEIntegrate routines, but unsure how their intended to be used and don't see any fortran wrappers.

I hope this question is clear. Please let me know if you require more information. I appreciate any help and guidance that anyone can offer.

Here is the plain vanilla integration routine:

  https://bitbucket.org/petsc/petsc/src/1c9d18e7633f5b782021dca6d8727de2838ee261/src/dm/dt/interface/dtfe.c?at=master#cl-3380

which is an implementation of PetscFEIntegrate(). The idea is that this integrates a bunch of elements, and then there is an outer
loop over the whole mesh, which is currently DMPlexIntegrateResidual(). Unfortunately, this function also does a bunch of other
stuff, and right now I do not see how to simplify it.

Does this make sense?

  Thanks,

     Matt

Many thanks,

Justin



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/7209f9bb/attachment.html>

From john.mousel at gmail.com  Tue Mar 31 15:09:40 2015
From: john.mousel at gmail.com (John Mousel)
Date: Tue, 31 Mar 2015 15:09:40 -0500
Subject: [petsc-users] Propagating a constant marker with AMG
Message-ID: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>

I'm trying to mark two distinct regions of a domain by solving the Laplace
equation with constant Dirichlet BC's. Because I'm just rounding to the
nearest integer, I just need cheap AMG like behavior to move the constant
across the domain quickly. Are there any recommended solver options to do
this efficiently?

Thanks,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/5173d6c1/attachment.html>

From knepley at gmail.com  Tue Mar 31 15:20:13 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 15:20:13 -0500
Subject: [petsc-users] FEM 101
In-Reply-To: <38871A58892BC5409E1B66DFE5843DE791197C0E@uxcn10-1.UoA.auckland.ac.nz>
References: <38871A58892BC5409E1B66DFE5843DE791197AC9@uxcn10-1.UoA.auckland.ac.nz>
	<CAMYG4GkabOonmntUwh=YXE3VnKkUfrgq0Ep0HgC6PVYmeE2_sQ@mail.gmail.com>
	<38871A58892BC5409E1B66DFE5843DE791197C0E@uxcn10-1.UoA.auckland.ac.nz>
Message-ID: <CAMYG4GmZNkbc9OueZDB7w5p+Yq=BY_u=EvO7ii=tMuYzKAEaVA@mail.gmail.com>

On Tue, Mar 31, 2015 at 2:33 PM, Justin Pogacnik <j.pogacnik at auckland.ac.nz>
wrote:

>  Yep, thanks for that Matt. I'll start playing around with the
> PetscFEIntegrate routines in C++. Is there any plan to make those
> accessible in F90?
>

I will write any wrappers you need for F90. However, I wait until people
ask because its a pain with arrays. There are legit
reasons for writing in F90, but I would think deeply one more time before
committing to a new project in that language.


> Also, I'm not seeing DMPlexIntegrateResidual.
>

I typed it wrong. Its DMPlexComputeResidualFEM():


https://bitbucket.org/petsc/petsc/src/569f457242c46be7b11883bb57411ddc0c41c340/src/snes/utils/dmplexsnes.c?at=master#cl-1939

   Thanks,

     Matt


> Thanks for the help,
>
>  Justin
>  ------------------------------
> *From:* Matthew Knepley [knepley at gmail.com]
> *Sent:* Tuesday, March 31, 2015 5:14 PM
> *To:* Justin Pogacnik
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] FEM 101
>
>    On Mon, Mar 30, 2015 at 4:39 PM, Justin Pogacnik <
> j.pogacnik at auckland.ac.nz> wrote:
>
>>  Hello,
>>
>>  I'm beginning to write a finite element solid mechanics code in F90 and
>> am having some problems getting started with finite element basics within
>> Petsc. I'm using an unstructured grid and the DMPlex class and able to
>> access most of the mesh information required for FE numerical integration.
>> I've used PetscFECreateDefault and have found the quadrature point
>> locations/data. I'm trying to determine how to find the values of standard
>> trilinear basis functions and their derivatives at the element quadrature
>> points and then to perform the integration on the element. Can anyone
>> advise what the proper routines are that I should call? It's possible that
>> I'm way off target. I've seen "PetscFEGetBasisSpace", but I'm unsure what
>> to do with the resulting output space. I've also seen the PetscFEIntegrate
>> routines, but unsure how their intended to be used and don't see any
>> fortran wrappers.
>>
>>  I hope this question is clear. Please let me know if you require more
>> information. I appreciate any help and guidance that anyone can offer.
>>
>
>  Here is the plain vanilla integration routine:
>
>
> https://bitbucket.org/petsc/petsc/src/1c9d18e7633f5b782021dca6d8727de2838ee261/src/dm/dt/interface/dtfe.c?at=master#cl-3380
>
>  which is an implementation of PetscFEIntegrate(). The idea is that this
> integrates a bunch of elements, and then there is an outer
> loop over the whole mesh, which is currently DMPlexIntegrateResidual().
> Unfortunately, this function also does a bunch of other
> stuff, and right now I do not see how to simplify it.
>
>  Does this make sense?
>
>    Thanks,
>
>       Matt
>
>
>>  Many thanks,
>>
>>  Justin
>>
>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/e71bd0ad/attachment-0001.html>

From jed at jedbrown.org  Tue Mar 31 15:22:18 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 31 Mar 2015 14:22:18 -0600
Subject: [petsc-users] Propagating a constant marker with AMG
In-Reply-To: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>
References: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>
Message-ID: <87h9t03ojp.fsf@jedbrown.org>

John Mousel <john.mousel at gmail.com> writes:

> I'm trying to mark two distinct regions of a domain by solving the Laplace
> equation with constant Dirichlet BC's. Because I'm just rounding to the
> nearest integer, I just need cheap AMG like behavior to move the constant
> across the domain quickly. 

What is the physical propagation mechanism?  Is there a concept of
conservation that should be satisfied?  What's wrong with a normal AMG
configuration for the Laplacian?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/ba0fa1c1/attachment.pgp>

From john.mousel at gmail.com  Tue Mar 31 15:38:10 2015
From: john.mousel at gmail.com (John Mousel)
Date: Tue, 31 Mar 2015 15:38:10 -0500
Subject: [petsc-users] Propagating a constant marker with AMG
In-Reply-To: <87h9t03ojp.fsf@jedbrown.org>
References: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>
	<87h9t03ojp.fsf@jedbrown.org>
Message-ID: <CA+zegpnG51G6JTt1gvJd=R7JvDCBDhiUDm4YXecDYFdww44OPA@mail.gmail.com>

If I recall correctly, GAMG has a somewhat high setup cost and this is a
solver I'll only be using for one single solve. It's not a physical field,
it's just a marker; 1 inside, 0 outside a surface described by a CAD file.
This marker is being propagated into the volume of an octree volumetric
grid. I'm just taking advantage of the fact that the solution of the
Laplace equation with constant BC's is the constant.

On Tue, Mar 31, 2015 at 3:22 PM, Jed Brown <jed at jedbrown.org> wrote:

> John Mousel <john.mousel at gmail.com> writes:
>
> > I'm trying to mark two distinct regions of a domain by solving the
> Laplace
> > equation with constant Dirichlet BC's. Because I'm just rounding to the
> > nearest integer, I just need cheap AMG like behavior to move the constant
> > across the domain quickly.
>
> What is the physical propagation mechanism?  Is there a concept of
> conservation that should be satisfied?  What's wrong with a normal AMG
> configuration for the Laplacian?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/c2e6a746/attachment.html>

From jed at jedbrown.org  Tue Mar 31 15:47:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 31 Mar 2015 14:47:12 -0600
Subject: [petsc-users] Propagating a constant marker with AMG
In-Reply-To: <CA+zegpnG51G6JTt1gvJd=R7JvDCBDhiUDm4YXecDYFdww44OPA@mail.gmail.com>
References: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>
	<87h9t03ojp.fsf@jedbrown.org>
	<CA+zegpnG51G6JTt1gvJd=R7JvDCBDhiUDm4YXecDYFdww44OPA@mail.gmail.com>
Message-ID: <87bnj83ne7.fsf@jedbrown.org>

John Mousel <john.mousel at gmail.com> writes:

> If I recall correctly, GAMG has a somewhat high setup cost and this is a
> solver I'll only be using for one single solve. It's not a physical field,
> it's just a marker; 1 inside, 0 outside a surface described by a CAD file.
> This marker is being propagated into the volume of an octree volumetric
> grid. I'm just taking advantage of the fact that the solution of the
> Laplace equation with constant BC's is the constant.

Oh, I see.  Enforce your boundary conditions so that the problems on
each side decouple, then use -pc_gamg_agg_nsmooths 0 (unsmoothed
aggregation).  Profile and let us know if it's a bottleneck.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/d8bb8829/attachment.pgp>

From john.mousel at gmail.com  Tue Mar 31 15:48:11 2015
From: john.mousel at gmail.com (John Mousel)
Date: Tue, 31 Mar 2015 15:48:11 -0500
Subject: [petsc-users] Propagating a constant marker with AMG
In-Reply-To: <87bnj83ne7.fsf@jedbrown.org>
References: <CA+zegpn7Hnb5oiHq8w55XORxxspDoYyHxDpsga0-WtaMBX6kjw@mail.gmail.com>
	<87h9t03ojp.fsf@jedbrown.org>
	<CA+zegpnG51G6JTt1gvJd=R7JvDCBDhiUDm4YXecDYFdww44OPA@mail.gmail.com>
	<87bnj83ne7.fsf@jedbrown.org>
Message-ID: <CA+zegp=MZnMwaYihmz8SEF1kWrXW964WOZvTt8evPMK1Ddh9tQ@mail.gmail.com>

Thanks Jed,

That'll most likely do what I'm looking for.

John

On Tue, Mar 31, 2015 at 3:47 PM, Jed Brown <jed at jedbrown.org> wrote:

> John Mousel <john.mousel at gmail.com> writes:
>
> > If I recall correctly, GAMG has a somewhat high setup cost and this is a
> > solver I'll only be using for one single solve. It's not a physical
> field,
> > it's just a marker; 1 inside, 0 outside a surface described by a CAD
> file.
> > This marker is being propagated into the volume of an octree volumetric
> > grid. I'm just taking advantage of the fact that the solution of the
> > Laplace equation with constant BC's is the constant.
>
> Oh, I see.  Enforce your boundary conditions so that the problems on
> each side decouple, then use -pc_gamg_agg_nsmooths 0 (unsmoothed
> aggregation).  Profile and let us know if it's a bottleneck.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/2ba4ae0d/attachment.html>

From jchang27 at uh.edu  Tue Mar 31 17:06:45 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Tue, 31 Mar 2015 17:06:45 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
Message-ID: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>

Hi all,

So in my DMPlex FEM (serial) simulations, I want to output the solution U
to a binary file. I have the following:

ierr =
PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
ierr = VecView(U,viewer);CHKERRQ(ierr);
ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

The mesh I am solving has 13368 nodes, and 404 of them are constrained
using the DMPlexAddBoundary(...)  function. However, when I output the
vector I only get 12964 values, which I believe is leaving out the 404
constrained ones. How do I get all dofs in the vector U (both free and
constrained) printed?

Thanks,


-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/ed73411c/attachment.html>

From knepley at gmail.com  Tue Mar 31 17:13:55 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 17:13:55 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
In-Reply-To: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
References: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
Message-ID: <CAMYG4G=ukP0zh5Sg7X21rHOt5MAWbBEXaffpdKb=YJYZpOEyvA@mail.gmail.com>

On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang <jchang27 at uh.edu> wrote:

> Hi all,
>
> So in my DMPlex FEM (serial) simulations, I want to output the solution U
> to a binary file. I have the following:
>
> ierr =
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
> ierr = VecView(U,viewer);CHKERRQ(ierr);
> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>
> The mesh I am solving has 13368 nodes, and 404 of them are constrained
> using the DMPlexAddBoundary(...)  function. However, when I output the
> vector I only get 12964 values, which I believe is leaving out the 404
> constrained ones. How do I get all dofs in the vector U (both free and
> constrained) printed?
>

I would use HDF5 which will do this automatically:

  -vec_view hdf5:sol.h5

  Thanks,

     Matt


> Thanks,
>
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/753430bf/attachment.html>

From knepley at gmail.com  Tue Mar 31 17:46:34 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 17:46:34 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
In-Reply-To: <CAP2=TMgYLdohGVht+1uSjrjTqTLkkQyOVDUJ6mR2qg_zyB8H8g@mail.gmail.com>
References: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
	<CAMYG4G=ukP0zh5Sg7X21rHOt5MAWbBEXaffpdKb=YJYZpOEyvA@mail.gmail.com>
	<CAP2=TMgYLdohGVht+1uSjrjTqTLkkQyOVDUJ6mR2qg_zyB8H8g@mail.gmail.com>
Message-ID: <CAMYG4G=mVNxoWPukuHfuvqbD-gpgfzVwV8-Bg-hStZoxRdUo-g@mail.gmail.com>

On Tue, Mar 31, 2015 at 5:29 PM, Justin Chang <jchang27 at uh.edu> wrote:

> How do I invoke this command into my program? When I add "-vec_view
> hdf5:sol.h5" my program says this has been left out.
>

You can call VecView() directly with an HDF5 Viewer, or I think its easier
to call

  VecViewFromOptions(U, NULL, "-vec_view");

  Thanks,

    Matt


> On Tue, Mar 31, 2015 at 5:13 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>
>>> Hi all,
>>>
>>> So in my DMPlex FEM (serial) simulations, I want to output the solution
>>> U to a binary file. I have the following:
>>>
>>> ierr =
>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
>>> ierr = VecView(U,viewer);CHKERRQ(ierr);
>>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>>>
>>> The mesh I am solving has 13368 nodes, and 404 of them are constrained
>>> using the DMPlexAddBoundary(...)  function. However, when I output the
>>> vector I only get 12964 values, which I believe is leaving out the 404
>>> constrained ones. How do I get all dofs in the vector U (both free and
>>> constrained) printed?
>>>
>>
>> I would use HDF5 which will do this automatically:
>>
>>   -vec_view hdf5:sol.h5
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>>
>>>
>>> --
>>> Justin Chang
>>> PhD Candidate, Civil Engineering - Computational Sciences
>>> University of Houston, Department of Civil and Environmental Engineering
>>> Houston, TX 77004
>>> (512) 963-3262
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/dfd85deb/attachment-0001.html>

From jchang27 at uh.edu  Tue Mar 31 17:54:03 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Tue, 31 Mar 2015 17:54:03 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
In-Reply-To: <CAMYG4G=mVNxoWPukuHfuvqbD-gpgfzVwV8-Bg-hStZoxRdUo-g@mail.gmail.com>
References: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
	<CAMYG4G=ukP0zh5Sg7X21rHOt5MAWbBEXaffpdKb=YJYZpOEyvA@mail.gmail.com>
	<CAP2=TMgYLdohGVht+1uSjrjTqTLkkQyOVDUJ6mR2qg_zyB8H8g@mail.gmail.com>
	<CAMYG4G=mVNxoWPukuHfuvqbD-gpgfzVwV8-Bg-hStZoxRdUo-g@mail.gmail.com>
Message-ID: <CAP2=TMi4Nhwr+CEoJz+EK_pdJMS3oU=yWrSytSu8XFKaXLgsKw@mail.gmail.com>

Matt thanks for the responses. I plan on loading the solution sol.h5
obtained from one program into another (as the auxiliary). So in my other
program, I am solving a different problem but on the same mesh. I have the
following lines in my other code:

ierr =
PetscViewerHDF5Open(PETSC_COMM_SELF,"sol.h5",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
ierr = VecLoad(auxVec,&viewer);CHKERRQ(ierr);
ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

But when this program reaches VecLoad i get the following errors:

HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
  #000: H5D.c line 334 in H5Dopen2(): not found
    major: Dataset
    minor: Object not found
  #001: H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal
failed
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator
failed
    major: Symbol table
    minor: Callback failed
  #004: H5Gloc.c line 385 in H5G_loc_find_cb(): object
'Vec_0x7f9831516fb0_0' doesn't exist
    major: Symbol table
    minor: Object not found
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f  GIT
Date: 2015-03-29 20:36:38 -0500
[0]PETSC ERROR: ./trans on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU
by jychang48 Tue Mar 31 17:45:38 2015
[0]PETSC ERROR: Configure options --download-chaco --download-exodusii
--download-fblaslapack --download-hdf5 --download-metis --download-mumps
--download-netcdf --download-openmpi --download-parmetis
--download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake
--with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1
CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug
[0]PETSC ERROR: #1 VecLoad_HDF5() line 268 in
/Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #2 VecLoad_Default() line 391 in
/Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #3 VecLoad_Plex_Local() line 158 in
/Users/jychang48/Software/petsc-dev/src/dm/impls/plex/plex.c
[0]PETSC ERROR: #4 VecLoad() line 969 in
/Users/jychang48/Software/petsc-dev/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #5 CreateMesh() line 265 in
/Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c
[0]PETSC ERROR: #6 main() line 381 in
/Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c

Any idea what's going on here?

On Tue, Mar 31, 2015 at 5:46 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Mar 31, 2015 at 5:29 PM, Justin Chang <jchang27 at uh.edu> wrote:
>
>> How do I invoke this command into my program? When I add "-vec_view
>> hdf5:sol.h5" my program says this has been left out.
>>
>
> You can call VecView() directly with an HDF5 Viewer, or I think its easier
> to call
>
>   VecViewFromOptions(U, NULL, "-vec_view");
>
>   Thanks,
>
>     Matt
>
>
>> On Tue, Mar 31, 2015 at 5:13 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>
>>>> Hi all,
>>>>
>>>> So in my DMPlex FEM (serial) simulations, I want to output the solution
>>>> U to a binary file. I have the following:
>>>>
>>>> ierr =
>>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
>>>> ierr = VecView(U,viewer);CHKERRQ(ierr);
>>>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>>>>
>>>> The mesh I am solving has 13368 nodes, and 404 of them are constrained
>>>> using the DMPlexAddBoundary(...)  function. However, when I output the
>>>> vector I only get 12964 values, which I believe is leaving out the 404
>>>> constrained ones. How do I get all dofs in the vector U (both free and
>>>> constrained) printed?
>>>>
>>>
>>> I would use HDF5 which will do this automatically:
>>>
>>>   -vec_view hdf5:sol.h5
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>>
>>>>
>>>> --
>>>> Justin Chang
>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>> University of Houston, Department of Civil and Environmental Engineering
>>>> Houston, TX 77004
>>>> (512) 963-3262
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Justin Chang
>> PhD Candidate, Civil Engineering - Computational Sciences
>> University of Houston, Department of Civil and Environmental Engineering
>> Houston, TX 77004
>> (512) 963-3262
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/dcf16a02/attachment.html>

From knepley at gmail.com  Tue Mar 31 18:00:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 18:00:03 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
In-Reply-To: <CAP2=TMi4Nhwr+CEoJz+EK_pdJMS3oU=yWrSytSu8XFKaXLgsKw@mail.gmail.com>
References: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
	<CAMYG4G=ukP0zh5Sg7X21rHOt5MAWbBEXaffpdKb=YJYZpOEyvA@mail.gmail.com>
	<CAP2=TMgYLdohGVht+1uSjrjTqTLkkQyOVDUJ6mR2qg_zyB8H8g@mail.gmail.com>
	<CAMYG4G=mVNxoWPukuHfuvqbD-gpgfzVwV8-Bg-hStZoxRdUo-g@mail.gmail.com>
	<CAP2=TMi4Nhwr+CEoJz+EK_pdJMS3oU=yWrSytSu8XFKaXLgsKw@mail.gmail.com>
Message-ID: <CAMYG4G=gtf4rmAXVwmCs1_9pXtOCT7=0CU+8XW=A7+gbXttXTg@mail.gmail.com>

On Tue, Mar 31, 2015 at 5:54 PM, Justin Chang <jchang27 at uh.edu> wrote:

> Matt thanks for the responses. I plan on loading the solution sol.h5
> obtained from one program into another (as the auxiliary). So in my other
> program, I am solving a different problem but on the same mesh. I have the
> following lines in my other code:
>

Yes, if you want to do this, you have to give these vectors proper names:

  PetscObjectSetName((PetscObject), U, "solution");

  Thanks,

    Matt


> ierr =
> PetscViewerHDF5Open(PETSC_COMM_SELF,"sol.h5",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
> ierr = VecLoad(auxVec,&viewer);CHKERRQ(ierr);
> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>
> But when this program reaches VecLoad i get the following errors:
>
> HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
>   #000: H5D.c line 334 in H5Dopen2(): not found
>     major: Dataset
>     minor: Object not found
>   #001: H5Gloc.c line 430 in H5G_loc_find(): can't find object
>     major: Symbol table
>     minor: Object not found
>   #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal
> failed
>     major: Symbol table
>     minor: Object not found
>   #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator
> failed
>     major: Symbol table
>     minor: Callback failed
>   #004: H5Gloc.c line 385 in H5G_loc_find_cb(): object
> 'Vec_0x7f9831516fb0_0' doesn't exist
>     major: Symbol table
>     minor: Object not found
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f  GIT
> Date: 2015-03-29 20:36:38 -0500
> [0]PETSC ERROR: ./trans on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU
> by jychang48 Tue Mar 31 17:45:38 2015
> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii
> --download-fblaslapack --download-hdf5 --download-metis --download-mumps
> --download-netcdf --download-openmpi --download-parmetis
> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake
> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1
> CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug
> [0]PETSC ERROR: #1 VecLoad_HDF5() line 268 in
> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #2 VecLoad_Default() line 391 in
> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #3 VecLoad_Plex_Local() line 158 in
> /Users/jychang48/Software/petsc-dev/src/dm/impls/plex/plex.c
> [0]PETSC ERROR: #4 VecLoad() line 969 in
> /Users/jychang48/Software/petsc-dev/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #5 CreateMesh() line 265 in
> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c
> [0]PETSC ERROR: #6 main() line 381 in
> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c
>
> Any idea what's going on here?
>
> On Tue, Mar 31, 2015 at 5:46 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Mar 31, 2015 at 5:29 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>
>>> How do I invoke this command into my program? When I add "-vec_view
>>> hdf5:sol.h5" my program says this has been left out.
>>>
>>
>> You can call VecView() directly with an HDF5 Viewer, or I think its
>> easier to call
>>
>>   VecViewFromOptions(U, NULL, "-vec_view");
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> On Tue, Mar 31, 2015 at 5:13 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> So in my DMPlex FEM (serial) simulations, I want to output the
>>>>> solution U to a binary file. I have the following:
>>>>>
>>>>> ierr =
>>>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
>>>>> ierr = VecView(U,viewer);CHKERRQ(ierr);
>>>>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>>>>>
>>>>> The mesh I am solving has 13368 nodes, and 404 of them are constrained
>>>>> using the DMPlexAddBoundary(...)  function. However, when I output the
>>>>> vector I only get 12964 values, which I believe is leaving out the 404
>>>>> constrained ones. How do I get all dofs in the vector U (both free and
>>>>> constrained) printed?
>>>>>
>>>>
>>>> I would use HDF5 which will do this automatically:
>>>>
>>>>   -vec_view hdf5:sol.h5
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> --
>>>>> Justin Chang
>>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>>> University of Houston, Department of Civil and Environmental
>>>>> Engineering
>>>>> Houston, TX 77004
>>>>> (512) 963-3262
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>>
>>> --
>>> Justin Chang
>>> PhD Candidate, Civil Engineering - Computational Sciences
>>> University of Houston, Department of Civil and Environmental Engineering
>>> Houston, TX 77004
>>> (512) 963-3262
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/91fa2db9/attachment.html>

From jchang27 at uh.edu  Tue Mar 31 18:08:45 2015
From: jchang27 at uh.edu (Justin Chang)
Date: Tue, 31 Mar 2015 18:08:45 -0500
Subject: [petsc-users] Obtaining constraint values in DM vector solution
In-Reply-To: <CAMYG4G=gtf4rmAXVwmCs1_9pXtOCT7=0CU+8XW=A7+gbXttXTg@mail.gmail.com>
References: <CAP2=TMgYTMjLe1FizAotJB0jCahftwnrkUGdfSeZxdnDSb5WRg@mail.gmail.com>
	<CAMYG4G=ukP0zh5Sg7X21rHOt5MAWbBEXaffpdKb=YJYZpOEyvA@mail.gmail.com>
	<CAP2=TMgYLdohGVht+1uSjrjTqTLkkQyOVDUJ6mR2qg_zyB8H8g@mail.gmail.com>
	<CAMYG4G=mVNxoWPukuHfuvqbD-gpgfzVwV8-Bg-hStZoxRdUo-g@mail.gmail.com>
	<CAP2=TMi4Nhwr+CEoJz+EK_pdJMS3oU=yWrSytSu8XFKaXLgsKw@mail.gmail.com>
	<CAMYG4G=gtf4rmAXVwmCs1_9pXtOCT7=0CU+8XW=A7+gbXttXTg@mail.gmail.com>
Message-ID: <CAP2=TMg+K4exbfL6s=iLwUov_SqWwV34xwEJx5kSX9Bf1EWFAA@mail.gmail.com>

In my first program I have already set the name for the solution. Vec U has
the name "pressure".

In my second program, how do I obtain and use/load "pressure"? Would I have
to use PetscObjectQuery somehow?

On Tue, Mar 31, 2015 at 6:00 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Mar 31, 2015 at 5:54 PM, Justin Chang <jchang27 at uh.edu> wrote:
>
>> Matt thanks for the responses. I plan on loading the solution sol.h5
>> obtained from one program into another (as the auxiliary). So in my other
>> program, I am solving a different problem but on the same mesh. I have the
>> following lines in my other code:
>>
>
> Yes, if you want to do this, you have to give these vectors proper names:
>
>   PetscObjectSetName((PetscObject), U, "solution");
>
>   Thanks,
>
>     Matt
>
>
>> ierr =
>> PetscViewerHDF5Open(PETSC_COMM_SELF,"sol.h5",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
>> ierr = VecLoad(auxVec,&viewer);CHKERRQ(ierr);
>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>>
>> But when this program reaches VecLoad i get the following errors:
>>
>> HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0:
>>   #000: H5D.c line 334 in H5Dopen2(): not found
>>     major: Dataset
>>     minor: Object not found
>>   #001: H5Gloc.c line 430 in H5G_loc_find(): can't find object
>>     major: Symbol table
>>     minor: Object not found
>>   #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal
>> failed
>>     major: Symbol table
>>     minor: Object not found
>>   #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator
>> failed
>>     major: Symbol table
>>     minor: Callback failed
>>   #004: H5Gloc.c line 385 in H5G_loc_find_cb(): object
>> 'Vec_0x7f9831516fb0_0' doesn't exist
>>     major: Symbol table
>>     minor: Object not found
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Error in external library
>> [0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f  GIT
>> Date: 2015-03-29 20:36:38 -0500
>> [0]PETSC ERROR: ./trans on a arch-darwin-c-debug named
>> CAML-03.CIVE.UH.EDU by jychang48 Tue Mar 31 17:45:38 2015
>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii
>> --download-fblaslapack --download-hdf5 --download-metis --download-mumps
>> --download-netcdf --download-openmpi --download-parmetis
>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake
>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1
>> CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug
>> [0]PETSC ERROR: #1 VecLoad_HDF5() line 268 in
>> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
>> [0]PETSC ERROR: #2 VecLoad_Default() line 391 in
>> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c
>> [0]PETSC ERROR: #3 VecLoad_Plex_Local() line 158 in
>> /Users/jychang48/Software/petsc-dev/src/dm/impls/plex/plex.c
>> [0]PETSC ERROR: #4 VecLoad() line 969 in
>> /Users/jychang48/Software/petsc-dev/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: #5 CreateMesh() line 265 in
>> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c
>> [0]PETSC ERROR: #6 main() line 381 in
>> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c
>>
>> Any idea what's going on here?
>>
>> On Tue, Mar 31, 2015 at 5:46 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Mar 31, 2015 at 5:29 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>
>>>> How do I invoke this command into my program? When I add "-vec_view
>>>> hdf5:sol.h5" my program says this has been left out.
>>>>
>>>
>>> You can call VecView() directly with an HDF5 Viewer, or I think its
>>> easier to call
>>>
>>>   VecViewFromOptions(U, NULL, "-vec_view");
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> On Tue, Mar 31, 2015 at 5:13 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang <jchang27 at uh.edu> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> So in my DMPlex FEM (serial) simulations, I want to output the
>>>>>> solution U to a binary file. I have the following:
>>>>>>
>>>>>> ierr =
>>>>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr);
>>>>>> ierr = VecView(U,viewer);CHKERRQ(ierr);
>>>>>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
>>>>>>
>>>>>> The mesh I am solving has 13368 nodes, and 404 of them are
>>>>>> constrained using the DMPlexAddBoundary(...)  function. However, when I
>>>>>> output the vector I only get 12964 values, which I believe is leaving out
>>>>>> the 404 constrained ones. How do I get all dofs in the vector U (both free
>>>>>> and constrained) printed?
>>>>>>
>>>>>
>>>>> I would use HDF5 which will do this automatically:
>>>>>
>>>>>   -vec_view hdf5:sol.h5
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Justin Chang
>>>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>>>> University of Houston, Department of Civil and Environmental
>>>>>> Engineering
>>>>>> Houston, TX 77004
>>>>>> (512) 963-3262
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Justin Chang
>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>> University of Houston, Department of Civil and Environmental Engineering
>>>> Houston, TX 77004
>>>> (512) 963-3262
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Justin Chang
>> PhD Candidate, Civil Engineering - Computational Sciences
>> University of Houston, Department of Civil and Environmental Engineering
>> Houston, TX 77004
>> (512) 963-3262
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Justin Chang
PhD Candidate, Civil Engineering - Computational Sciences
University of Houston, Department of Civil and Environmental Engineering
Houston, TX 77004
(512) 963-3262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/1c5ca187/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Mar 31 18:19:21 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 31 Mar 2015 18:19:21 -0500
Subject: [petsc-users] Two questions on PETSc installation
In-Reply-To: <87oan92dtg.fsf@jedbrown.org>
References: <CAJWxZDQXFjgw+Fx2+4ckK46Q2jrE=+rS1S45MBEcYbwHYttkDg@mail.gmail.com>
	<87oan92dtg.fsf@jedbrown.org>
Message-ID: <75A0359C-905F-4A6F-AB65-4EFCBB0A9443@mcs.anl.gov>


> On Mar 31, 2015, at 1:59 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> "Zou (Non-US), Ling" <ling.zou at inl.gov> writes:
> 
>> Hi All,
>> 
>> Following the instructions, I installed PETSc 3.5.2 on my Mac and
>> everything works great.
>> Two questions:
>> 1, I suppose the name, PETSC_ARCH=arch-darwin-c-debug, suggests the
>> installation is a debug version?
>> 2, How can I install a faster (optimized?) version of PETSc ?
> 
> --with-debugging=0

  When you run ./configure again.   



From stm8086 at yahoo.com  Tue Mar 31 18:51:57 2015
From: stm8086 at yahoo.com (Steena M)
Date: Tue, 31 Mar 2015 16:51:57 -0700
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <5819F2A4-76EF-459A-B0A6-72A98071D093@mcs.anl.gov>
Message-ID: <1427845917.6226.YahooMailBasic@web125404.mail.ne1.yahoo.com>

Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. 
--------------------------------------------
On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:

 Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
 To: "Steena M" <stm8086 at yahoo.com>
 Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
 Date: Sunday, March 29, 2015, 9:26 PM
 
 
 > On
 Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
 wrote:
 > 
 > Thanks
 Matt. I used PETSC_DETERMINE but I'm now getting an
 allocation-based error:
 > 
 > [0]PETSC ERROR: ---------------------
 Error Message ------------------------------------
 > [0]PETSC ERROR: Argument out of range!
 > [0]PETSC ERROR: New nonzero at (2,18)
 caused a malloc!
 > [0]PETSC ERROR:
 ------------------------------------------------------------------------
 > 
 > I tried
 preallocating on each rank for the diagonal and off diagonal
 section of the matrix as the next step? My current
 approximations for preallocation
 > 
 > CHKERRQ(
 MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));?
 
 
 ???These
 arguments where you pass PETSC_DEFAULT are expecting a
 pointer not an integer. You can pass NULL in those
 locations. Though it is better to provide the correct
 preallocation rather than some defaults.
 
 ? Barry
 
 > 
 > are throwing segmentation errors.
 > 
 > [0]PETSC ERROR:?
 Caught signal number 11 SEGV: Segmentation Violation,
 probably memory access out of range
 > 
 > Any insights into what I'm doing
 wrong?
 > 
 > Thanks,
 > Steena
 > 
 > 
 > 
 > On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
 wrote:
 > 
 > Subject:
 Re: [petsc-users] Unequal sparse matrix row distribution for
 MPI MatMult
 > To: "Steena M"
 <stm8086 at yahoo.com>
 > Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
 petsc-users at mcs.anl.gov
 > Date: Sunday, March 29, 2015, 10:02 PM
 > 
 > On Sun, Mar 29, 2015
 at
 > 9:56 PM, Steena M <stm8086 at yahoo.com>
 > wrote:
 > Hi
 > Barry,
 > 
 > 
 > 
 > I am trying to partition a 20 row and 20
 col sparse matrix
 > between two procs
 such that proc 0 has 15 rows and 20 cols
 > and proc 1 has 5 rows and 20 cols. The
 code snippet:
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >? ? ?
 ???CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
 //
 > at runtime: -matload_block_size 1
 > 
 > 
 > 
 > 
 > 
 >? ? ?
 ???if (rank ==0)
 > 
 >? ? ? ???{
 > 
 >? ? ? ? ? ? ?
 ???CHKERRQ( MatSetSizes(A, 15, 20, 20,
 > 20) ); //rank 0 gets 75% of the rows
 > 
 >? ? ? ? ? ? ?
 ???CHKERRQ( MatSetType(A, MATMPIBAIJ)
 > );
 > 
 >? ? ? ? ? ? ? ???CHKERRQ(
 MatLoad(A,fd) );
 > 
 >? ? ? ? ? }
 > 
 > 
 > 
 >? ? ? ???else
 > 
 >? ? ?
 ???{
 > 
 >?
 ? ? ? ? ? ? ???CHKERRQ( MatSetSizes(A, 5,
 20, 20,
 > 20) ); //rank 1 gets 25% of the
 rows
 > 
 >? ? ? ?
 ? ? ? ???CHKERRQ( MatSetType(A, MATMPIBAIJ)
 > );
 > 
 >? ? ? ? ? ? ? ???CHKERRQ(
 MatLoad(A,fd) );
 > 
 >? ? ???}
 >
 
 > 
 > 
 > This throws the following error (probably
 from psplit.c):
 > 
 >
 [1]PETSC ERROR: --------------------- Error Message
 > ------------------------------------
 > 
 > [1]PETSC ERROR:
 Nonconforming object sizes!
 > 
 > [1]PETSC ERROR: Sum of local lengths 40
 does not equal
 > global length 20, my
 local length 20
 > 
 >???likely a call to
 VecSetSizes() or MatSetSizes() is
 >
 wrong.
 > 
 > See
 http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
 > 
 > 
 > 
 > This error printout
 doesn't quite make sense to me.
 >
 I'm trying to specify a total matrix size of 20x20...
 I
 > haven't yet figured out where the
 '40' comes
 > from in the error
 message.
 > 
 > 
 > 
 > Any thoughts on what
 might be going wrong?
 > 
 > Its the column specification. Just
 > use PETSC_DETERMINE for the local columns
 since all our
 > sparse matrixformats are
 row divisions
 > anyway.
 >? 
 > Thanks,
 >? ? 
 >? Matt 
 > Thanks in advance,
 >
 
 > Steena
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >
 --------------------------------------------
 > 
 > On Sun, 3/22/15,
 Barry Smith <bsmith at mcs.anl.gov>
 > wrote:
 > 
 > 
 > 
 >? Subject: Re: [petsc-users] Unequal
 sparse matrix row
 > distribution for MPI
 MatMult
 > 
 >? To:
 "Steena M" <stm8086 at yahoo.com>
 > 
 >? Cc: petsc-users at mcs.anl.gov
 > 
 >? Date: Sunday,
 March 22, 2015, 3:58 PM
 > 
 > 
 > 
 > 
 > 
 >???
 > 
 >? Steena,
 > 
 > 
 > 
 >? ???I am
 >
 
 >? a little unsure of your question.
 
 > 
 > 
 > 
 >? ???1)
 you can create a MPIBAIJ
 > 
 >? matrix with any distribution of block
 rows per process
 > you
 > 
 >? want, just set the
 local row size for each process to
 >
 be
 > 
 >? what you
 like.? Use MatCreateVecs() to get
 >
 correspondingly
 > 
 >?
 laid out vectors.
 > 
 >
 
 > 
 >?
 ???or 2) if you have a MPIBAIJ
 > 
 >? matrix with
 "equal" row layout and you want a
 > new
 > 
 >? one with uneven row layout you can
 simply use
 > 
 >?
 MatGetSubMatrix() to create that new matrix.
 > 
 > 
 > 
 >? ? Barry
 > 
 > 
 > 
 >? Unless you have
 another reason to have the
 > 
 >? matrix with an equal number row layout I
 would just
 > generate
 > 
 >? the matrix with
 the layout you want.
 > 
 > 
 > 
 > 
 > 
 >? > On Mar 22, 2015, at 5:50 PM, Steena
 M
 > 
 >? <stm8086 at yahoo.com>
 > 
 >? wrote:
 > 
 >? >
 > 
 >? > Hello,
 > 
 >? >
 > 
 >? > I need to
 distribute
 > 
 >? a
 sparse matrix such that each proc owns an unequal
 > number
 > 
 >? of blocked rows before I proceed with
 MPI MatMult. My
 > 
 >?
 initial thoughts on doing this:
 > 
 >? >
 > 
 >? > 1) Use? MatGetSubMatrices() on the
 test
 > 
 >? MATMPIBAIJ
 matrix to produce a new matrix where each
 > proc
 > 
 >? has an unequal number of rows.
 > 
 >? >
 > 
 >? > 2) Provide
 scatter context for vector X
 > 
 >? (for MatMult )using IS iscol from
 MatGetSubMatrices()
 > while
 > 
 >? creating the
 vector X.
 > 
 >?
 >
 > 
 >? > 3)
 Call MatMult()
 > 
 >?
 >
 > 
 >? > Will
 MatMult_MPIBAIJ continue to scatter
 > 
 >? this matrix and vector such that each
 proc will own an
 > equal
 > 
 >? number of matrix
 rows and corresponding diagonal vector
 >
 
 >? elements? Should I write my own
 MPIMatMult function to
 > 
 >? retain my redistribution of the matrix
 and vector?
 > 
 >?
 >
 > 
 >? >
 Thanks in
 > 
 >?
 advance,
 > 
 >? >
 Steena
 > 
 > 
 > 
 > 
 > 
 > 
 > -- 
 > What most
 experimenters
 > take for granted before
 they begin their experiments is
 >
 infinitely more interesting than any results to which
 their
 > experiments lead.
 > -- Norbert
 > Wiener
 > 
 

From knepley at gmail.com  Tue Mar 31 20:10:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 20:10:16 -0500
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <1427845917.6226.YahooMailBasic@web125404.mail.ne1.yahoo.com>
References: <5819F2A4-76EF-459A-B0A6-72A98071D093@mcs.anl.gov>
	<1427845917.6226.YahooMailBasic@web125404.mail.ne1.yahoo.com>
Message-ID: <CAMYG4GmQ=t+HdPvc27+5WC6LNwLJQUnR-9fP86zqC28aNdwDqQ@mail.gmail.com>

On Tue, Mar 31, 2015 at 6:51 PM, Steena M <stm8086 at yahoo.com> wrote:

> Thanks Barry. I'm still getting the malloc error with NULL. Is there a way
> to distribute the matrix without explicit preallocation? Different matrices
> will be loaded during runtime and assigning preallocation parameters would
> mean an additional preprocessing step.
>

1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)

2) Note that this is never ever ever more efficient than making another
pass and preallocating

  Thanks,

     Matt

> --------------------------------------------
> On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>  Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI
> MatMult
>  To: "Steena M" <stm8086 at yahoo.com>
>  Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
>  Date: Sunday, March 29, 2015, 9:26 PM
>
>
>  > On
>  Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
>  wrote:
>  >
>  > Thanks
>  Matt. I used PETSC_DETERMINE but I'm now getting an
>  allocation-based error:
>  >
>  > [0]PETSC ERROR: ---------------------
>  Error Message ------------------------------------
>  > [0]PETSC ERROR: Argument out of range!
>  > [0]PETSC ERROR: New nonzero at (2,18)
>  caused a malloc!
>  > [0]PETSC ERROR:
>  ------------------------------------------------------------------------
>  >
>  > I tried
>  preallocating on each rank for the diagonal and off diagonal
>  section of the matrix as the next step  My current
>  approximations for preallocation
>  >
>  > CHKERRQ(
>  MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));
>
>
>     These
>  arguments where you pass PETSC_DEFAULT are expecting a
>  pointer not an integer. You can pass NULL in those
>  locations. Though it is better to provide the correct
>  preallocation rather than some defaults.
>
>    Barry
>
>  >
>  > are throwing segmentation errors.
>  >
>  > [0]PETSC ERROR:
>  Caught signal number 11 SEGV: Segmentation Violation,
>  probably memory access out of range
>  >
>  > Any insights into what I'm doing
>  wrong?
>  >
>  > Thanks,
>  > Steena
>  >
>  >
>  >
>  > On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
>  wrote:
>  >
>  > Subject:
>  Re: [petsc-users] Unequal sparse matrix row distribution for
>  MPI MatMult
>  > To: "Steena M"
>  <stm8086 at yahoo.com>
>  > Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
>  petsc-users at mcs.anl.gov
>  > Date: Sunday, March 29, 2015, 10:02 PM
>  >
>  > On Sun, Mar 29, 2015
>  at
>  > 9:56 PM, Steena M <stm8086 at yahoo.com>
>  > wrote:
>  > Hi
>  > Barry,
>  >
>  >
>  >
>  > I am trying to partition a 20 row and 20
>  col sparse matrix
>  > between two procs
>  such that proc 0 has 15 rows and 20 cols
>  > and proc 1 has 5 rows and 20 cols. The
>  code snippet:
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>     CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
>  //
>  > at runtime: -matload_block_size 1
>  >
>  >
>  >
>  >
>  >
>  >
>     if (rank ==0)
>  >
>  >         {
>  >
>  >
>     CHKERRQ( MatSetSizes(A, 15, 20, 20,
>  > 20) ); //rank 0 gets 75% of the rows
>  >
>  >
>     CHKERRQ( MatSetType(A, MATMPIBAIJ)
>  > );
>  >
>  >                 CHKERRQ(
>  MatLoad(A,fd) );
>  >
>  >          }
>  >
>  >
>  >
>  >         else
>  >
>  >
>     {
>  >
>  >
>                 CHKERRQ( MatSetSizes(A, 5,
>  20, 20,
>  > 20) ); //rank 1 gets 25% of the
>  rows
>  >
>  >
>           CHKERRQ( MatSetType(A, MATMPIBAIJ)
>  > );
>  >
>  >                 CHKERRQ(
>  MatLoad(A,fd) );
>  >
>  >       }
>  >
>
>  >
>  >
>  > This throws the following error (probably
>  from psplit.c):
>  >
>  >
>  [1]PETSC ERROR: --------------------- Error Message
>  > ------------------------------------
>  >
>  > [1]PETSC ERROR:
>  Nonconforming object sizes!
>  >
>  > [1]PETSC ERROR: Sum of local lengths 40
>  does not equal
>  > global length 20, my
>  local length 20
>  >
>  >   likely a call to
>  VecSetSizes() or MatSetSizes() is
>  >
>  wrong.
>  >
>  > See
>  http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
>  >
>  >
>  >
>  > This error printout
>  doesn't quite make sense to me.
>  >
>  I'm trying to specify a total matrix size of 20x20...
>  I
>  > haven't yet figured out where the
>  '40' comes
>  > from in the error
>  message.
>  >
>  >
>  >
>  > Any thoughts on what
>  might be going wrong?
>  >
>  > Its the column specification. Just
>  > use PETSC_DETERMINE for the local columns
>  since all our
>  > sparse matrixformats are
>  row divisions
>  > anyway.
>  >
>  > Thanks,
>  >
>  >  Matt
>  > Thanks in advance,
>  >
>
>  > Steena
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  --------------------------------------------
>  >
>  > On Sun, 3/22/15,
>  Barry Smith <bsmith at mcs.anl.gov>
>  > wrote:
>  >
>  >
>  >
>  >  Subject: Re: [petsc-users] Unequal
>  sparse matrix row
>  > distribution for MPI
>  MatMult
>  >
>  >  To:
>  "Steena M" <stm8086 at yahoo.com>
>  >
>  >  Cc: petsc-users at mcs.anl.gov
>  >
>  >  Date: Sunday,
>  March 22, 2015, 3:58 PM
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  >  Steena,
>  >
>  >
>  >
>  >     I am
>  >
>
>  >  a little unsure of your question.
>
>  >
>  >
>  >
>  >     1)
>  you can create a MPIBAIJ
>  >
>  >  matrix with any distribution of block
>  rows per process
>  > you
>  >
>  >  want, just set the
>  local row size for each process to
>  >
>  be
>  >
>  >  what you
>  like.  Use MatCreateVecs() to get
>  >
>  correspondingly
>  >
>  >
>  laid out vectors.
>  >
>  >
>
>  >
>  >
>     or 2) if you have a MPIBAIJ
>  >
>  >  matrix with
>  "equal" row layout and you want a
>  > new
>  >
>  >  one with uneven row layout you can
>  simply use
>  >
>  >
>  MatGetSubMatrix() to create that new matrix.
>  >
>  >
>  >
>  >    Barry
>  >
>  >
>  >
>  >  Unless you have
>  another reason to have the
>  >
>  >  matrix with an equal number row layout I
>  would just
>  > generate
>  >
>  >  the matrix with
>  the layout you want.
>  >
>  >
>  >
>  >
>  >
>  >  > On Mar 22, 2015, at 5:50 PM, Steena
>  M
>  >
>  >  <stm8086 at yahoo.com>
>  >
>  >  wrote:
>  >
>  >  >
>  >
>  >  > Hello,
>  >
>  >  >
>  >
>  >  > I need to
>  distribute
>  >
>  >  a
>  sparse matrix such that each proc owns an unequal
>  > number
>  >
>  >  of blocked rows before I proceed with
>  MPI MatMult. My
>  >
>  >
>  initial thoughts on doing this:
>  >
>  >  >
>  >
>  >  > 1) Use  MatGetSubMatrices() on the
>  test
>  >
>  >  MATMPIBAIJ
>  matrix to produce a new matrix where each
>  > proc
>  >
>  >  has an unequal number of rows.
>  >
>  >  >
>  >
>  >  > 2) Provide
>  scatter context for vector X
>  >
>  >  (for MatMult )using IS iscol from
>  MatGetSubMatrices()
>  > while
>  >
>  >  creating the
>  vector X.
>  >
>  >
>  >
>  >
>  >  > 3)
>  Call MatMult()
>  >
>  >
>  >
>  >
>  >  > Will
>  MatMult_MPIBAIJ continue to scatter
>  >
>  >  this matrix and vector such that each
>  proc will own an
>  > equal
>  >
>  >  number of matrix
>  rows and corresponding diagonal vector
>  >
>
>  >  elements? Should I write my own
>  MPIMatMult function to
>  >
>  >  retain my redistribution of the matrix
>  and vector?
>  >
>  >
>  >
>  >
>  >  >
>  Thanks in
>  >
>  >
>  advance,
>  >
>  >  >
>  Steena
>  >
>  >
>  >
>  >
>  >
>  >
>  > --
>  > What most
>  experimenters
>  > take for granted before
>  they begin their experiments is
>  >
>  infinitely more interesting than any results to which
>  their
>  > experiments lead.
>  > -- Norbert
>  > Wiener
>  >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/d8847929/attachment-0001.html>

From emconsta at mcs.anl.gov  Tue Mar 31 20:16:51 2015
From: emconsta at mcs.anl.gov (Emil Constantinescu)
Date: Tue, 31 Mar 2015 20:16:51 -0500
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <1427769165045.62947bec@Nodemailer>
References: <5519F72B.3060308@mcs.anl.gov> <1427769165045.62947bec@Nodemailer>
Message-ID: <551B4703.9040005@mcs.anl.gov>

Hi Andrew - we found a bug in TSComputeRHSFunctionLinear.

In your code you need to provide the context to TSSetRHSFunction:
TSSetRHSFunction( tss, NULL, TSComputeRHSFunctionLinear, &c );

For now you can check out a branch that has implemented the fix by 
running git checkout emil/fix-ts-dae-rebased-v2 and then make everything.

The fix is at 
https://bitbucket.org/petsc/petsc/commits/557d2b190687a728e86290bdc604ce0d2dc53473

This is working for us.

We'll push the fix soon in master.

Emil

-- 
Emil M. Constantinescu, Ph.D.
Computational Mathematician
Argonne National Laboratory
Mathematics and Computer Science Division
9700 S Cass Avenue,
Argonne, IL 60439, USA

Ph: 630-252-0926
http://www.mcs.anl.gov/~emconsta

On 3/30/15 9:32 PM, Andrew Spott wrote:
> I?m sorry, I didn?t want to imply you were taking too much time.  I was
> just interested in trying to solve the problem myself, especially if it
> was a simpler fix.
>
> I don?t have enough of an understanding of the internals of PETSc
> though, if it depends on the compiler flags.
>
> Good luck!
>
> -Andrew
>
> ?
> Andrew
>
>
> On Mon, Mar 30, 2015 at 7:23 PM, Emil Constantinescu
> <emconsta at mcs.anl.gov <mailto:emconsta at mcs.anl.gov>> wrote:
>
>     Hi Andrew,
>
>     Sorry for the delay; things are going slower with me over the spring
>     break but we are working on it.
>
>     The problem that you see may not be related to the integrator, which
>     runs well (we get expected convergence rates and good results) for
>     other
>     problems, in particular, for problems with real variables (and no
>     c++11). So the high level th->endpoint implementation seems fine.
>
>     For some reason the linear solver is misbehaving when using cn. If you
>     use option -snes_mf the problem disappears.
>
>     Please do not use -ts_theta_adapt until it is fixed.
>
>     In conclusion I don't think that looking at theta.c will be productive.
>     We are currently looking at what happens at the lower levels.
>
>     Thanks,
>     Emil
>
>     On 3/30/15 4:25 PM, Andrew Spott wrote:
>      > Emil: If I want to fix this myself, do you have any idea where I
>     should
>      > start looking? I assume in the th->endpoint section of
>     TSStep_Theta in
>      >
>     http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?
>
>      >
>      > Thanks,
>      >
>      > -Andrew
>      >
>      >
>      >
>
>

From ansp6066 at colorado.edu  Tue Mar 31 20:21:46 2015
From: ansp6066 at colorado.edu (Andrew Spott)
Date: Tue, 31 Mar 2015 18:21:46 -0700 (PDT)
Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this
In-Reply-To: <551B4703.9040005@mcs.anl.gov>
References: <551B4703.9040005@mcs.anl.gov>
Message-ID: <1427851306414.f2c86496@Nodemailer>

awesome, great to hear.




Thanks!




-Andrew

On Tue, Mar 31, 2015 at 7:16 PM, Emil Constantinescu <emconsta at mcs.anl.gov>
wrote:

> Hi Andrew - we found a bug in TSComputeRHSFunctionLinear.
> In your code you need to provide the context to TSSetRHSFunction:
> TSSetRHSFunction( tss, NULL, TSComputeRHSFunctionLinear, &c );
> For now you can check out a branch that has implemented the fix by 
> running git checkout emil/fix-ts-dae-rebased-v2 and then make everything.
> The fix is at 
> https://bitbucket.org/petsc/petsc/commits/557d2b190687a728e86290bdc604ce0d2dc53473
> This is working for us.
> We'll push the fix soon in master.
> Emil
> -- 
> Emil M. Constantinescu, Ph.D.
> Computational Mathematician
> Argonne National Laboratory
> Mathematics and Computer Science Division
> 9700 S Cass Avenue,
> Argonne, IL 60439, USA
> Ph: 630-252-0926
> http://www.mcs.anl.gov/~emconsta
> On 3/30/15 9:32 PM, Andrew Spott wrote:
>> I?m sorry, I didn?t want to imply you were taking too much time.  I was
>> just interested in trying to solve the problem myself, especially if it
>> was a simpler fix.
>>
>> I don?t have enough of an understanding of the internals of PETSc
>> though, if it depends on the compiler flags.
>>
>> Good luck!
>>
>> -Andrew
>>
>> ?
>> Andrew
>>
>>
>> On Mon, Mar 30, 2015 at 7:23 PM, Emil Constantinescu
>> <emconsta at mcs.anl.gov <mailto:emconsta at mcs.anl.gov>> wrote:
>>
>>     Hi Andrew,
>>
>>     Sorry for the delay; things are going slower with me over the spring
>>     break but we are working on it.
>>
>>     The problem that you see may not be related to the integrator, which
>>     runs well (we get expected convergence rates and good results) for
>>     other
>>     problems, in particular, for problems with real variables (and no
>>     c++11). So the high level th->endpoint implementation seems fine.
>>
>>     For some reason the linear solver is misbehaving when using cn. If you
>>     use option -snes_mf the problem disappears.
>>
>>     Please do not use -ts_theta_adapt until it is fixed.
>>
>>     In conclusion I don't think that looking at theta.c will be productive.
>>     We are currently looking at what happens at the lower levels.
>>
>>     Thanks,
>>     Emil
>>
>>     On 3/30/15 4:25 PM, Andrew Spott wrote:
>>      > Emil: If I want to fix this myself, do you have any idea where I
>>     should
>>      > start looking? I assume in the th->endpoint section of
>>     TSStep_Theta in
>>      >
>>     http://www.mcs.anl.gov/petsc/petsc-current/src/ts/impls/implicit/theta/theta.c.html?
>>
>>      >
>>      > Thanks,
>>      >
>>      > -Andrew
>>      >
>>      >
>>      >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/7a36a61b/attachment.html>

From juris.vencels at gmail.com  Tue Mar 31 21:03:13 2015
From: juris.vencels at gmail.com (Juris Vencels)
Date: Tue, 31 Mar 2015 20:03:13 -0600
Subject: [petsc-users] Preconditioning Matrix-free SNES,
	preconditioner update, PCView
Message-ID: <551B51E1.8060901@gmail.com>

Hello PETSc Users,

I am using Matrix-Free SNES solver with the following options
    PETSC_OPTIONS="ksp_type gmres -snes_monitor_short -snes_mf 
-user_precond"

and shell-ILU preconditioner that I create as following:

! ===============================

       call PCSetType(pc,PCSHELL,ierr)
       call PCShellSetUp
       call PCShellSetApply(pc,PCShellApply,ierr)
       call PCShellSetName(pc,"ShellPC",ierr)

! ===============================

   subroutine PCShellSetUp()
     implicit none

      call SetInitialMatrix   ! Assambles matrix A

      call PCCreate(PETSC_COMM_WORLD,mf_prec,ierr)
      call PCSetType(mf_prec,PCILU,ierr)
      call PCSetOperators(mf_prec,A,A,ierr)
      call PCSetUp(mf_prec,ierr)
      call PCShellSetName(mf_prec,"LinPC",ierr)

   end subroutine PCShellSetUp

! ===============================

   subroutine PCShellApply(pc_tmp,x_tmp,y_tmp,ierr)
     implicit none
     PC  :: pc_tmp
     Vec :: x_tmp,y_tmp
     PetscErrorCode :: ierr

     call PCApply(mf_prec,x_tmp,y_tmp,ierr)

   end subroutine PCShellApply

! ===============================

* If matrix A changes during a simulation, how can I update 
preconditioner? Is calling "PCSetUp(mf_prec,ierr)" enough?

* Function PCView prints out information about type and size of 
preconditioner. Can I view/get preconditioner matrix itself? I want to 
compare it with reference matrix.

If I try to MatView factored matrix:
     call PCFactorGetMatrix(mf_prec,pcmat,ierr)
     call MatView(pcmat, PETSC_VIEWER_STDOUT_SELF,ierr)

Then I get the following error:
     [0]PETSC ERROR: No viewers for factored matrix except ASCII info or 
info_detailed

Thanks!



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/8d1b7256/attachment.html>

From bsmith at mcs.anl.gov  Tue Mar 31 21:26:49 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 31 Mar 2015 21:26:49 -0500
Subject: [petsc-users] Preconditioning Matrix-free SNES,
	preconditioner update, PCView
In-Reply-To: <551B51E1.8060901@gmail.com>
References: <551B51E1.8060901@gmail.com>
Message-ID: <0D2B2AB5-D789-49F6-9EF8-81399BB7A100@mcs.anl.gov>


> On Mar 31, 2015, at 9:03 PM, Juris Vencels <juris.vencels at gmail.com> wrote:
> 
> Hello PETSc Users,
> 
> I am using Matrix-Free SNES solver with the following options
>    PETSC_OPTIONS="ksp_type gmres -snes_monitor_short -snes_mf -user_precond"
> 
> and shell-ILU preconditioner that I create as following:
> 
> ! ===============================
> 
>       call PCSetType(pc,PCSHELL,ierr)
>       call PCShellSetUp
>       call PCShellSetApply(pc,PCShellApply,ierr)
>       call PCShellSetName(pc,"ShellPC",ierr)
> 
> ! ===============================
> 
>   subroutine PCShellSetUp() 
>     implicit none
> 
>      call SetInitialMatrix   ! Assambles matrix A
> 
>      call PCCreate(PETSC_COMM_WORLD,mf_prec,ierr)
>      call PCSetType(mf_prec,PCILU,ierr)
>      call PCSetOperators(mf_prec,A,A,ierr)
>      call PCSetUp(mf_prec,ierr)
>      call PCShellSetName(mf_prec,"LinPC",ierr)
> 
>   end subroutine PCShellSetUp
> 
> ! ===============================
> 
>   subroutine PCShellApply(pc_tmp,x_tmp,y_tmp,ierr)
>     implicit none
>     PC  :: pc_tmp
>     Vec :: x_tmp,y_tmp
>     PetscErrorCode :: ierr
> 
>     call PCApply(mf_prec,x_tmp,y_tmp,ierr)
> 
>   end subroutine PCShellApply
> 
> ! ===============================
> 
> * If matrix A changes during a simulation, how can I update preconditioner? Is calling "PCSetUp(mf_prec,ierr)" enough?

  No, the model is that each time the matrix changes the PCSetUp is called automatically and so is a shell set up if you have provided it, so you need to change your PCShellSetUp() function so that it can be called repeatedly for each new matrix and call 

> call PCShellSetApply(pc,PCShellSetUp,ierr)

when you create the Shell PC.

  Then make a test run in the debugger to make sure that your setup is being called each time the matrix changes.



> 
> * Function PCView prints out information about type and size of preconditioner. Can I view/get preconditioner matrix itself? I want to compare it with reference matrix.
> 
> If I try to MatView factored matrix:
>     call PCFactorGetMatrix(mf_prec,pcmat,ierr)
>     call MatView(pcmat, PETSC_VIEWER_STDOUT_SELF,ierr)
> 
> Then I get the following error:
>     [0]PETSC ERROR: No viewers for factored matrix except ASCII info or info_detailed

  The LU factors of sparse matrices are rather complicated beasts, there is no reasonable way to see their numerical values and related them to original matrix.

  A note about terminology:  We refer to the "preconditioner matrix" sometimes, it means "the matrix FROM WHICH a preconditioner is built." With ILU the "preconditioner matrix" is factored into L and U (which is what PCFactorGetMatrix() returns. The L and U are the resulting preconditioner, they are not the "matrix from which the preconditioner is built."

  Note: also if you are using ILU preconditioning, you do not need to use PCSHELL, you can just use PCSetType(pc,PCILU) or -pc_type ilu.



  Barry

> 
> Thanks!
> 
> 
> 


From knepley at gmail.com  Tue Mar 31 21:27:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 31 Mar 2015 21:27:09 -0500
Subject: [petsc-users] Preconditioning Matrix-free SNES,
 preconditioner update, PCView
In-Reply-To: <551B51E1.8060901@gmail.com>
References: <551B51E1.8060901@gmail.com>
Message-ID: <CAMYG4GkSYpOtKiJWWLyYyisJz0tEju5VXzhx0TPRKcVJ6OgW4A@mail.gmail.com>

On Tue, Mar 31, 2015 at 9:03 PM, Juris Vencels <juris.vencels at gmail.com>
wrote:

>  Hello PETSc Users,
>
> I am using Matrix-Free SNES solver with the following options
>    PETSC_OPTIONS="ksp_type gmres -snes_monitor_short -snes_mf
> -user_precond"
>
> and shell-ILU preconditioner that I create as following:
>

This is not the canonical PETSc way to do this. We would suggest using the
matrix A
as the preconditioner matrix which you would give back in FormJacobian().
Then all
the reforming would be handled automatically.

  Matt


> ! ===============================
>
>       call PCSetType(pc,PCSHELL,ierr)
>       call PCShellSetUp
>       call PCShellSetApply(pc,PCShellApply,ierr)
>       call PCShellSetName(pc,"ShellPC",ierr)
>
> ! ===============================
>
>   subroutine PCShellSetUp()
>     implicit none
>
>      call SetInitialMatrix   ! Assambles matrix A
>
>      call PCCreate(PETSC_COMM_WORLD,mf_prec,ierr)
>      call PCSetType(mf_prec,PCILU,ierr)
>      call PCSetOperators(mf_prec,A,A,ierr)
>      call PCSetUp(mf_prec,ierr)
>      call PCShellSetName(mf_prec,"LinPC",ierr)
>
>   end subroutine PCShellSetUp
>
> ! ===============================
>
>   subroutine PCShellApply(pc_tmp,x_tmp,y_tmp,ierr)
>     implicit none
>     PC  :: pc_tmp
>     Vec :: x_tmp,y_tmp
>     PetscErrorCode :: ierr
>
>     call PCApply(mf_prec,x_tmp,y_tmp,ierr)
>
>   end subroutine PCShellApply
>
> ! ===============================
>
> * If matrix A changes during a simulation, how can I update
> preconditioner? Is calling "PCSetUp(mf_prec,ierr)" enough?
>
> * Function PCView prints out information about type and size of
> preconditioner. Can I view/get preconditioner matrix itself? I want to
> compare it with reference matrix.
>
> If I try to MatView factored matrix:
>     call PCFactorGetMatrix(mf_prec,pcmat,ierr)
>     call MatView(pcmat, PETSC_VIEWER_STDOUT_SELF,ierr)
>
> Then I get the following error:
>     [0]PETSC ERROR: No viewers for factored matrix except ASCII info or
> info_detailed
>
> Thanks!
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150331/55c82d2b/attachment-0001.html>

From stm8086 at yahoo.com  Tue Mar 31 23:05:06 2015
From: stm8086 at yahoo.com (Steena M)
Date: Wed, 1 Apr 2015 04:05:06 +0000 (UTC)
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
 MatMult
In-Reply-To: <CAMYG4GmQ=t+HdPvc27+5WC6LNwLJQUnR-9fP86zqC28aNdwDqQ@mail.gmail.com>
References: <CAMYG4GmQ=t+HdPvc27+5WC6LNwLJQUnR-9fP86zqC28aNdwDqQ@mail.gmail.com>
Message-ID: <1358222418.2626161.1427861106397.JavaMail.yahoo@mail.yahoo.com>

Thanks Matt. I'm still getting the malloc error?
[0]PETSC ERROR: Argument out of range![0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
and?
a new incorrect matrix file error:
[0]PETSC ERROR: Unexpected data in file![0]PETSC ERROR: not matrix object!
Maybe the order of calls is mixed up. This is the code snippet:
 if (rank ==0) { ?PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);  ?? CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); CHKERRQ(MatSetType(A, MATMPIBAIJ)); CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); CHKERRQ( MatLoad(A,fd));? ?CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); }  else { PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);  CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); CHKERRQ(MatSetType(A, MATMPIBAIJ)); CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); CHKERRQ(MatLoad(A,fd)); CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE));  }
Is there something I'm missing??
Thanks,Steena  



     On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
   

 On Tue, Mar 31, 2015 at 6:51 PM, Steena M <stm8086 at yahoo.com> wrote:

Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step.


1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)?2) Note that this is never ever ever more efficient than making another pass and preallocating
? Thanks,
? ? ?Matt
--------------------------------------------
On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:

?Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
?To: "Steena M" <stm8086 at yahoo.com>
?Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
?Date: Sunday, March 29, 2015, 9:26 PM


?> On
?Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
?wrote:
?>
?> Thanks
?Matt. I used PETSC_DETERMINE but I'm now getting an
?allocation-based error:
?>
?> [0]PETSC ERROR: ---------------------
?Error Message ------------------------------------
?> [0]PETSC ERROR: Argument out of range!
?> [0]PETSC ERROR: New nonzero at (2,18)
?caused a malloc!
?> [0]PETSC ERROR:
?------------------------------------------------------------------------
?>
?> I tried
?preallocating on each rank for the diagonal and off diagonal
?section of the matrix as the next step? My current
?approximations for preallocation
?>
?> CHKERRQ(
?MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));?


????These
?arguments where you pass PETSC_DEFAULT are expecting a
?pointer not an integer. You can pass NULL in those
?locations. Though it is better to provide the correct
?preallocation rather than some defaults.

?? Barry

?>
?> are throwing segmentation errors.
?>
?> [0]PETSC ERROR:?
?Caught signal number 11 SEGV: Segmentation Violation,
?probably memory access out of range
?>
?> Any insights into what I'm doing
?wrong?
?>
?> Thanks,
?> Steena
?>
?>
?>
?> On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
?wrote:
?>
?> Subject:
?Re: [petsc-users] Unequal sparse matrix row distribution for
?MPI MatMult
?> To: "Steena M"
?<stm8086 at yahoo.com>
?> Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
?petsc-users at mcs.anl.gov
?> Date: Sunday, March 29, 2015, 10:02 PM
?>
?> On Sun, Mar 29, 2015
?at
?> 9:56 PM, Steena M <stm8086 at yahoo.com>
?> wrote:
?> Hi
?> Barry,
?>
?>
?>
?> I am trying to partition a 20 row and 20
?col sparse matrix
?> between two procs
?such that proc 0 has 15 rows and 20 cols
?> and proc 1 has 5 rows and 20 cols. The
?code snippet:
?>
?>
?>
?>
?>
?>
?>
?>? ? ?
????CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
?//
?> at runtime: -matload_block_size 1
?>
?>
?>
?>
?>
?>? ? ?
????if (rank ==0)
?>
?>? ? ? ???{
?>
?>? ? ? ? ? ? ?
????CHKERRQ( MatSetSizes(A, 15, 20, 20,
?> 20) ); //rank 0 gets 75% of the rows
?>
?>? ? ? ? ? ? ?
????CHKERRQ( MatSetType(A, MATMPIBAIJ)
?> );
?>
?>? ? ? ? ? ? ? ???CHKERRQ(
?MatLoad(A,fd) );
?>
?>? ? ? ? ? }
?>
?>
?>
?>? ? ? ???else
?>
?>? ? ?
????{
?>
?>?
?? ? ? ? ? ? ???CHKERRQ( MatSetSizes(A, 5,
?20, 20,
?> 20) ); //rank 1 gets 25% of the
?rows
?>
?>? ? ? ?
?? ? ? ???CHKERRQ( MatSetType(A, MATMPIBAIJ)
?> );
?>
?>? ? ? ? ? ? ? ???CHKERRQ(
?MatLoad(A,fd) );
?>
?>? ? ???}
?>

?>
?>
?> This throws the following error (probably
?from psplit.c):
?>
?>
?[1]PETSC ERROR: --------------------- Error Message
?> ------------------------------------
?>
?> [1]PETSC ERROR:
?Nonconforming object sizes!
?>
?> [1]PETSC ERROR: Sum of local lengths 40
?does not equal
?> global length 20, my
?local length 20
?>
?>???likely a call to
?VecSetSizes() or MatSetSizes() is
?>
?wrong.
?>
?> See
?http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
?>
?>
?>
?> This error printout
?doesn't quite make sense to me.
?>
?I'm trying to specify a total matrix size of 20x20...
?I
?> haven't yet figured out where the
?'40' comes
?> from in the error
?message.
?>
?>
?>
?> Any thoughts on what
?might be going wrong?
?>
?> Its the column specification. Just
?> use PETSC_DETERMINE for the local columns
?since all our
?> sparse matrixformats are
?row divisions
?> anyway.
?>?
?> Thanks,
?>? ?
?>? Matt
?> Thanks in advance,
?>

?> Steena
?>
?>
?>
?>
?>
?>
?>
?>
?--------------------------------------------
?>
?> On Sun, 3/22/15,
?Barry Smith <bsmith at mcs.anl.gov>
?> wrote:
?>
?>
?>
?>? Subject: Re: [petsc-users] Unequal
?sparse matrix row
?> distribution for MPI
?MatMult
?>
?>? To:
?"Steena M" <stm8086 at yahoo.com>
?>
?>? Cc: petsc-users at mcs.anl.gov
?>
?>? Date: Sunday,
?March 22, 2015, 3:58 PM
?>
?>
?>
?>
?>
?>???
?>
?>? Steena,
?>
?>
?>
?>? ???I am
?>

?>? a little unsure of your question.

?>
?>
?>
?>? ???1)
?you can create a MPIBAIJ
?>
?>? matrix with any distribution of block
?rows per process
?> you
?>
?>? want, just set the
?local row size for each process to
?>
?be
?>
?>? what you
?like.? Use MatCreateVecs() to get
?>
?correspondingly
?>
?>?
?laid out vectors.
?>
?>

?>
?>?
????or 2) if you have a MPIBAIJ
?>
?>? matrix with
?"equal" row layout and you want a
?> new
?>
?>? one with uneven row layout you can
?simply use
?>
?>?
?MatGetSubMatrix() to create that new matrix.
?>
?>
?>
?>? ? Barry
?>
?>
?>
?>? Unless you have
?another reason to have the
?>
?>? matrix with an equal number row layout I
?would just
?> generate
?>
?>? the matrix with
?the layout you want.
?>
?>
?>
?>
?>
?>? > On Mar 22, 2015, at 5:50 PM, Steena
?M
?>
?>? <stm8086 at yahoo.com>
?>
?>? wrote:
?>
?>? >
?>
?>? > Hello,
?>
?>? >
?>
?>? > I need to
?distribute
?>
?>? a
?sparse matrix such that each proc owns an unequal
?> number
?>
?>? of blocked rows before I proceed with
?MPI MatMult. My
?>
?>?
?initial thoughts on doing this:
?>
?>? >
?>
?>? > 1) Use? MatGetSubMatrices() on the
?test
?>
?>? MATMPIBAIJ
?matrix to produce a new matrix where each
?> proc
?>
?>? has an unequal number of rows.
?>
?>? >
?>
?>? > 2) Provide
?scatter context for vector X
?>
?>? (for MatMult )using IS iscol from
?MatGetSubMatrices()
?> while
?>
?>? creating the
?vector X.
?>
?>?
?>
?>
?>? > 3)
?Call MatMult()
?>
?>?
?>
?>
?>? > Will
?MatMult_MPIBAIJ continue to scatter
?>
?>? this matrix and vector such that each
?proc will own an
?> equal
?>
?>? number of matrix
?rows and corresponding diagonal vector
?>

?>? elements? Should I write my own
?MPIMatMult function to
?>
?>? retain my redistribution of the matrix
?and vector?
?>
?>?
?>
?>
?>? >
?Thanks in
?>
?>?
?advance,
?>
?>? >
?Steena
?>
?>
?>
?>
?>
?>
?> --
?> What most
?experimenters
?> take for granted before
?they begin their experiments is
?>
?infinitely more interesting than any results to which
?their
?> experiments lead.
?> -- Norbert
?> Wiener
?>





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150401/90d2c473/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Mar 31 23:15:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 31 Mar 2015 23:15:00 -0500
Subject: [petsc-users] Unequal sparse matrix row distribution for MPI
	MatMult
In-Reply-To: <1358222418.2626161.1427861106397.JavaMail.yahoo@mail.yahoo.com>
References: <CAMYG4GmQ=t+HdPvc27+5WC6LNwLJQUnR-9fP86zqC28aNdwDqQ@mail.gmail.com>
	<1358222418.2626161.1427861106397.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <656F0F36-03B5-43C2-9FDC-82F1B1337E42@mcs.anl.gov>


  You should not need to call any preallocation routines when using MatLoad()


   How did you generate the file? Are you sure it has the correct information for the matrix? 

  Barry



> On Mar 31, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com> wrote:
> 
> Thanks Matt. I'm still getting the malloc error 
> 
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
> 
> and 
> 
> a new incorrect matrix file error:
> 
> [0]PETSC ERROR: Unexpected data in file!
> [0]PETSC ERROR: not matrix object!
> 
> Maybe the order of calls is mixed up. This is the code snippet:
> 
> 	if (rank ==0)
> 	{
> 	 PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);
> 	   
> 		CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20));
> 		CHKERRQ(MatSetType(A, MATMPIBAIJ));
> 		CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL));
> 		CHKERRQ( MatLoad(A,fd)); 
> 	 CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE));
> 	}
> 	
> 	else
> 	{
> 		PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);
> 		
> 		CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) );
> 		CHKERRQ(MatSetType(A, MATMPIBAIJ));
> 		CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL));
> 		CHKERRQ(MatLoad(A,fd));
> 	CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE));
> 		}
> 
> Is there something I'm missing? 
> 
> Thanks,
> Steena
> 	
> 
> 
> 
> 
> On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> 
> On Tue, Mar 31, 2015 at 6:51 PM, Steena M <stm8086 at yahoo.com> wrote:
> Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step.
> 
> 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>  
> 2) Note that this is never ever ever more efficient than making another pass and preallocating
> 
>   Thanks,
> 
>      Matt
> --------------------------------------------
> On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>  Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
>  To: "Steena M" <stm8086 at yahoo.com>
>  Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
>  Date: Sunday, March 29, 2015, 9:26 PM
> 
> 
>  > On
>  Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
>  wrote:
>  >
>  > Thanks
>  Matt. I used PETSC_DETERMINE but I'm now getting an
>  allocation-based error:
>  >
>  > [0]PETSC ERROR: ---------------------
>  Error Message ------------------------------------
>  > [0]PETSC ERROR: Argument out of range!
>  > [0]PETSC ERROR: New nonzero at (2,18)
>  caused a malloc!
>  > [0]PETSC ERROR:
>  ------------------------------------------------------------------------
>  >
>  > I tried
>  preallocating on each rank for the diagonal and off diagonal
>  section of the matrix as the next step  My current
>  approximations for preallocation
>  >
>  > CHKERRQ(
>  MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); 
> 
> 
>     These
>  arguments where you pass PETSC_DEFAULT are expecting a
>  pointer not an integer. You can pass NULL in those
>  locations. Though it is better to provide the correct
>  preallocation rather than some defaults.
> 
>    Barry
> 
>  >
>  > are throwing segmentation errors.
>  >
>  > [0]PETSC ERROR: 
>  Caught signal number 11 SEGV: Segmentation Violation,
>  probably memory access out of range
>  >
>  > Any insights into what I'm doing
>  wrong?
>  >
>  > Thanks,
>  > Steena
>  >
>  >
>  >
>  > On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
>  wrote:
>  >
>  > Subject:
>  Re: [petsc-users] Unequal sparse matrix row distribution for
>  MPI MatMult
>  > To: "Steena M"
>  <stm8086 at yahoo.com>
>  > Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
>  petsc-users at mcs.anl.gov
>  > Date: Sunday, March 29, 2015, 10:02 PM
>  >
>  > On Sun, Mar 29, 2015
>  at
>  > 9:56 PM, Steena M <stm8086 at yahoo.com>
>  > wrote:
>  > Hi
>  > Barry,
>  >
>  >
>  >
>  > I am trying to partition a 20 row and 20
>  col sparse matrix
>  > between two procs
>  such that proc 0 has 15 rows and 20 cols
>  > and proc 1 has 5 rows and 20 cols. The
>  code snippet:
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  >     
>     CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
>  //
>  > at runtime: -matload_block_size 1
>  >
>  >
>  >
>  >
>  >
>  >     
>     if (rank ==0)
>  >
>  >         {
>  >
>  >             
>     CHKERRQ( MatSetSizes(A, 15, 20, 20,
>  > 20) ); //rank 0 gets 75% of the rows
>  >
>  >             
>     CHKERRQ( MatSetType(A, MATMPIBAIJ)
>  > );
>  >
>  >                 CHKERRQ(
>  MatLoad(A,fd) );
>  >
>  >          }
>  >
>  >
>  >
>  >         else
>  >
>  >     
>     {
>  >
>  > 
>                 CHKERRQ( MatSetSizes(A, 5,
>  20, 20,
>  > 20) ); //rank 1 gets 25% of the
>  rows
>  >
>  >       
>           CHKERRQ( MatSetType(A, MATMPIBAIJ)
>  > );
>  >
>  >                 CHKERRQ(
>  MatLoad(A,fd) );
>  >
>  >       }
>  >
> 
>  >
>  >
>  > This throws the following error (probably
>  from psplit.c):
>  >
>  >
>  [1]PETSC ERROR: --------------------- Error Message
>  > ------------------------------------
>  >
>  > [1]PETSC ERROR:
>  Nonconforming object sizes!
>  >
>  > [1]PETSC ERROR: Sum of local lengths 40
>  does not equal
>  > global length 20, my
>  local length 20
>  >
>  >   likely a call to
>  VecSetSizes() or MatSetSizes() is
>  >
>  wrong.
>  >
>  > See
>  http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
>  >
>  >
>  >
>  > This error printout
>  doesn't quite make sense to me.
>  >
>  I'm trying to specify a total matrix size of 20x20...
>  I
>  > haven't yet figured out where the
>  '40' comes
>  > from in the error
>  message.
>  >
>  >
>  >
>  > Any thoughts on what
>  might be going wrong?
>  >
>  > Its the column specification. Just
>  > use PETSC_DETERMINE for the local columns
>  since all our
>  > sparse matrixformats are
>  row divisions
>  > anyway.
>  > 
>  > Thanks,
>  >   
>  >  Matt
>  > Thanks in advance,
>  >
> 
>  > Steena
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  >
>  --------------------------------------------
>  >
>  > On Sun, 3/22/15,
>  Barry Smith <bsmith at mcs.anl.gov>
>  > wrote:
>  >
>  >
>  >
>  >  Subject: Re: [petsc-users] Unequal
>  sparse matrix row
>  > distribution for MPI
>  MatMult
>  >
>  >  To:
>  "Steena M" <stm8086 at yahoo.com>
>  >
>  >  Cc: petsc-users at mcs.anl.gov
>  >
>  >  Date: Sunday,
>  March 22, 2015, 3:58 PM
>  >
>  >
>  >
>  >
>  >
>  >   
>  >
>  >  Steena,
>  >
>  >
>  >
>  >     I am
>  >
> 
>  >  a little unsure of your question.
> 
>  >
>  >
>  >
>  >     1)
>  you can create a MPIBAIJ
>  >
>  >  matrix with any distribution of block
>  rows per process
>  > you
>  >
>  >  want, just set the
>  local row size for each process to
>  >
>  be
>  >
>  >  what you
>  like.  Use MatCreateVecs() to get
>  >
>  correspondingly
>  >
>  > 
>  laid out vectors.
>  >
>  >
> 
>  >
>  > 
>     or 2) if you have a MPIBAIJ
>  >
>  >  matrix with
>  "equal" row layout and you want a
>  > new
>  >
>  >  one with uneven row layout you can
>  simply use
>  >
>  > 
>  MatGetSubMatrix() to create that new matrix.
>  >
>  >
>  >
>  >    Barry
>  >
>  >
>  >
>  >  Unless you have
>  another reason to have the
>  >
>  >  matrix with an equal number row layout I
>  would just
>  > generate
>  >
>  >  the matrix with
>  the layout you want.
>  >
>  >
>  >
>  >
>  >
>  >  > On Mar 22, 2015, at 5:50 PM, Steena
>  M
>  >
>  >  <stm8086 at yahoo.com>
>  >
>  >  wrote:
>  >
>  >  >
>  >
>  >  > Hello,
>  >
>  >  >
>  >
>  >  > I need to
>  distribute
>  >
>  >  a
>  sparse matrix such that each proc owns an unequal
>  > number
>  >
>  >  of blocked rows before I proceed with
>  MPI MatMult. My
>  >
>  > 
>  initial thoughts on doing this:
>  >
>  >  >
>  >
>  >  > 1) Use  MatGetSubMatrices() on the
>  test
>  >
>  >  MATMPIBAIJ
>  matrix to produce a new matrix where each
>  > proc
>  >
>  >  has an unequal number of rows.
>  >
>  >  >
>  >
>  >  > 2) Provide
>  scatter context for vector X
>  >
>  >  (for MatMult )using IS iscol from
>  MatGetSubMatrices()
>  > while
>  >
>  >  creating the
>  vector X.
>  >
>  > 
>  >
>  >
>  >  > 3)
>  Call MatMult()
>  >
>  > 
>  >
>  >
>  >  > Will
>  MatMult_MPIBAIJ continue to scatter
>  >
>  >  this matrix and vector such that each
>  proc will own an
>  > equal
>  >
>  >  number of matrix
>  rows and corresponding diagonal vector
>  >
> 
>  >  elements? Should I write my own
>  MPIMatMult function to
>  >
>  >  retain my redistribution of the matrix
>  and vector?
>  >
>  > 
>  >
>  >
>  >  >
>  Thanks in
>  >
>  > 
>  advance,
>  >
>  >  >
>  Steena
>  >
>  >
>  >
>  >
>  >
>  >
>  > --
>  > What most
>  experimenters
>  > take for granted before
>  they begin their experiments is
>  >
>  infinitely more interesting than any results to which
>  their
>  > experiments lead.
>  > -- Norbert
>  > Wiener
> 
>  >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
>