From jiangwen84 at gmail.com  Sun Apr  1 14:12:43 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Sun, 1 Apr 2012 15:12:43 -0400
Subject: [petsc-users] code hangs inside SUPERLU DIST
Message-ID: <CAMJxm+DQwQMSFfk8129K01L4bECUT=qjPCFPJw4iNAy6MWBd3w@mail.gmail.com>

Hello Everyone,

I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my code
hangs inside superlu_dist solver and never finishes.

The problem I am solving is a time-dependent nonlinear pde and I use fully
Newton-Raphson for each time step. So my code will look like,

For Time STEP = start to end

    For Newton-Raphson Iterations = 1 to max iterations

        Calculate Matrix A;



    END Newton-Raphson Iterations

End Time STEP


Could you give me any hints what the problem might be? I am using Petsc 3.1
p8 and superlu dist v2.4.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120401/daf74ebd/attachment.htm>

From jiangwen84 at gmail.com  Sun Apr  1 14:18:32 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Sun, 1 Apr 2012 15:18:32 -0400
Subject: [petsc-users] code hangs inside SUPERLU DIST
In-Reply-To: <CAMJxm+DQwQMSFfk8129K01L4bECUT=qjPCFPJw4iNAy6MWBd3w@mail.gmail.com>
References: <CAMJxm+DQwQMSFfk8129K01L4bECUT=qjPCFPJw4iNAy6MWBd3w@mail.gmail.com>
Message-ID: <CAMJxm+AvVjvV1UZABMqjUgLzRwGHUO-edT67D_MspZkR987LxA@mail.gmail.com>

Sorry for the last incomplete email. Please forget about it. Let me finish
it up here.

Hello Everyone,

I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my code
hangs inside superlu_dist solver and never finishes.

The problem I am solving is a time-dependent nonlinear pde and I use fully
Newton-Raphson for each time step. So my code will look like,

For Time STEP = From start to end

    For Newton-Raphson Iterations = From 1 to max iterations

        Calculate Matrix A; (A is different for each Newton-Raphson
iteration)

        Create KSP
        KSPSetOperator(A,A,same_nonzero_pattern)
        KSPSolve;

       KSPDestroy;

    END Newton-Raphson Iterations

End Time STEP

Could you give me any hints what the problem might be? I am using Petsc 3.1
p8 and superlu dist v2.4. And my code works fine with KSP iterative solver
like GMRES.

Regards,
Wen

On Sun, Apr 1, 2012 at 3:12 PM, Wen Jiang <jiangwen84 at gmail.com> wrote:

> Hello Everyone,
>
> I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my
> code hangs inside superlu_dist solver and never finishes.
>
> The problem I am solving is a time-dependent nonlinear pde and I use fully
> Newton-Raphson for each time step. So my code will look like,
>
> For Time STEP = start to end
>
>     For Newton-Raphson Iterations = 1 to max iterations
>
>         Calculate Matrix A;
>
>
>
>     END Newton-Raphson Iterations
>
> End Time STEP
>
>
> Could you give me any hints what the problem might be? I am using Petsc
> 3.1 p8 and superlu dist v2.4.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120401/27e715d2/attachment.htm>

From knepley at gmail.com  Sun Apr  1 15:01:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 1 Apr 2012 15:01:34 -0500
Subject: [petsc-users] code hangs inside SUPERLU DIST
In-Reply-To: <CAMJxm+AvVjvV1UZABMqjUgLzRwGHUO-edT67D_MspZkR987LxA@mail.gmail.com>
References: <CAMJxm+DQwQMSFfk8129K01L4bECUT=qjPCFPJw4iNAy6MWBd3w@mail.gmail.com>
	<CAMJxm+AvVjvV1UZABMqjUgLzRwGHUO-edT67D_MspZkR987LxA@mail.gmail.com>
Message-ID: <CAMYG4Gms=SLMawHern7DnB3kYpqcKEZfSE90DuA=Dn_jobUULA@mail.gmail.com>

On Sun, Apr 1, 2012 at 2:18 PM, Wen Jiang <jiangwen84 at gmail.com> wrote:

> Sorry for the last incomplete email. Please forget about it. Let me finish
> it up here.
>
>
> Hello Everyone,
>
> I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my
> code hangs inside superlu_dist solver and never finishes.
>
> The problem I am solving is a time-dependent nonlinear pde and I use fully
> Newton-Raphson for each time step. So my code will look like,
>
> For Time STEP = From start to end
>
>     For Newton-Raphson Iterations = From 1 to max iterations
>
>         Calculate Matrix A; (A is different for each Newton-Raphson
> iteration)
>
>         Create KSP
>         KSPSetOperator(A,A,same_nonzero_pattern)
>         KSPSolve;
>
>        KSPDestroy;
>
>
>     END Newton-Raphson Iterations
>
> End Time STEP
>
> Could you give me any hints what the problem might be? I am using Petsc
> 3.1 p8 and superlu dist v2.4. And my code works fine with KSP iterative
> solver like GMRES.
>

There are a list of things I would try:

1) Upgrading to the latest release

2) Trying MUMPS instead of SuperLU_dist

3) Get a stack trace from gdb to determine where it is hanging.

I would do whichever one is easiest.

    Matt


> Regards,
> Wen
>
>
> On Sun, Apr 1, 2012 at 3:12 PM, Wen Jiang <jiangwen84 at gmail.com> wrote:
>
>> Hello Everyone,
>>
>> I am using KSPPRECONLY type ksp with SUPERLU DIST. However, I found my
>> code hangs inside superlu_dist solver and never finishes.
>>
>> The problem I am solving is a time-dependent nonlinear pde and I use
>> fully Newton-Raphson for each time step. So my code will look like,
>>
>> For Time STEP = start to end
>>
>>     For Newton-Raphson Iterations = 1 to max iterations
>>
>>         Calculate Matrix A;
>>
>>
>>
>>     END Newton-Raphson Iterations
>>
>> End Time STEP
>>
>>
>> Could you give me any hints what the problem might be? I am using Petsc
>> 3.1 p8 and superlu dist v2.4.
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120401/a1e5611e/attachment.htm>

From khalid_eee at yahoo.com  Mon Apr  2 00:01:55 2012
From: khalid_eee at yahoo.com (khalid ashraf)
Date: Sun, 1 Apr 2012 22:01:55 -0700 (PDT)
Subject: [petsc-users] transfer vector data diagonally on DA
Message-ID: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>

Hi,?
I want to transfer vector data diagonally in the DA grid like?
?for (k=zs; k<zs+zm; k++) {
??for (j=ys; j<ys+ym; j++) {
??for (i=xs; i<xs+xm; i++) {
if(i!=mx-1 || j!=my-1 || k!=mz-1){
u_new[k+1][j+1][i+1]=u[k][j][i];}
}}}

Could you please suggest the best way to do it minimizing interprocessor assignments.

Thanks in advance.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120401/7289c2b9/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  2 00:07:09 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 1 Apr 2012 22:07:09 -0700
Subject: [petsc-users] transfer vector data diagonally on DA
In-Reply-To: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
Message-ID: <CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>

On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:

> I want to transfer vector data diagonally in the DA grid like
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
> if(i!=mx-1 || j!=my-1 || k!=mz-1){
> u_new[k+1][j+1][i+1]=u[k][j][i];}
> }}}
>
> Could you please suggest the best way to do it minimizing interprocessor
> assignments.
>

Both are on the same DMDA?

Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with
width at least 1, get the global array u_new[][][] from UGlobalNew and the
local arrays u[][][] from Ulocal, then assign u_new[k][j][i] =
u[k-1][j-1][i-1].
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120401/8a1be8b5/attachment.htm>

From khalid_eee at yahoo.com  Mon Apr  2 03:36:26 2012
From: khalid_eee at yahoo.com (khalid ashraf)
Date: Mon, 2 Apr 2012 01:36:26 -0700 (PDT)
Subject: [petsc-users] transfer vector data diagonally on DA
In-Reply-To: <CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
Message-ID: <1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>

Hi Jed,

I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h"
and DACreate3D.?
unew_localptr[][][] is from a global vector using VecGetArray
u_localptr[][][] is from a local vector communicated from a global vector.
I tried?
unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]

but it gives Segmentation Violation error.?
I also tried?unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but?
unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or?unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
does not work. I am running the program on4 processors.


Thanks.


??ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
?? ? ? ? ? ? ? ? ? ?PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
?? ? ? ? ? ? ? ? ? ?&appctx.da);CHKERRQ(ierr);
?for (k=zs; k<zs+zm; k++) {
??for (j=ys; j<ys+ym; j++) {
??for (i=xs; i<xs+xm; i++) {

??if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)

??unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];

}}}

________________________________
 From: Jed Brown <jedbrown at mcs.anl.gov>
To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
Sent: Sunday, April 1, 2012 10:07 PM
Subject: Re: [petsc-users] transfer vector data diagonally on DA
 

On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:

I want to transfer vector data diagonally in the DA grid like?
>?for (k=zs; k<zs+zm; k++) {
>??for (j=ys; j<ys+ym; j++) {
>??for (i=xs; i<xs+xm; i++) {
>if(i!=mx-1 || j!=my-1 || k!=mz-1){
>u_new[k+1][j+1][i+1]=u[k][j][i];}
>}}}
>
>
>Could you please suggest the best way to do it minimizing interprocessor assignments.

Both are on the same DMDA?

Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/b9318138/attachment-0001.htm>

From bibrakc at gmail.com  Mon Apr  2 07:37:52 2012
From: bibrakc at gmail.com (Bibrak Qamar)
Date: Mon, 2 Apr 2012 16:37:52 +0400
Subject: [petsc-users] Scalability of PETSc
Message-ID: <CANFHntDoR07VaQzsDTVxrQEU7=sAjqBEHcbwuP86Mo2VhgjgPw@mail.gmail.com>

Hello all,

I am using PETSc to solve linear equation using CG. I am not getting good
scalability (using the MPI implementation, not GPUs). One reason can be
that may be for my platform I haven't tuned the library. Other reason may
be the nature of CG, that it is communication bound, at each iteration the
vector (search direction) has to be communicated.

Where can I find statistics of the scalability of CG PETSc (if there is
any, people may have reported)?

Or can anyone give a number on how well CG PETSc scales on their
environment. Lets take the simple example provided in the PETSc KSP
tutorials - KSP Laplacian, 2d (ex2.c) for mesh dim of
n = 150000
m = 100


Thanks
Bibrak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/6cbea21c/attachment.htm>

From knepley at gmail.com  Mon Apr  2 07:41:08 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 07:41:08 -0500
Subject: [petsc-users] transfer vector data diagonally on DA
In-Reply-To: <1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
Message-ID: <CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>

On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com> wrote:

> Hi Jed,
>
> I am using petsc/3.1 and the include file "petscdmda.h" is not working. I
> am using the "petscda.h"
> and DACreate3D.
> unew_localptr[][][] is from a global vector using VecGetArray
> u_localptr[][][] is from a local vector communicated from a global vector.
> I tried
> unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
> but it gives Segmentation Violation error.
>

I would guess (because you provide almost no information about what you are
doing), that
this is a domain on the edge. If you truly mean to have a periodic domain,
you must set that in the
creation call. Then the local vector will also ghost regions outside the
domain boundary.

   Matt


> I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works
> but
> unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1]
> or unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
> does not work. I am running the program on4 processors.
>
> Thanks.
>
>
>   ierr =
> DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
>
>  PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>                     &appctx.da);CHKERRQ(ierr);
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>
>   if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
>
>   unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
> }}}
>   ------------------------------
> *From:* Jed Brown <jedbrown at mcs.anl.gov>
> *To:* khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Sunday, April 1, 2012 10:07 PM
> *Subject:* Re: [petsc-users] transfer vector data diagonally on DA
>
> On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:
>
> I want to transfer vector data diagonally in the DA grid like
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
> if(i!=mx-1 || j!=my-1 || k!=mz-1){
> u_new[k+1][j+1][i+1]=u[k][j][i];}
> }}}
>
> Could you please suggest the best way to do it minimizing interprocessor
> assignments.
>
>
> Both are on the same DMDA?
>
> Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil
> with width at least 1, get the global array u_new[][][] from UGlobalNew and
> the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] =
> u[k-1][j-1][i-1].
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/485ddc8d/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr  2 07:42:14 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 2 Apr 2012 07:42:14 -0500
Subject: [petsc-users] Scalability of PETSc
In-Reply-To: <CANFHntDoR07VaQzsDTVxrQEU7=sAjqBEHcbwuP86Mo2VhgjgPw@mail.gmail.com>
References: <CANFHntDoR07VaQzsDTVxrQEU7=sAjqBEHcbwuP86Mo2VhgjgPw@mail.gmail.com>
Message-ID: <E8AB0001-65C6-4452-AD6D-616490F48C06@mcs.anl.gov>


   Bibrak,

   Start with http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

    Then http://www.mcs.anl.gov/petsc/documentation/faq.html#log-summary

     and http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel

    Barry

On Apr 2, 2012, at 7:37 AM, Bibrak Qamar wrote:

> Hello all,
> 
> I am using PETSc to solve linear equation using CG. I am not getting good scalability (using the MPI implementation, not GPUs). One reason can be that may be for my platform I haven't tuned the library. Other reason may be the nature of CG, that it is communication bound, at each iteration the vector (search direction) has to be communicated. 
> 
> Where can I find statistics of the scalability of CG PETSc (if there is any, people may have reported)?
> 
> Or can anyone give a number on how well CG PETSc scales on their environment. Lets take the simple example provided in the PETSc KSP tutorials - KSP Laplacian, 2d (ex2.c) for mesh dim of 
> n = 150000
> m = 100
> 
> 
> Thanks
> Bibrak 


From niko.karin at gmail.com  Mon Apr  2 08:59:33 2012
From: niko.karin at gmail.com (Karin&NiKo)
Date: Mon, 2 Apr 2012 15:59:33 +0200
Subject: [petsc-users] ML and linear elasticity
Message-ID: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>

Dear Petsc team,

I would like to use ML as a preconditioner for a linear elasticity problem.
I wonder if there is a way to indicate ML that it is dealing with a vector
problem.
I know that when using Prometheus, one must use a block matrix, and I would
like to know if there is a trick
in order to use ML in an optimal manner for vector problems.

Regards,
Nicolas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/9bee5c8a/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  2 09:44:18 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 2 Apr 2012 07:44:18 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
Message-ID: <CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>

On Mon, Apr 2, 2012 at 06:59, Karin&NiKo <niko.karin at gmail.com> wrote:

> I would like to use ML as a preconditioner for a linear elasticity problem.
> I wonder if there is a way to indicate ML that it is dealing with a vector
> problem.
> I know that when using Prometheus, one must use a block matrix, and I
> would like to know if there is a trick
> in order to use ML in an optimal manner for vector problems.
>

The block size is used if you don't provide any other information. The
preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll
make sure this is currently working with ML and reply to this message.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/bc15d0bf/attachment.htm>

From mark.adams at columbia.edu  Mon Apr  2 10:09:36 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Mon, 2 Apr 2012 11:09:36 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
Message-ID: <27A4BE6D-B6B2-4724-AB19-A3958892DA3A@columbia.edu>

FYI, you can try GAMG also.  It uses the same algorithm as ML:

Here are parameters:

-pc_type gamg 
-pc_gamg_type agg 
-pc_gamg_agg_nsmooths 1     ! this is good for elliptic problems
-pc_gamg_verbose 2                ! this will help me to give a quick sanity check
-pc_gamg_threshold .05           ! for 3D problems this parameter can be useful for optimization
-pc_gamg_coarse_eq_limit 50  ! a detail but you can keep in your input deck

The vector Lapacian has 6 null space vectors (3 translational, 3 rotational).  GAMG will construct these with the a setCoordinates method (see the ksp tutorial example ex56.c).   If you prefer to give me the null space vectors explicitly there is the MatSetNearNullSpace() method that Jed mentioned.  GAMG has not implemented this but I should at some point so if you want to use it I can implement it.

If you do not give it coordinates it will construct the 3 translational null space vectors (don't need anything to compute these) and it can be OK.  At least a good place to start.

Mark

On Apr 2, 2012, at 10:44 AM, Jed Brown wrote:

> On Mon, Apr 2, 2012 at 06:59, Karin&NiKo <niko.karin at gmail.com> wrote:
> I would like to use ML as a preconditioner for a linear elasticity problem.
> I wonder if there is a way to indicate ML that it is dealing with a vector problem. 
> I know that when using Prometheus, one must use a block matrix, and I would like to know if there is a trick 
> in order to use ML in an optimal manner for vector problems.
> 
> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/ff42ce83/attachment.htm>

From li at loyno.edu  Mon Apr  2 10:11:32 2012
From: li at loyno.edu (Xuefeng Li)
Date: Mon, 2 Apr 2012 10:11:32 -0500 (CDT)
Subject: [petsc-users] dmmg_grid_sequence
In-Reply-To: <C6361099-5817-4D37-88BC-28FD0337CEB3@mcs.anl.gov>
References: <58356696.1460341259111616362.JavaMail.root@mail8.gatech.edu>
	<DABA8EB2-07CA-497A-BB8D-8F4C6273D886@mcs.anl.gov>
	<Pine.A41.4.61.1201061608490.7373010@loyno.edu>
	<CAM9tzSk0YW5+Xzxg2UQSDaXEnzLUOxCbXn9_EZcrnuC3HA08+A@mail.gmail.com>
	<Pine.A41.4.61.1201061629510.3997766@loyno.edu>
	<CAMYG4GkUnT+rbudt5xmSyT0S-vqgC80zSdjP9VOjRCbxBYNMSA@mail.gmail.com>
	<Pine.A41.4.61.1201061639020.9515068@loyno.edu>
	<C6361099-5817-4D37-88BC-28FD0337CEB3@mcs.anl.gov>
Message-ID: <Pine.A41.4.61.1204020958410.7057648@loyno.edu>

Hi, all.

I am using multigrid with DMMG (5 levels) in my petsc program.
Turning on the dmmg_grid_sequence option seems
to slow down the program dramatically. For instance,
the program converges in 3 iterations in 6.270e+01 sec
when dmmg_grid_sequence is OFF; it also converges
in 3 iterations but in 1.699e+04 sec when dmmg_grid_sequence
is ON.

Is this typical of petsc (I'm using petsc-3.1-p8)?
Are there ways to improve the performance when
dmmg_grid_sequence is ON?

Regards,

--Xuefeng Li, (504)865-3340(phone)
   Like floating clouds, the heart rests easy
   Like flowing water, the spirit stays free
        http://www.loyno.edu/~li/home
   New Orleans, Louisiana (504)865-2051(fax)


From knepley at gmail.com  Mon Apr  2 10:21:52 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 10:21:52 -0500
Subject: [petsc-users] dmmg_grid_sequence
In-Reply-To: <Pine.A41.4.61.1204020958410.7057648@loyno.edu>
References: <58356696.1460341259111616362.JavaMail.root@mail8.gatech.edu>
	<DABA8EB2-07CA-497A-BB8D-8F4C6273D886@mcs.anl.gov>
	<Pine.A41.4.61.1201061608490.7373010@loyno.edu>
	<CAM9tzSk0YW5+Xzxg2UQSDaXEnzLUOxCbXn9_EZcrnuC3HA08+A@mail.gmail.com>
	<Pine.A41.4.61.1201061629510.3997766@loyno.edu>
	<CAMYG4GkUnT+rbudt5xmSyT0S-vqgC80zSdjP9VOjRCbxBYNMSA@mail.gmail.com>
	<Pine.A41.4.61.1201061639020.9515068@loyno.edu>
	<C6361099-5817-4D37-88BC-28FD0337CEB3@mcs.anl.gov>
	<Pine.A41.4.61.1204020958410.7057648@loyno.edu>
Message-ID: <CAMYG4GmBXuHHez4biRh7Y9RpG7Ju0sU1XAwfHKjF1U-oM+NF1Q@mail.gmail.com>

On Mon, Apr 2, 2012 at 10:11 AM, Xuefeng Li <li at loyno.edu> wrote:

> Hi, all.
>
> I am using multigrid with DMMG (5 levels) in my petsc program.
> Turning on the dmmg_grid_sequence option seems
> to slow down the program dramatically. For instance,
> the program converges in 3 iterations in 6.270e+01 sec
> when dmmg_grid_sequence is OFF; it also converges
> in 3 iterations but in 1.699e+04 sec when dmmg_grid_sequence
> is ON.
>
> Is this typical of petsc (I'm using petsc-3.1-p8)?
> Are there ways to improve the performance when
> dmmg_grid_sequence is ON?
>

Its impossible to know what is happening from this description. There is no
"generic" performance of grid sequencing. The best thing I can say to this
is,
it looks like the interpolated solution to your coarse problem is not a
great
solution to your fine problem. However, you could be oversolving the coarse
problems, blowing up the work, or interpolating badly for your problem.

    Matt


> Regards,
>
> --Xuefeng Li, (504)865-3340(phone)
>  Like floating clouds, the heart rests easy
>  Like flowing water, the spirit stays free
>       http://www.loyno.edu/~li/home
>  New Orleans, Louisiana (504)865-2051(fax)
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/555a58f0/attachment.htm>

From niko.karin at gmail.com  Mon Apr  2 10:37:26 2012
From: niko.karin at gmail.com (Karin&NiKo)
Date: Mon, 2 Apr 2012 17:37:26 +0200
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <27A4BE6D-B6B2-4724-AB19-A3958892DA3A@columbia.edu>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<27A4BE6D-B6B2-4724-AB19-A3958892DA3A@columbia.edu>
Message-ID: <CA+gX-L8T0rUAMcQGSveMBQt6PU=BFtmLNw0ptsq_q=zNx6L+Zg@mail.gmail.com>

Dear Jed and Mark.

Thank you for your quick answers.
1. I am stuck to petsc-3.2 since I am using PETSc in a production software.
For the next release, I'll give GAMG a try (I have been a satisfied user of
Prometheus).
2. It seems to me that ML does not support block matrices, doesn't it?
3. I would like to give ML the information about the vector nature of my
problem in order that all 3 dofs of a given node be treated in the same
manner (in the coarsening phase for instance).

Nicolas

2012/4/2 Mark F. Adams <mark.adams at columbia.edu>

> FYI, you can try GAMG also.  It uses the same algorithm as ML:
>
> Here are parameters:
>
> -pc_type gamg
> -pc_gamg_type agg
> -pc_gamg_agg_nsmooths 1     ! this is good for elliptic problems
> -pc_gamg_verbose 2                ! this will help me to give a quick
> sanity check
> -pc_gamg_threshold .05           ! for 3D problems this parameter can be
> useful for optimization
> -pc_gamg_coarse_eq_limit 50  ! a detail but you can keep in your input deck
>
> The vector Lapacian has 6 null space vectors (3 translational, 3
> rotational).  GAMG will construct these with the a setCoordinates method
> (see the ksp tutorial example ex56.c).   If you prefer to give me the null
> space vectors explicitly there is the MatSetNearNullSpace() method that Jed
> mentioned.  GAMG has not implemented this but I should at some point so if
> you want to use it I can implement it.
>
> If you do not give it coordinates it will construct the 3 translational
> null space vectors (don't need anything to compute these) and it can be OK.
>  At least a good place to start.
>
> Mark
>
> On Apr 2, 2012, at 10:44 AM, Jed Brown wrote:
>
> On Mon, Apr 2, 2012 at 06:59, Karin&NiKo <niko.karin at gmail.com> wrote:
>
>> I would like to use ML as a preconditioner for a linear elasticity
>> problem.
>> I wonder if there is a way to indicate ML that it is dealing with a
>> vector problem.
>> I know that when using Prometheus, one must use a block matrix, and I
>> would like to know if there is a trick
>> in order to use ML in an optimal manner for vector problems.
>>
>
> The block size is used if you don't provide any other information. The
> preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll
> make sure this is currently working with ML and reply to this message.)
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/67a9a7ab/attachment.htm>

From mark.adams at columbia.edu  Mon Apr  2 11:01:38 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Mon, 2 Apr 2012 12:01:38 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CA+gX-L8T0rUAMcQGSveMBQt6PU=BFtmLNw0ptsq_q=zNx6L+Zg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<27A4BE6D-B6B2-4724-AB19-A3958892DA3A@columbia.edu>
	<CA+gX-L8T0rUAMcQGSveMBQt6PU=BFtmLNw0ptsq_q=zNx6L+Zg@mail.gmail.com>
Message-ID: <3E21E9D8-CDB1-483D-9975-F780D0D2A0DA@columbia.edu>


On Apr 2, 2012, at 11:37 AM, Karin&NiKo wrote:

> Dear Jed and Mark.
> 
> Thank you for your quick answers.
> 1. I am stuck to petsc-3.2 since I am using PETSc in a production software. For the next release, I'll give GAMG a try (I have been a satisfied user of Prometheus).

3.2 should work ... 

> 2. It seems to me that ML does not support block matrices, doesn't it?

The ML interface does get the block size of the matrix.  So it does the right thing.

Mark

> 3. I would like to give ML the information about the vector nature of my problem in order that all 3 dofs of a given node be treated in the same manner (in the coarsening phase for instance).
> 
> Nicolas
> 
> 2012/4/2 Mark F. Adams <mark.adams at columbia.edu>
> FYI, you can try GAMG also.  It uses the same algorithm as ML:
> 
> Here are parameters:
> 
> -pc_type gamg 
> -pc_gamg_type agg 
> -pc_gamg_agg_nsmooths 1     ! this is good for elliptic problems
> -pc_gamg_verbose 2                ! this will help me to give a quick sanity check
> -pc_gamg_threshold .05           ! for 3D problems this parameter can be useful for optimization
> -pc_gamg_coarse_eq_limit 50  ! a detail but you can keep in your input deck
> 
> The vector Lapacian has 6 null space vectors (3 translational, 3 rotational).  GAMG will construct these with the a setCoordinates method (see the ksp tutorial example ex56.c).   If you prefer to give me the null space vectors explicitly there is the MatSetNearNullSpace() method that Jed mentioned.  GAMG has not implemented this but I should at some point so if you want to use it I can implement it.
> 
> If you do not give it coordinates it will construct the 3 translational null space vectors (don't need anything to compute these) and it can be OK.  At least a good place to start.
> 
> Mark
> 
> On Apr 2, 2012, at 10:44 AM, Jed Brown wrote:
> 
>> On Mon, Apr 2, 2012 at 06:59, Karin&NiKo <niko.karin at gmail.com> wrote:
>> I would like to use ML as a preconditioner for a linear elasticity problem.
>> I wonder if there is a way to indicate ML that it is dealing with a vector problem. 
>> I know that when using Prometheus, one must use a block matrix, and I would like to know if there is a trick 
>> in order to use ML in an optimal manner for vector problems.
>> 
>> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/d994becc/attachment.htm>

From li at loyno.edu  Mon Apr  2 11:33:12 2012
From: li at loyno.edu (Xuefeng Li)
Date: Mon, 2 Apr 2012 11:33:12 -0500 (CDT)
Subject: [petsc-users] dmmg_grid_sequence
In-Reply-To: <CAMYG4GmBXuHHez4biRh7Y9RpG7Ju0sU1XAwfHKjF1U-oM+NF1Q@mail.gmail.com>
References: <58356696.1460341259111616362.JavaMail.root@mail8.gatech.edu>
	<DABA8EB2-07CA-497A-BB8D-8F4C6273D886@mcs.anl.gov>
	<Pine.A41.4.61.1201061608490.7373010@loyno.edu>
	<CAM9tzSk0YW5+Xzxg2UQSDaXEnzLUOxCbXn9_EZcrnuC3HA08+A@mail.gmail.com>
	<Pine.A41.4.61.1201061629510.3997766@loyno.edu>
	<CAMYG4GkUnT+rbudt5xmSyT0S-vqgC80zSdjP9VOjRCbxBYNMSA@mail.gmail.com>
	<Pine.A41.4.61.1201061639020.9515068@loyno.edu>
	<C6361099-5817-4D37-88BC-28FD0337CEB3@mcs.anl.gov>
	<Pine.A41.4.61.1204020958410.7057648@loyno.edu>
	<CAMYG4GmBXuHHez4biRh7Y9RpG7Ju0sU1XAwfHKjF1U-oM+NF1Q@mail.gmail.com>
Message-ID: <Pine.A41.4.61.1204021126290.3117120@loyno.edu>

On Mon, 2 Apr 2012, Matthew Knepley wrote:

> On Mon, Apr 2, 2012 at 10:11 AM, Xuefeng Li <li at loyno.edu> wrote:
>
>> Hi, all.
>>
>> I am using multigrid with DMMG (5 levels) in my petsc program.
>> Turning on the dmmg_grid_sequence option seems
>> to slow down the program dramatically. For instance,
>> the program converges in 3 iterations in 6.270e+01 sec
>> when dmmg_grid_sequence is OFF; it also converges
>> in 3 iterations but in 1.699e+04 sec when dmmg_grid_sequence
>> is ON.
>>
>> Is this typical of petsc (I'm using petsc-3.1-p8)?
>> Are there ways to improve the performance when
>> dmmg_grid_sequence is ON?
>>
>
> Its impossible to know what is happening from this description. There is no
> "generic" performance of grid sequencing. The best thing I can say to this
> is,
> it looks like the interpolated solution to your coarse problem is not a
> great
> solution to your fine problem.
>
That is possible.



> However, you could be oversolving the coarse
> problems, blowing up the work, or interpolating badly for your problem.
>
What do you mean by "oversolving the coarse problems"?
I am using the same stopping conditions in the two runs.
The only difference is dmmg_grid_sequence ON and OFF.

I also observed that in other situations, when dmmg_grid_sequence
is ON, the execution will abort due to "Out of memory" error,
whereas when OFF, the program converges without error.
How much more memory does grid-sequence employ? Twice as much?

Regards,

--Xuefeng Li, (504)865-3340(phone)
   Like floating clouds, the heart rests easy
   Like flowing water, the spirit stays free
        http://www.loyno.edu/~li/home
   New Orleans, Louisiana (504)865-2051(fax)


From knepley at gmail.com  Mon Apr  2 12:20:21 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 12:20:21 -0500
Subject: [petsc-users] dmmg_grid_sequence
In-Reply-To: <Pine.A41.4.61.1204021126290.3117120@loyno.edu>
References: <58356696.1460341259111616362.JavaMail.root@mail8.gatech.edu>
	<DABA8EB2-07CA-497A-BB8D-8F4C6273D886@mcs.anl.gov>
	<Pine.A41.4.61.1201061608490.7373010@loyno.edu>
	<CAM9tzSk0YW5+Xzxg2UQSDaXEnzLUOxCbXn9_EZcrnuC3HA08+A@mail.gmail.com>
	<Pine.A41.4.61.1201061629510.3997766@loyno.edu>
	<CAMYG4GkUnT+rbudt5xmSyT0S-vqgC80zSdjP9VOjRCbxBYNMSA@mail.gmail.com>
	<Pine.A41.4.61.1201061639020.9515068@loyno.edu>
	<C6361099-5817-4D37-88BC-28FD0337CEB3@mcs.anl.gov>
	<Pine.A41.4.61.1204020958410.7057648@loyno.edu>
	<CAMYG4GmBXuHHez4biRh7Y9RpG7Ju0sU1XAwfHKjF1U-oM+NF1Q@mail.gmail.com>
	<Pine.A41.4.61.1204021126290.3117120@loyno.edu>
Message-ID: <CAMYG4GmLrDesqHsKcPg7pn5bnEOgXMiQ7faqFWDbz2JuTco__Q@mail.gmail.com>

On Mon, Apr 2, 2012 at 11:33 AM, Xuefeng Li <li at loyno.edu> wrote:

> On Mon, 2 Apr 2012, Matthew Knepley wrote:
>
>  On Mon, Apr 2, 2012 at 10:11 AM, Xuefeng Li <li at loyno.edu> wrote:
>>
>>  Hi, all.
>>>
>>> I am using multigrid with DMMG (5 levels) in my petsc program.
>>> Turning on the dmmg_grid_sequence option seems
>>> to slow down the program dramatically. For instance,
>>> the program converges in 3 iterations in 6.270e+01 sec
>>> when dmmg_grid_sequence is OFF; it also converges
>>> in 3 iterations but in 1.699e+04 sec when dmmg_grid_sequence
>>> is ON.
>>>
>>> Is this typical of petsc (I'm using petsc-3.1-p8)?
>>> Are there ways to improve the performance when
>>> dmmg_grid_sequence is ON?
>>>
>>>
>> Its impossible to know what is happening from this description. There is
>> no
>> "generic" performance of grid sequencing. The best thing I can say to this
>> is,
>> it looks like the interpolated solution to your coarse problem is not a
>> great
>> solution to your fine problem.
>>
>>  That is possible.
>
>
>
>  However, you could be oversolving the coarse
>> problems, blowing up the work, or interpolating badly for your problem.
>>
>>  What do you mean by "oversolving the coarse problems"?
> I am using the same stopping conditions in the two runs.
>

Which may be too tight for the coarse runs.


> The only difference is dmmg_grid_sequence ON and OFF.
>
> I also observed that in other situations, when dmmg_grid_sequence
> is ON, the execution will abort due to "Out of memory" error,
> whereas when OFF, the program converges without error.
> How much more memory does grid-sequence employ? Twice as much?
>

If you use Galerkin MG, then t would depend on how full your Jacobian is.

   Matt


> Regards,
>
> --Xuefeng Li, (504)865-3340(phone)
>  Like floating clouds, the heart rests easy
>  Like flowing water, the spirit stays free
>       http://www.loyno.edu/~li/home
>  New Orleans, Louisiana (504)865-2051(fax)
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/b6ef910c/attachment-0001.htm>

From khalid_eee at yahoo.com  Mon Apr  2 16:53:08 2012
From: khalid_eee at yahoo.com (khalid ashraf)
Date: Mon, 2 Apr 2012 14:53:08 -0700 (PDT)
Subject: [petsc-users] transfer vector data diagonally on DA
In-Reply-To: <CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
	<CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
Message-ID: <1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>

Thanks Matt. It works now.?

I have another problem. I am writing this program.?
?
?for (k=zs; k<zs+zm; k++) {
??for (j=ys; j<ys+ym; j++) {
??for (i=xs; i<xs+xm; i++) {
?if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
{
bodyFx[0]=1;
bodyFx[1]=-0.3;
bodyFx[2]=-0.3;
bodyFx[3]=-0.3;
row[0]=k*mx*my+j*mx+i;
row[1]=k*mx*my+j*mx+i+1;
row[2]=k*mx*my+(j+1)*mx+i;
row[3]=(k+1)*mx*my+j*mx+i;
VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);
}
VecAssemblyBegin(fx_test);
VecAssemblyEnd(fx_test);

Output: Print fx_test

?Here fx_test is a global vector. This program gives correct output on 1 proc. but on 4 processors,
the program runs but the outputs are wrong.

I also tried
?for (k=zs; k<zs+zm; k++) {
??for (j=ys; j<ys+ym; j++) {
??for (i=xs; i<xs+xm; i++) {
?if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
{
bodyFx[0]=1;
bodyFx[1]=-0.3;
bodyFx[2]=-0.3;
bodyFx[3]=-0.3;
?fx1_localptr[k][j][i+1]+=bodyFx[4];
?fx1_localptr[k][j+1][i]+=bodyFx[3];
?fx1_localptr[k+1][j][i]+=bodyFx[1];
?fx1_localptr[k][j][i]+=bodyFx[0];
}
Here, fx1_localptr is an array from a local vector that is communicated from the global vector fx_test.
In this case as well, I get similar errors.?



Thanks.
Khalid

Some final numbers of the vector with 8*8*6 grid, 4 proc, ?
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
1.0000000000000009e-01
1.0000000000000009e-01
1.0000000000000009e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
0.0000000000000000e+00
4.0000000000000002e-01
1.0000000000000009e-01
1.0000000000000009e-01
1.0000000000000009e-01
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
1.0000000000000009e-01
1.0000000000000009e-01
1.0000000000000009e-01
-2.9999999999999999e-01
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00


Some final numbers of the vector with 8*8*6 grid, 1 proc, ?

0.0000000000000000e+00
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
0.0000000000000000e+00
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
0.0000000000000000e+00
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
-2.9999999999999999e-01
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00
0.0000000000000000e+00

?? ? ? ? ? ? ? ? ? ? ? ? ? ? ??


________________________________
 From: Matthew Knepley <knepley at gmail.com>
To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
Sent: Monday, April 2, 2012 5:41 AM
Subject: Re: [petsc-users] transfer vector data diagonally on DA
 

On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com> wrote:

Hi Jed,
>
>
>I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h"
>and DACreate3D.?
>unew_localptr[][][] is from a global vector using VecGetArray
>u_localptr[][][] is from a local vector communicated from a global vector.
>I tried?
>unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
>
>but it gives Segmentation Violation error.?

I would guess (because you provide almost no information about what you are doing), that
this is a domain on the edge. If you truly mean to have a periodic domain, you must set that in the
creation call. Then the local vector will also ghost regions outside the domain boundary.

? ?Matt
?
I also tried?unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but?
>unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or?unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
>does not work. I am running the program on4 processors.
>
>
>
>Thanks.
>
>
>
>
>??ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
>?? ? ? ? ? ? ? ? ? ?PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>?? ? ? ? ? ? ? ? ? ?&appctx.da);CHKERRQ(ierr);
>?for (k=zs; k<zs+zm; k++) {
>??for (j=ys; j<ys+ym; j++) {
>??for (i=xs; i<xs+xm; i++) {
>
>
>??if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
>
>
>??unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
>
>}}}
>
>________________________________
>
>From: Jed Brown <jedbrown at mcs.anl.gov>
>To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
>Sent: Sunday, April 1, 2012 10:07 PM
>Subject: Re: [petsc-users] transfer vector data diagonally on DA
>
>
>
>On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:
>
>I want to transfer vector data diagonally in the DA grid like?
>>?for (k=zs; k<zs+zm; k++) {
>>??for (j=ys; j<ys+ym; j++) {
>>??for (i=xs; i<xs+xm; i++) {
>>if(i!=mx-1 || j!=my-1 || k!=mz-1){
>>u_new[k+1][j+1][i+1]=u[k][j][i];}
>>}}}
>>
>>
>>Could you please suggest the best way to do it minimizing interprocessor assignments.
>
>Both are on the same DMDA?
>
>
>Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].
>
>


-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/198a0763/attachment.htm>

From knepley at gmail.com  Mon Apr  2 17:08:46 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 17:08:46 -0500
Subject: [petsc-users] transfer vector data diagonally on DA
In-Reply-To: <1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
	<CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
	<1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>
Message-ID: <CAMYG4GkVuCvqiZjYCwTWoQs3RUPgRGZ0sdHgkSAR_CCEhAZH8g@mail.gmail.com>

On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <khalid_eee at yahoo.com> wrote:

> Thanks Matt. It works now.
>
> I have another problem. I am writing this program.
>

1) Always test on 2 procs once

2) I have no idea what you actually want to do. However, it really simple
to just print out
    the indices you are using in row[] and the values in bodyFx[] and check
that VecSetValues()
    is working as you expect.

   Matt


>
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> {
> bodyFx[0]=1;
> bodyFx[1]=-0.3;
> bodyFx[2]=-0.3;
> bodyFx[3]=-0.3;
> row[0]=k*mx*my+j*mx+i;
> row[1]=k*mx*my+j*mx+i+1;
> row[2]=k*mx*my+(j+1)*mx+i;
> row[3]=(k+1)*mx*my+j*mx+i;
> VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);
> }
> VecAssemblyBegin(fx_test);
> VecAssemblyEnd(fx_test);
>
> Output: Print fx_test
>
>  Here fx_test is a global vector. This program gives correct output on 1
> proc. but on 4 processors,
> the program runs but the outputs are wrong.
>
> I also tried
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> {
> bodyFx[0]=1;
> bodyFx[1]=-0.3;
> bodyFx[2]=-0.3;
> bodyFx[3]=-0.3;
>  fx1_localptr[k][j][i+1]+=bodyFx[4];
>  fx1_localptr[k][j+1][i]+=bodyFx[3];
>  fx1_localptr[k+1][j][i]+=bodyFx[1];
>  fx1_localptr[k][j][i]+=bodyFx[0];
> }
> Here, fx1_localptr is an array from a local vector that is communicated
> from the global vector fx_test.
> In this case as well, I get similar errors.
>
> Thanks.
> Khalid
>
> Some final numbers of the vector with 8*8*6 grid, 4 proc,
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 4.0000000000000002e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
>
>
> Some final numbers of the vector with 8*8*6 grid, 1 proc,
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
>
>
>
>   ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *To:* khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Monday, April 2, 2012 5:41 AM
> *Subject:* Re: [petsc-users] transfer vector data diagonally on DA
>
> On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com>wrote:
>
> Hi Jed,
>
> I am using petsc/3.1 and the include file "petscdmda.h" is not working. I
> am using the "petscda.h"
> and DACreate3D.
> unew_localptr[][][] is from a global vector using VecGetArray
> u_localptr[][][] is from a local vector communicated from a global vector.
> I tried
> unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
> but it gives Segmentation Violation error.
>
>
> I would guess (because you provide almost no information about what you
> are doing), that
> this is a domain on the edge. If you truly mean to have a periodic domain,
> you must set that in the
> creation call. Then the local vector will also ghost regions outside the
> domain boundary.
>
>    Matt
>
>
> I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works
> but
> unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1]
> or unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
> does not work. I am running the program on4 processors.
>
> Thanks.
>
>
>   ierr =
> DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
>
>  PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>                     &appctx.da);CHKERRQ(ierr);
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>
>   if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
>
>   unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
> }}}
>   ------------------------------
> *From:* Jed Brown <jedbrown at mcs.anl.gov>
> *To:* khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Sunday, April 1, 2012 10:07 PM
> *Subject:* Re: [petsc-users] transfer vector data diagonally on DA
>
> On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:
>
> I want to transfer vector data diagonally in the DA grid like
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
> if(i!=mx-1 || j!=my-1 || k!=mz-1){
> u_new[k+1][j+1][i+1]=u[k][j][i];}
> }}}
>
> Could you please suggest the best way to do it minimizing interprocessor
> assignments.
>
>
> Both are on the same DMDA?
>
> Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil
> with width at least 1, get the global array u_new[][][] from UGlobalNew and
> the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] =
> u[k-1][j-1][i-1].
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/1e98dcc6/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Mon Apr  2 18:24:44 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 2 Apr 2012 16:24:44 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
Message-ID: <CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>

On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> The block size is used if you don't provide any other information. The
> preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll
> make sure this is currently working with ML and reply to this message.)


I pushed ML support for user-provided null spaces to petsc-dev. Call
MatSetNearNullSpace() to provide the information. You can use
MatNullSpaceCreateRigidBody() if you have nodal coordinates.


Mark, what is the best way to make PCGAMG use this interface?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/b6214df8/attachment.htm>

From mark.adams at columbia.edu  Mon Apr  2 18:40:20 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Mon, 2 Apr 2012 19:40:20 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
Message-ID: <7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>


On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:

> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
> 
> I pushed ML support for user-provided null spaces to petsc-dev. Call MatSetNearNullSpace() to provide the information. You can use MatNullSpaceCreateRigidBody() if you have nodal coordinates.
> 
> 
> Mark, what is the best way to make PCGAMG use this interface?


It looks like you are adding this to the Mat, so I can check to see if coordinates have been set and if not check the matrix.  This should be easy to do; I'll look at the ML code to clone the API.

One thing to keep in mind is that diagonal scaling breaks the null space (ie, the rigid body modes have to be scaled appropriately).  Who owns the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do this, otherwise we should think of a good way to deal with this.  It is very error prone to not do the right thing here, we should at least throw an error.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/d671cf89/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  2 18:40:40 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 2 Apr 2012 16:40:40 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
Message-ID: <CAM9tzSnim=NiUoWTr=9_BYkxEL-WRSo1K2TpjE2772+nifv1Eg@mail.gmail.com>

We don't keep coordinates, just use the provided null space.

We should make MatDiagonalScale update the near null space.
On Apr 2, 2012 4:38 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:

>
> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
>
> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> The block size is used if you don't provide any other information. The
>> preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll
>> make sure this is currently working with ML and reply to this message.)
>
>
> I pushed ML support for user-provided null spaces to petsc-dev. Call
> MatSetNearNullSpace() to provide the information. You can use
> MatNullSpaceCreateRigidBody() if you have nodal coordinates.
>
>
> Mark, what is the best way to make PCGAMG use this interface?
>
>
> It looks like you are adding this to the Mat, so I can check to see if
> coordinates have been set and if not check the matrix.  This should be easy
> to do; I'll look at the ML code to clone the API.
>
> One thing to keep in mind is that diagonal scaling breaks the null space
> (ie, the rigid body modes have to be scaled appropriately).  Who owns the
> diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do
> this, otherwise we should think of a good way to deal with this.  It is
> very error prone to not do the right thing here, we should at least throw
> an error.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/d18642d0/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr  2 18:40:42 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 2 Apr 2012 18:40:42 -0500
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
Message-ID: <8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>


  Mark,

   Who would be doing the "diagonal scaling". Under normal conditions we never want to be passing to any preconditioner a diagonally scaled matrix (for exactly the issue you point out) so why are you worried about that case?

   Barry

On Apr 2, 2012, at 6:40 PM, Mark F. Adams wrote:

> 
> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
> 
>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
>> 
>> I pushed ML support for user-provided null spaces to petsc-dev. Call MatSetNearNullSpace() to provide the information. You can use MatNullSpaceCreateRigidBody() if you have nodal coordinates.
>> 
>> 
>> Mark, what is the best way to make PCGAMG use this interface?
> 
> It looks like you are adding this to the Mat, so I can check to see if coordinates have been set and if not check the matrix.  This should be easy to do; I'll look at the ML code to clone the API.
> 
> One thing to keep in mind is that diagonal scaling breaks the null space (ie, the rigid body modes have to be scaled appropriately).  Who owns the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do this, otherwise we should think of a good way to deal with this.  It is very error prone to not do the right thing here, we should at least throw an error.
> 
> 


From mark.adams at columbia.edu  Mon Apr  2 18:50:30 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Mon, 2 Apr 2012 19:50:30 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzSnim=NiUoWTr=9_BYkxEL-WRSo1K2TpjE2772+nifv1Eg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<CAM9tzSnim=NiUoWTr=9_BYkxEL-WRSo1K2TpjE2772+nifv1Eg@mail.gmail.com>
Message-ID: <2D4D4B18-3183-41B2-9F43-1AAD7B08868E@columbia.edu>

But your MatNullSpaceCreateRigidBody takes coordinates right?

I was saying that I would see if the RBMs have been created with the PCSetCoordinated method that exists now.  If the data is not set then I now just create the translational RBMs.  I can now simply check the matrix and see if it has "null space" vectors attached and use those instead of creating the default null space.  So this should be very easy to do.


On Apr 2, 2012, at 7:40 PM, Jed Brown wrote:

> We don't keep coordinates, just use the provided null space.
> 
> We should make MatDiagonalScale update the near null space.
> 
> On Apr 2, 2012 4:38 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:
> 
> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
> 
>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
>> 
>> I pushed ML support for user-provided null spaces to petsc-dev. Call MatSetNearNullSpace() to provide the information. You can use MatNullSpaceCreateRigidBody() if you have nodal coordinates.
>> 
>> 
>> Mark, what is the best way to make PCGAMG use this interface?
> 
> 
> It looks like you are adding this to the Mat, so I can check to see if coordinates have been set and if not check the matrix.  This should be easy to do; I'll look at the ML code to clone the API.
> 
> One thing to keep in mind is that diagonal scaling breaks the null space (ie, the rigid body modes have to be scaled appropriately).  Who owns the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do this, otherwise we should think of a good way to deal with this.  It is very error prone to not do the right thing here, we should at least throw an error.
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/ca84edbd/attachment.htm>

From mark.adams at columbia.edu  Mon Apr  2 18:53:03 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Mon, 2 Apr 2012 19:53:03 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
Message-ID: <6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>

Oh, OK, so you do not explicitly scale the matrix good.  We had a user (John) recently that was using diagonal scaling and was getting failure with GAMG and then GAMG worked and he was no longer using diagonal scaling so I was thinking this was the problem.

Mark

On Apr 2, 2012, at 7:40 PM, Barry Smith wrote:

> 
>  Mark,
> 
>   Who would be doing the "diagonal scaling". Under normal conditions we never want to be passing to any preconditioner a diagonally scaled matrix (for exactly the issue you point out) so why are you worried about that case?
> 
>   Barry
> 
> On Apr 2, 2012, at 6:40 PM, Mark F. Adams wrote:
> 
>> 
>> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
>> 
>>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
>>> 
>>> I pushed ML support for user-provided null spaces to petsc-dev. Call MatSetNearNullSpace() to provide the information. You can use MatNullSpaceCreateRigidBody() if you have nodal coordinates.
>>> 
>>> 
>>> Mark, what is the best way to make PCGAMG use this interface?
>> 
>> It looks like you are adding this to the Mat, so I can check to see if coordinates have been set and if not check the matrix.  This should be easy to do; I'll look at the ML code to clone the API.
>> 
>> One thing to keep in mind is that diagonal scaling breaks the null space (ie, the rigid body modes have to be scaled appropriately).  Who owns the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do this, otherwise we should think of a good way to deal with this.  It is very error prone to not do the right thing here, we should at least throw an error.
>> 
>> 
> 
> 


From jedbrown at mcs.anl.gov  Mon Apr  2 18:53:35 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 2 Apr 2012 16:53:35 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
Message-ID: <CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>

Well, the KSP can diagonally scale it and we should fix up the near null
space. The PC should not need to find out about diagonal scaling. (My
opinion.)
On Apr 2, 2012 4:50 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:

> Oh, OK, so you do not explicitly scale the matrix good.  We had a user
> (John) recently that was using diagonal scaling and was getting failure
> with GAMG and then GAMG worked and he was no longer using diagonal scaling
> so I was thinking this was the problem.
>
> Mark
>
> On Apr 2, 2012, at 7:40 PM, Barry Smith wrote:
>
> >
> >  Mark,
> >
> >   Who would be doing the "diagonal scaling". Under normal conditions we
> never want to be passing to any preconditioner a diagonally scaled matrix
> (for exactly the issue you point out) so why are you worried about that
> case?
> >
> >   Barry
> >
> > On Apr 2, 2012, at 6:40 PM, Mark F. Adams wrote:
> >
> >>
> >> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
> >>
> >>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> >>> The block size is used if you don't provide any other information. The
> preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll
> make sure this is currently working with ML and reply to this message.)
> >>>
> >>> I pushed ML support for user-provided null spaces to petsc-dev. Call
> MatSetNearNullSpace() to provide the information. You can use
> MatNullSpaceCreateRigidBody() if you have nodal coordinates.
> >>>
> >>>
> >>> Mark, what is the best way to make PCGAMG use this interface?
> >>
> >> It looks like you are adding this to the Mat, so I can check to see if
> coordinates have been set and if not check the matrix.  This should be easy
> to do; I'll look at the ML code to clone the API.
> >>
> >> One thing to keep in mind is that diagonal scaling breaks the null
> space (ie, the rigid body modes have to be scaled appropriately).  Who owns
> the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace
> to do this, otherwise we should think of a good way to deal with this.  It
> is very error prone to not do the right thing here, we should at least
> throw an error.
> >>
> >>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/e30f220d/attachment-0001.htm>

From stali at geology.wisc.edu  Mon Apr  2 20:10:11 2012
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Mon, 02 Apr 2012 20:10:11 -0500
Subject: [petsc-users] Debugger question
Message-ID: <4F7A4DF3.9090907@geology.wisc.edu>

Hello

I am trying to debug a program using the switch 
'-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 
is unable to start the debugger in xterm (see text below). But xterm is 
installed. What am I doing wrong?

Btw the segfault happens during a call to MatMult but only with 
vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and 
_not_ with CRAY or GNU compilers.

I also dont get the segfault if I build PETSc 3.2-p7 myself with 
PGI/Intel compilers.

Any ideas on how to diagnose the problem? Unfortunately I cannot seem to 
run valgrind on this particular machine.

Thanks in advance.

Tabrez

---

stali at krakenpf1:~/meshes> which xterm
/usr/bin/xterm
stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f 
2d_point_load_dyn_abc.inp -on_error_attach_debugger
...
...
...
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display 
localhost:20.0 on machine nid10649
Unable to start debugger in xterm: No such file or directory
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
_pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 
2012] PE 0 exit signal Aborted
Application 133198 exit codes: 134
Application 133198 resources: utime ~1s, stime ~0s

From knepley at gmail.com  Mon Apr  2 20:45:13 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 20:45:13 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7A4DF3.9090907@geology.wisc.edu>
References: <4F7A4DF3.9090907@geology.wisc.edu>
Message-ID: <CAMYG4GmgG=4+YSZCAX4fEHAc_5gvDf1AMhfhOpFH+MUzAG3Nhg@mail.gmail.com>

On Mon, Apr 2, 2012 at 8:10 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:

> Hello
>
> I am trying to debug a program using the switch
> '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is
> unable to start the debugger in xterm (see text below). But xterm is
> installed. What am I doing wrong?
>

We need the entire output since the X messages may be farther up. Also, run
it with -start_in_debugger so that they
come right at startup.

  Thanks,

     Matt


> Btw the segfault happens during a call to MatMult but only with
> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and
> _not_ with CRAY or GNU compilers.
>
> I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel
> compilers.
>
> Any ideas on how to diagnose the problem? Unfortunately I cannot seem to
> run valgrind on this particular machine.
>
> Thanks in advance.
>
> Tabrez
>
> ---
>
> stali at krakenpf1:~/meshes> which xterm
> /usr/bin/xterm
> stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f
> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
> ...
> ...
> ...
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> petsc-as/documentation/faq.**html#valgrind[0]PETSC<http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display
> localhost:20.0 on machine nid10649
> Unable to start debugger in xterm: No such file or directory
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48
> 2012] PE 0 exit signal Aborted
> Application 133198 exit codes: 134
> Application 133198 resources: utime ~1s, stime ~0s
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/cef9d865/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr  2 20:57:19 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 2 Apr 2012 20:57:19 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7A4DF3.9090907@geology.wisc.edu>
References: <4F7A4DF3.9090907@geology.wisc.edu>
Message-ID: <F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>


On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:

> Hello
> 
> I am trying to debug a program using the switch '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is unable to start the debugger in xterm (see text below). But xterm is installed. What am I doing wrong?
> 
> Btw the segfault happens during a call to MatMult but only with vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and _not_ with CRAY or GNU compilers.

   My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just built it yourself so you can get real work done instead of trying to debug their mess.   I promise the vendor one is not like a billion times faster or anything.

   Barry


   
> 
> I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel compilers.
> 
> Any ideas on how to diagnose the problem? Unfortunately I cannot seem to run valgrind on this particular machine.
> 
> Thanks in advance.
> 
> Tabrez
> 
> ---
> 
> stali at krakenpf1:~/meshes> which xterm
> /usr/bin/xterm
> stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f 2d_point_load_dyn_abc.inp -on_error_attach_debugger
> ...
> ...
> ...
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display localhost:20.0 on machine nid10649
> Unable to start debugger in xterm: No such file or directory
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 2012] PE 0 exit signal Aborted
> Application 133198 exit codes: 134
> Application 133198 resources: utime ~1s, stime ~0s


From knepley at gmail.com  Mon Apr  2 21:04:24 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 21:04:24 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
Message-ID: <CAMYG4G=H_GPVjtWqwJQ=v4y18acj5W1Kwaf7k3pO6mVvB=-83Q@mail.gmail.com>

On Mon, Apr 2, 2012 at 8:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
>
> > Hello
> >
> > I am trying to debug a program using the switch
> '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is
> unable to start the debugger in xterm (see text below). But xterm is
> installed. What am I doing wrong?
> >
> > Btw the segfault happens during a call to MatMult but only with
> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and
> _not_ with CRAY or GNU compilers.
>
>    My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just
> built it yourself so you can get real work done instead of trying to debug
> their mess.   I promise the vendor one is not like a billion times faster
> or anything.


If you want to justify this to anyone (like a funder), just run both on ex5
for a large size and look at the flops on MatMult. That
is probably your dominant cost (or your PC).

   Matt


>
>   Barry
>
>
>
> >
> > I also dont get the segfault if I build PETSc 3.2-p7 myself with
> PGI/Intel compilers.
> >
> > Any ideas on how to diagnose the problem? Unfortunately I cannot seem to
> run valgrind on this particular machine.
> >
> > Thanks in advance.
> >
> > Tabrez
> >
> > ---
> >
> > stali at krakenpf1:~/meshes> which xterm
> > /usr/bin/xterm
> > stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f
> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
> > ...
> > ...
> > ...
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> > [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> > [0]PETSC ERROR: to get more information on the crash.
> > [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> > [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display
> localhost:20.0 on machine nid10649
> > Unable to start debugger in xterm: No such file or directory
> > aborting job:
> > application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> > _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48
> 2012] PE 0 exit signal Aborted
> > Application 133198 exit codes: 134
> > Application 133198 resources: utime ~1s, stime ~0s
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/954a208a/attachment.htm>

From balay at mcs.anl.gov  Mon Apr  2 21:05:07 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 2 Apr 2012 21:05:07 -0500 (CDT)
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7A4DF3.9090907@geology.wisc.edu>
References: <4F7A4DF3.9090907@geology.wisc.edu>
Message-ID: <alpine.LFD.2.02.1204022101020.2087@asterix>

Sounds like a Cray machine.

start_in_debugger is useful for debugging on workstations [or
clusters] etc where there is some control on X11 tunnels. Also
'xterm','gdb' or similar debugger should be available on the compute
nodes [along with a x/ssh tunnel].

On a cray - you are better off looking for a parallel debugger. Don't
know if cray has one available.

Wrt debugging - you might want to run your code with valgrind on a
linux box..

Satish

On Mon, 2 Apr 2012, Tabrez Ali wrote:

> Hello
> 
> I am trying to debug a program using the switch '-on_error_attach_debugger'
> but  the vendor/sysadmin built PETSc 3.2.00 is unable to start the debugger in
> xterm (see text below). But xterm is installed. What am I doing wrong?
> 
> Btw the segfault happens during a call to MatMult but only with
> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and _not_
> with CRAY or GNU compilers.
> 
> I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel
> compilers.
> 
> Any ideas on how to diagnose the problem? Unfortunately I cannot seem to run
> valgrind on this particular machine.
> 
> Thanks in advance.
> 
> Tabrez
> 
> ---
> 
> stali at krakenpf1:~/meshes> which xterm
> /usr/bin/xterm
> stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f 2d_point_load_dyn_abc.inp
> -on_error_attach_debugger
> ...
> ...
> ...
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably
> memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown
> file
> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display
> localhost:20.0 on machine nid10649
> Unable to start debugger in xterm: No such file or directory
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 2012] PE
> 0 exit signal Aborted
> Application 133198 exit codes: 134
> Application 133198 resources: utime ~1s, stime ~0s
> 


From knepley at gmail.com  Mon Apr  2 21:37:46 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 21:37:46 -0500
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
Message-ID: <CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>

On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>
>> Yes. This would work.
>> I had trouble compiling in single precision using the some of the
>> external package options I was using for double.
>>
>
> Okay,  getting on it now.
>

Okay, I have tested using PetscReal=double with floats on the GPU. If you
want doubles on the GPU, you can
currently change PETSc/FEM.py:434. I am deciding what the best way to
specify this type is.

  Thanks,

     Matt


>    Matt
>
>
>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>
>>>> works!
>>>>
>>>
>>> Excellent. Now, my thinking was that GPUs are most useful doing single
>>> work, but
>>> I can see the utility of double accuracy for a residual.
>>>
>>> My inclination is to define another type, say GPUReal, and use it for
>>> all kernels.
>>> Would that do what you want?
>>>
>>>    Matt
>>>
>>>
>>>> SCRGP2$ make ex52
>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>> declared but never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>> declared but never referenced
>>>>
>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>  -lpetsc
>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>
>>>>
>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>  N_t: 3, N_cb: 1
>>>> Residual:
>>>> Vector Object: 1 MPI processes
>>>>   type: seq
>>>> -0.25
>>>> -0.5
>>>> 0.25
>>>> -0.5
>>>> -1
>>>> 0.5
>>>> 0.25
>>>> 0.5
>>>> 0.75
>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>> Residual:
>>>> Vector Object: 1 MPI processes
>>>>   type: seq
>>>> -0.25
>>>> -0.5
>>>> 0.25
>>>> -0.5
>>>> -1
>>>> 0.5
>>>> 0.25
>>>> 0.5
>>>> 0.75
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>
>>>>> sure. will do.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>
>>>>>>> thanks! its running, but I seem to be getting different answer for
>>>>>>> cpu/gpu ?
>>>>>>> i had some floating point problems on this Tesla M2070 gpu before,
>>>>>>> but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>
>>>>>>>
>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>
>>>>>>
>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>> have not decided exactly how to handle it. Can you
>>>>>> make another ARCH --with-precision=single and make sure it works, and
>>>>>> then we can fix the double issue?
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>  N_t: 3, N_cb: 1
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>>   type: seq
>>>>>>> 0
>>>>>>> 755712
>>>>>>> 0
>>>>>>> -58720
>>>>>>> -2953.13
>>>>>>> 0.375
>>>>>>> 1.50323e+07
>>>>>>> 0.875
>>>>>>> 0
>>>>>>> SCRGP2$
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>>   type: seq
>>>>>>> -0.25
>>>>>>> -0.5
>>>>>>> 0.25
>>>>>>> -0.5
>>>>>>> -1
>>>>>>> 0.5
>>>>>>> 0.25
>>>>>>> 0.5
>>>>>>> 0.75
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <knepley at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>
>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>
>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> No.
>>>>>>>>
>>>>>>>>
>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>> sizeof(float));CHKERRQ(ierr);
>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SCRGP2$ python
>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>> laplacian ex52.h
>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}}
>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}}
>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>
>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>   type: seq
>>>>>>>>> -0.25
>>>>>>>>> -0.5
>>>>>>>>> 0.25
>>>>>>>>> -0.5
>>>>>>>>> -1
>>>>>>>>> 0.5
>>>>>>>>> 0.25
>>>>>>>>> 0.5
>>>>>>>>> 0.75
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>> If so, applications like Preview can lock up the GPU. I know of no
>>>>>>>> way to test this in CUDA while running. I just close
>>>>>>>> apps until it runs.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <bourdin at lsu.edu>wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>
>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>> using the Python build system
>>>>>>>>>>>
>>>>>>>>>>>   ./config/builder2.py check src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>
>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>
>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Blaise
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> David
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  --
>>>>>>>>>>> Department of Mathematics and Center for Computation & Technology
>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/bfdfce0e/attachment-0001.htm>

From stali at geology.wisc.edu  Mon Apr  2 21:52:47 2012
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Mon, 02 Apr 2012 21:52:47 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <CAMYG4G=H_GPVjtWqwJQ=v4y18acj5W1Kwaf7k3pO6mVvB=-83Q@mail.gmail.com>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
	<CAMYG4G=H_GPVjtWqwJQ=v4y18acj5W1Kwaf7k3pO6mVvB=-83Q@mail.gmail.com>
Message-ID: <4F7A65FF.1030109@geology.wisc.edu>

Matt/Barry

My intention was to make sure that the code is bug free and since PETSc 
was pre-installed on the cluster with various compilers it was easier to 
test quickly rather than build all combinations myself. Performance is 
of absolutely no concern.

Things were working fine with 3.1 but recently the OS (Cray Linux Env) 
was upgraded and so was PETSc (to 3.2).

Matt

I am attaching entire output.

Tabrez

---

stali at krakenpf2:~/meshes> which xterm
/usr/bin/xterm
stali at krakenpf2:~/meshes> aprun -n 1 ./defmod -f 
2d_point_load_dyn_abc.inp -on_error_attach_debugger
  Reading input ...
  Reading mesh data ...
  Forming [K] ...
  Forming [M] & [M]^-1 ...
  Applying constraints ...
  Forming RHS ...
  Setting up solver ...
  Solving ...
   Time Step 0
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 26164 on display 
:0.0 on machine nid03538
Unable to start debugger in xterm: No such file or directory
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
_pmii_daemon(SIGCHLD): [NID 03538] [c12-3c2s4n2] [Mon Apr  2 22:50:09 
2012] PE 0 exit signal Aborted
Application 134950 exit codes: 134
Application 134950 resources: utime ~1s, stime ~0s

On 04/02/2012 09:04 PM, Matthew Knepley wrote:
> On Mon, Apr 2, 2012 at 8:57 PM, Barry Smith <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>     On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
>
>     > Hello
>     >
>     > I am trying to debug a program using the switch
>     '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc
>     3.2.00 is unable to start the debugger in xterm (see text below).
>     But xterm is installed. What am I doing wrong?
>     >
>     > Btw the segfault happens during a call to MatMult but only with
>     vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers
>     only and _not_ with CRAY or GNU compilers.
>
>       My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and
>     just built it yourself so you can get real work done instead of
>     trying to debug their mess.   I promise the vendor one is not like
>     a billion times faster or anything.
>
>
> If you want to justify this to anyone (like a funder), just run both 
> on ex5 for a large size and look at the flops on MatMult. That
> is probably your dominant cost (or your PC).
>
>    Matt
>
>
>       Barry
>
>
>
>     >
>     > I also dont get the segfault if I build PETSc 3.2-p7 myself with
>     PGI/Intel compilers.
>     >
>     > Any ideas on how to diagnose the problem? Unfortunately I cannot
>     seem to run valgrind on this particular machine.
>     >
>     > Thanks in advance.
>     >
>     > Tabrez
>     >
>     > ---
>     >
>     > stali at krakenpf1:~/meshes> which xterm
>     > /usr/bin/xterm
>     > stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f
>     2d_point_load_dyn_abc.inp -on_error_attach_debugger
>     > ...
>     > ...
>     > ...
>     > [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     > [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     > [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
>     to find memory corruption errors
>     > [0]PETSC ERROR: configure using --with-debugging=yes, recompile,
>     link, and run
>     > [0]PETSC ERROR: to get more information on the crash.
>     > [0]PETSC ERROR: User provided function() line 0 in unknown
>     directory unknown file
>     > [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on
>     display localhost:20.0 on machine nid10649
>     > Unable to start debugger in xterm: No such file or directory
>     > aborting job:
>     > application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
>     > _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2
>     13:06:48 2012] PE 0 exit signal Aborted
>     > Application 133198 exit codes: 134
>     > Application 133198 resources: utime ~1s, stime ~0s
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/2b63becd/attachment.htm>

From stali at geology.wisc.edu  Mon Apr  2 22:04:00 2012
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Mon, 02 Apr 2012 22:04:00 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <alpine.LFD.2.02.1204022101020.2087@asterix>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<alpine.LFD.2.02.1204022101020.2087@asterix>
Message-ID: <4F7A68A0.5070805@geology.wisc.edu>

Satish

Things work fine on my linux machine (and other linux clusters) and 
valgrind shows no error. Unfortunately Totalview (GUI starts fine on the 
node) gives me a licensing error on the Cray.

I will continue to explore.

Thanks
Tabrez

On 04/02/2012 09:05 PM, Satish Balay wrote:
> Sounds like a Cray machine.
>
> start_in_debugger is useful for debugging on workstations [or
> clusters] etc where there is some control on X11 tunnels. Also
> 'xterm','gdb' or similar debugger should be available on the compute
> nodes [along with a x/ssh tunnel].
>
> On a cray - you are better off looking for a parallel debugger. Don't
> know if cray has one available.
>
> Wrt debugging - you might want to run your code with valgrind on a
> linux box..
>
> Satish
>
> On Mon, 2 Apr 2012, Tabrez Ali wrote:
>
>> Hello
>>
>> I am trying to debug a program using the switch '-on_error_attach_debugger'
>> but  the vendor/sysadmin built PETSc 3.2.00 is unable to start the debugger in
>> xterm (see text below). But xterm is installed. What am I doing wrong?
>>
>> Btw the segfault happens during a call to MatMult but only with
>> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and _not_
>> with CRAY or GNU compilers.
>>
>> I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel
>> compilers.
>>
>> Any ideas on how to diagnose the problem? Unfortunately I cannot seem to run
>> valgrind on this particular machine.
>>
>> Thanks in advance.
>>
>> Tabrez
>>
>> ---
>>
>> stali at krakenpf1:~/meshes>  which xterm
>> /usr/bin/xterm
>> stali at krakenpf1:~/meshes>  aprun -n 1 ./defmod -f 2d_point_load_dyn_abc.inp
>> -on_error_attach_debugger
>> ...
>> ...
>> ...
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably
>> memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
>> memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown
>> file
>> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display
>> localhost:20.0 on machine nid10649
>> Unable to start debugger in xterm: No such file or directory
>> aborting job:
>> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
>> _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 2012] PE
>> 0 exit signal Aborted
>> Application 133198 exit codes: 134
>> Application 133198 resources: utime ~1s, stime ~0s
>>


From knepley at gmail.com  Mon Apr  2 22:12:03 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 22:12:03 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7A65FF.1030109@geology.wisc.edu>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
	<CAMYG4G=H_GPVjtWqwJQ=v4y18acj5W1Kwaf7k3pO6mVvB=-83Q@mail.gmail.com>
	<4F7A65FF.1030109@geology.wisc.edu>
Message-ID: <CAMYG4GkJui02+P6AB05U9sKedUGyg_p2j0K4ZMY22QCi86gOCg@mail.gmail.com>

On Mon, Apr 2, 2012 at 9:52 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:

>  Matt/Barry
>
> My intention was to make sure that the code is bug free and since PETSc
> was pre-installed on the cluster with various compilers it was easier to
> test quickly rather than build all combinations myself. Performance is of
> absolutely no concern.
>
> Things were working fine with 3.1 but recently the OS (Cray Linux Env) was
> upgraded and so was PETSc (to 3.2).
>
> Matt
>
> I am attaching entire output.
>

> Unable to start debugger in xterm: No such file or directory
> aborting job:

xterm is not in the path.

   Matt


> Tabrez
>
> ---
>
> stali at krakenpf2:~/meshes> which xterm
> /usr/bin/xterm
> stali at krakenpf2:~/meshes> aprun -n 1 ./defmod -f
> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
>  Reading input ...
>  Reading mesh data ...
>  Forming [K] ...
>  Forming [M] & [M]^-1 ...
>  Applying constraints ...
>  Forming RHS ...
>  Setting up solver ...
>  Solving ...
>   Time Step 0
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 26164 on display
> :0.0 on machine nid03538
> Unable to start debugger in xterm: No such file or directory
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> _pmii_daemon(SIGCHLD): [NID 03538] [c12-3c2s4n2] [Mon Apr  2 22:50:09
> 2012] PE 0 exit signal Aborted
> Application 134950 exit codes: 134
> Application 134950 resources: utime ~1s, stime ~0s
>
> On 04/02/2012 09:04 PM, Matthew Knepley wrote:
>
> On Mon, Apr 2, 2012 at 8:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
>>
>> > Hello
>> >
>> > I am trying to debug a program using the switch
>> '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is
>> unable to start the debugger in xterm (see text below). But xterm is
>> installed. What am I doing wrong?
>> >
>> > Btw the segfault happens during a call to MatMult but only with
>> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and
>> _not_ with CRAY or GNU compilers.
>>
>>    My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just
>> built it yourself so you can get real work done instead of trying to debug
>> their mess.   I promise the vendor one is not like a billion times faster
>> or anything.
>
>
>  If you want to justify this to anyone (like a funder), just run both on
> ex5 for a large size and look at the flops on MatMult. That
> is probably your dominant cost (or your PC).
>
>     Matt
>
>
>>
>>   Barry
>>
>>
>>
>> >
>> > I also dont get the segfault if I build PETSc 3.2-p7 myself with
>> PGI/Intel compilers.
>> >
>> > Any ideas on how to diagnose the problem? Unfortunately I cannot seem
>> to run valgrind on this particular machine.
>> >
>> > Thanks in advance.
>> >
>> > Tabrez
>> >
>> > ---
>> >
>> > stali at krakenpf1:~/meshes> which xterm
>> > /usr/bin/xterm
>> > stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f
>> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
>> > ...
>> > ...
>> > ...
>> > [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> > [0]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> > [0]PETSC ERROR: to get more information on the crash.
>> > [0]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> > [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on
>> display localhost:20.0 on machine nid10649
>> > Unable to start debugger in xterm: No such file or directory
>> > aborting job:
>> > application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
>> > _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48
>> 2012] PE 0 exit signal Aborted
>> > Application 133198 exit codes: 134
>> > Application 133198 resources: utime ~1s, stime ~0s
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/93e0a49c/attachment-0001.htm>

From knepley at gmail.com  Mon Apr  2 22:11:19 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 2 Apr 2012 22:11:19 -0500
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7A65FF.1030109@geology.wisc.edu>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
	<CAMYG4G=H_GPVjtWqwJQ=v4y18acj5W1Kwaf7k3pO6mVvB=-83Q@mail.gmail.com>
	<4F7A65FF.1030109@geology.wisc.edu>
Message-ID: <CAMYG4Gm6ExJWTereM7fpGDohsCLS+XGdWMi3PLG-p4H_J43bVA@mail.gmail.com>

On Mon, Apr 2, 2012 at 9:52 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:

>  Matt/Barry
>
> My intention was to make sure that the code is bug free and since PETSc
> was pre-installed on the cluster with various compilers it was easier to
> test quickly rather than build all combinations myself. Performance is of
> absolutely no concern.
>
> Things were working fine with 3.1 but recently the OS (Cray Linux Env) was
> upgraded and so was PETSc (to 3.2).
>
> Matt
>
> I am attaching entire output.
>

> Unable to start debugger in xterm: No such file or directory
> aborting job:

xterm is not in the path.

   Matt


> Tabrez
>
> ---
>
> stali at krakenpf2:~/meshes> which xterm
> /usr/bin/xterm
> stali at krakenpf2:~/meshes> aprun -n 1 ./defmod -f
> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
>  Reading input ...
>  Reading mesh data ...
>  Forming [K] ...
>  Forming [M] & [M]^-1 ...
>  Applying constraints ...
>  Forming RHS ...
>  Setting up solver ...
>  Solving ...
>   Time Step 0
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 26164 on display
> :0.0 on machine nid03538
> Unable to start debugger in xterm: No such file or directory
> aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> _pmii_daemon(SIGCHLD): [NID 03538] [c12-3c2s4n2] [Mon Apr  2 22:50:09
> 2012] PE 0 exit signal Aborted
> Application 134950 exit codes: 134
> Application 134950 resources: utime ~1s, stime ~0s
>
> On 04/02/2012 09:04 PM, Matthew Knepley wrote:
>
> On Mon, Apr 2, 2012 at 8:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
>>
>> > Hello
>> >
>> > I am trying to debug a program using the switch
>> '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is
>> unable to start the debugger in xterm (see text below). But xterm is
>> installed. What am I doing wrong?
>> >
>> > Btw the segfault happens during a call to MatMult but only with
>> vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and
>> _not_ with CRAY or GNU compilers.
>>
>>    My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just
>> built it yourself so you can get real work done instead of trying to debug
>> their mess.   I promise the vendor one is not like a billion times faster
>> or anything.
>
>
>  If you want to justify this to anyone (like a funder), just run both on
> ex5 for a large size and look at the flops on MatMult. That
> is probably your dominant cost (or your PC).
>
>     Matt
>
>
>>
>>   Barry
>>
>>
>>
>> >
>> > I also dont get the segfault if I build PETSc 3.2-p7 myself with
>> PGI/Intel compilers.
>> >
>> > Any ideas on how to diagnose the problem? Unfortunately I cannot seem
>> to run valgrind on this particular machine.
>> >
>> > Thanks in advance.
>> >
>> > Tabrez
>> >
>> > ---
>> >
>> > stali at krakenpf1:~/meshes> which xterm
>> > /usr/bin/xterm
>> > stali at krakenpf1:~/meshes> aprun -n 1 ./defmod -f
>> 2d_point_load_dyn_abc.inp -on_error_attach_debugger
>> > ...
>> > ...
>> > ...
>> > [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> > [0]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> > [0]PETSC ERROR: to get more information on the crash.
>> > [0]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> > [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on
>> display localhost:20.0 on machine nid10649
>> > Unable to start debugger in xterm: No such file or directory
>> > aborting job:
>> > application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
>> > _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48
>> 2012] PE 0 exit signal Aborted
>> > Application 133198 exit codes: 134
>> > Application 133198 resources: utime ~1s, stime ~0s
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/dcbe3ead/attachment.htm>

From popov at uni-mainz.de  Tue Apr  3 02:03:57 2012
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 3 Apr 2012 09:03:57 +0200
Subject: [petsc-users] Debugger question
In-Reply-To: <F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
Message-ID: <4F7AA0DD.6000109@uni-mainz.de>

I support 100% what Barry said. Just get the work done. Cray and IBM 
Linux systems do not support ALL the systems calls that PETSc uses. So 
it's always kind of problem to purge manually petscconf.h in between of 
"configure" and "make" on their machines. I wander how you could install 
any PETSc without modifying petscconf.h. If you just don't care, usually 
you get segfaults right at PetscInitialize() step. Literally it means, 
there is no way you can debug anything, they should reinstall PETSc, 
keeping in mind the exact list of system calls they support, and PETSc 
requirements.

By the way, the times when GNU compilers were "order of magnitude" 
slower than "vendor compilers" have passed long ago. Just give it a try, 
compile some simple computationally intensive code with gcc and 
something from "vendor" with aggressive optimization, and check 
execution time on a large data set. I'm sure you'll be surprised.

Cheers,
Anton

On 4/3/12 3:57 AM, Barry Smith wrote:
> On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
>
>> Hello
>>
>> I am trying to debug a program using the switch '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is unable to start the debugger in xterm (see text below). But xterm is installed. What am I doing wrong?
>>
>> Btw the segfault happens during a call to MatMult but only with vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and _not_ with CRAY or GNU compilers.
>     My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just built it yourself so you can get real work done instead of trying to debug their mess.   I promise the vendor one is not like a billion times faster or anything.
>
>     Barry
>
>
>
>> I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel compilers.
>>
>> Any ideas on how to diagnose the problem? Unfortunately I cannot seem to run valgrind on this particular machine.
>>
>> Thanks in advance.
>>
>> Tabrez
>>
>> ---
>>
>> stali at krakenpf1:~/meshes>  which xterm
>> /usr/bin/xterm
>> stali at krakenpf1:~/meshes>  aprun -n 1 ./defmod -f 2d_point_load_dyn_abc.inp -on_error_attach_debugger
>> ...
>> ...
>> ...
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display localhost:20.0 on machine nid10649
>> Unable to start debugger in xterm: No such file or directory
>> aborting job:
>> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
>> _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 2012] PE 0 exit signal Aborted
>> Application 133198 exit codes: 134
>> Application 133198 resources: utime ~1s, stime ~0s

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/0433a79d/attachment.htm>

From niko.karin at gmail.com  Tue Apr  3 03:02:21 2012
From: niko.karin at gmail.com (Karin&NiKo)
Date: Tue, 3 Apr 2012 10:02:21 +0200
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
	<CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
Message-ID: <CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>

Dear all,

I am not sure if the definition of the near null space of the operator
suffice in order to get an optimal preconditioner for linear elasticity.
Considering ML, one must also set the "num_PDEs" attribute, mustn't he? But
it seems to me that this attribute cannot be set in the PETSc interface.
Am I wrong?

Regards,
Nicolas

2012/4/3 Jed Brown <jedbrown at mcs.anl.gov>

> Well, the KSP can diagonally scale it and we should fix up the near null
> space. The PC should not need to find out about diagonal scaling. (My
> opinion.)
> On Apr 2, 2012 4:50 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:
>
>> Oh, OK, so you do not explicitly scale the matrix good.  We had a user
>> (John) recently that was using diagonal scaling and was getting failure
>> with GAMG and then GAMG worked and he was no longer using diagonal scaling
>> so I was thinking this was the problem.
>>
>> Mark
>>
>> On Apr 2, 2012, at 7:40 PM, Barry Smith wrote:
>>
>> >
>> >  Mark,
>> >
>> >   Who would be doing the "diagonal scaling". Under normal conditions we
>> never want to be passing to any preconditioner a diagonally scaled matrix
>> (for exactly the issue you point out) so why are you worried about that
>> case?
>> >
>> >   Barry
>> >
>> > On Apr 2, 2012, at 6:40 PM, Mark F. Adams wrote:
>> >
>> >>
>> >> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
>> >>
>> >>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> >>> The block size is used if you don't provide any other information.
>> The preferred approach with petsc-dev is to use MatSetNearNullSpace().
>> (I'll make sure this is currently working with ML and reply to this
>> message.)
>> >>>
>> >>> I pushed ML support for user-provided null spaces to petsc-dev. Call
>> MatSetNearNullSpace() to provide the information. You can use
>> MatNullSpaceCreateRigidBody() if you have nodal coordinates.
>> >>>
>> >>>
>> >>> Mark, what is the best way to make PCGAMG use this interface?
>> >>
>> >> It looks like you are adding this to the Mat, so I can check to see if
>> coordinates have been set and if not check the matrix.  This should be easy
>> to do; I'll look at the ML code to clone the API.
>> >>
>> >> One thing to keep in mind is that diagonal scaling breaks the null
>> space (ie, the rigid body modes have to be scaled appropriately).  Who owns
>> the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace
>> to do this, otherwise we should think of a good way to deal with this.  It
>> is very error prone to not do the right thing here, we should at least
>> throw an error.
>> >>
>> >>
>> >
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/b89f0014/attachment.htm>

From khalid_eee at yahoo.com  Tue Apr  3 03:46:13 2012
From: khalid_eee at yahoo.com (khalid ashraf)
Date: Tue, 3 Apr 2012 01:46:13 -0700 (PDT)
Subject: [petsc-users] VecSetValues
In-Reply-To: <CAMYG4GkVuCvqiZjYCwTWoQs3RUPgRGZ0sdHgkSAR_CCEhAZH8g@mail.gmail.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
	<CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
	<1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>
	<CAMYG4GkVuCvqiZjYCwTWoQs3RUPgRGZ0sdHgkSAR_CCEhAZH8g@mail.gmail.com>
Message-ID: <1333442773.82597.YahooMailNeo@web140413.mail.bf1.yahoo.com>

Hi Matt, I ran the program on two processors. I am trying to enter value into a?
vector associated with DA. I am using VecSetValues since I may need to assign nonlocal?
values to the vector. However, I find that on 2 procs, a general parallel vector works but
the DAVector indexing is not working. Do you see any problem with the DA code ?

Thanks.

Khalid ? ?

THIS WORKS: 30 is written at index 5
-------------------------------
ppy=5; ppx=30;
VecCreate(PETSC_COMM_WORLD,&testVec);
VecSetFromOptions(testVec);
VecSetSizes(testVec,PETSC_DECIDE,384);

VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
VecAssemblyBegin(testVec);
VecAssemblyEnd(testVec);
=====================================

THIS DOES NOT WORK: puts 30 at index 9 instead of 5 when run on two procs.
---------------------------------
ppy=5; ppx=30;

ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,8,8,6,
?? ? ? ? ? ? ? ? ? ?PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
?? ? ? ? ? ? ? ? ? ?&appctx.da);CHKERRQ(ierr);
?ierr = DACreateGlobalVector(da,&testVec);CHKERRQ(ierr)

VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
VecAssemblyBegin(testVec);

VecAssemblyEnd(testVec);

________________________________
 From: Matthew Knepley <knepley at gmail.com>
To: khalid ashraf <khalid_eee at yahoo.com> 
Cc: PETSc users list <petsc-users at mcs.anl.gov> 
Sent: Monday, April 2, 2012 3:08 PM
Subject: Re: [petsc-users] transfer vector data diagonally on DA
 

On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <khalid_eee at yahoo.com> wrote:

Thanks Matt. It works now.?
>
>
>I have another problem. I am writing this program.?

1) Always test on 2 procs once

2) I have no idea what you actually want to do. However, it really simple to just print out
? ? the indices you are using in row[] and the values in bodyFx[] and check that VecSetValues()
? ? is working as you expect.

? ?Matt
?
?
>?for (k=zs; k<zs+zm; k++) {
>??for (j=ys; j<ys+ym; j++) {
>??for (i=xs; i<xs+xm; i++) {
>?if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
>{
>bodyFx[0]=1;
>bodyFx[1]=-0.3;
>bodyFx[2]=-0.3;
>bodyFx[3]=-0.3;
>row[0]=k*mx*my+j*mx+i;
>row[1]=k*mx*my+j*mx+i+1;
>row[2]=k*mx*my+(j+1)*mx+i;
>row[3]=(k+1)*mx*my+j*mx+i;
>VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);
>}
>VecAssemblyBegin(fx_test);
>VecAssemblyEnd(fx_test);
>
>
>Output: Print fx_test
>
>
>?Here fx_test is a global vector. This program gives correct output on 1 proc. but on 4 processors,
>the program runs but the outputs are wrong.
>
>
>I also tried
>?for (k=zs; k<zs+zm; k++) {
>??for (j=ys; j<ys+ym; j++) {
>??for (i=xs; i<xs+xm; i++) {
>?if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
>{
>bodyFx[0]=1;
>bodyFx[1]=-0.3;
>bodyFx[2]=-0.3;
>bodyFx[3]=-0.3;
>?fx1_localptr[k][j][i+1]+=bodyFx[4];
>?fx1_localptr[k][j+1][i]+=bodyFx[3];
>?fx1_localptr[k+1][j][i]+=bodyFx[1];
>?fx1_localptr[k][j][i]+=bodyFx[0];
>}
>Here, fx1_localptr is an array from a local vector that is communicated from the global vector fx_test.
>In this case as well, I get similar errors.?
>
>
>
>Thanks.
>Khalid
>
>
>Some final numbers of the vector with 8*8*6 grid, 4 proc, ?
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>1.0000000000000009e-01
>1.0000000000000009e-01
>1.0000000000000009e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>0.0000000000000000e+00
>4.0000000000000002e-01
>1.0000000000000009e-01
>1.0000000000000009e-01
>1.0000000000000009e-01
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>1.0000000000000009e-01
>1.0000000000000009e-01
>1.0000000000000009e-01
>-2.9999999999999999e-01
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>
>
>
>
>Some final numbers of the vector with 8*8*6 grid, 1 proc, ?
>
>0.0000000000000000e+00
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>0.0000000000000000e+00
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>0.0000000000000000e+00
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>-2.9999999999999999e-01
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>0.0000000000000000e+00
>
>
>?? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
>
>
>
>________________________________
> From: Matthew Knepley <knepley at gmail.com>
>To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
>Sent: Monday, April 2, 2012 5:41 AM
>Subject: Re: [petsc-users] transfer vector data diagonally on DA
> 
>
>On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com> wrote:
>
>Hi Jed,
>>
>>
>>I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h"
>>and DACreate3D.?
>>unew_localptr[][][] is from a global vector using VecGetArray
>>u_localptr[][][] is from a local vector communicated from a global vector.
>>I tried?
>>unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
>>
>>but it gives Segmentation Violation error.?
>
>
>I would guess (because you provide almost no information about what you are doing), that
>this is a domain on the edge. If you truly mean to have a periodic domain, you must set that in the
>creation call. Then the local vector will also ghost regions outside the domain boundary.
>
>
>? ?Matt
>?
>I also tried?unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but?
>>unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or?unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
>>does not work. I am running the program on4 processors.
>>
>>
>>
>>Thanks.
>>
>>
>>
>>
>>??ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
>>?? ? ? ? ? ? ? ? ? ?PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>>?? ? ? ? ? ? ? ? ? ?&appctx.da);CHKERRQ(ierr);
>>?for (k=zs; k<zs+zm; k++) {
>>??for (j=ys; j<ys+ym; j++) {
>>??for (i=xs; i<xs+xm; i++) {
>>
>>
>>??if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
>>
>>
>>??unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
>>
>>}}}
>>
>>________________________________
>>
>>From: Jed Brown <jedbrown at mcs.anl.gov>
>>To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
>>Sent: Sunday, April 1, 2012 10:07 PM
>>Subject: Re: [petsc-users] transfer vector data diagonally on DA
>>
>>
>>
>>On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:
>>
>>I want to transfer vector data diagonally in the DA grid like?
>>>?for (k=zs; k<zs+zm; k++) {
>>>??for (j=ys; j<ys+ym; j++) {
>>>??for (i=xs; i<xs+xm; i++) {
>>>if(i!=mx-1 || j!=my-1 || k!=mz-1){
>>>u_new[k+1][j+1][i+1]=u[k][j][i];}
>>>}}}
>>>
>>>
>>>Could you please suggest the best way to do it minimizing interprocessor assignments.
>>
>>Both are on the same DMDA?
>>
>>
>>Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].
>>
>>
>
>
>
>-- 
>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>-- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/5a8cceba/attachment-0001.htm>

From bsmith at mcs.anl.gov  Tue Apr  3 07:40:29 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Apr 2012 07:40:29 -0500
Subject: [petsc-users] VecSetValues
In-Reply-To: <1333442773.82597.YahooMailNeo@web140413.mail.bf1.yahoo.com>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
	<CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
	<1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>
	<CAMYG4GkVuCvqiZjYCwTWoQs3RUPgRGZ0sdHgkSAR_CCEhAZH8g@mail.gmail.com>
	<1333442773.82597.YahooMailNeo@web140413.mail.bf1.yahoo.com>
Message-ID: <C7498857-607E-4F8E-970A-599930559C9C@mcs.anl.gov>


   I think you are mis-understanding the global ordering when using DA vectors in parallel. You might consider using VecSetValuesLocal() or VecSetValuesStencil() instead of VecSetValues(). Read up on the users manual and examples about PETSc global ordering.

   Barry

On Apr 3, 2012, at 3:46 AM, khalid ashraf wrote:

> Hi Matt, I ran the program on two processors. I am trying to enter value into a 
> vector associated with DA. I am using VecSetValues since I may need to assign nonlocal 
> values to the vector. However, I find that on 2 procs, a general parallel vector works but
> the DAVector indexing is not working. Do you see any problem with the DA code ?
> 
> Thanks.
> 
> Khalid    
> 
> THIS WORKS: 30 is written at index 5
> -------------------------------
> ppy=5; ppx=30;
> VecCreate(PETSC_COMM_WORLD,&testVec);
> VecSetFromOptions(testVec);
> VecSetSizes(testVec,PETSC_DECIDE,384);
> 
> VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
> VecAssemblyBegin(testVec);
> VecAssemblyEnd(testVec);
> =====================================
> 
> THIS DOES NOT WORK: puts 30 at index 9 instead of 5 when run on two procs.
> ---------------------------------
> ppy=5; ppx=30;
> ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,8,8,6,
>                     PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>                     &appctx.da);CHKERRQ(ierr);
>  ierr = DACreateGlobalVector(da,&testVec);CHKERRQ(ierr)
> VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
> VecAssemblyBegin(testVec);
> VecAssemblyEnd(testVec);
> 
> From: Matthew Knepley <knepley at gmail.com>
> To: khalid ashraf <khalid_eee at yahoo.com> 
> Cc: PETSc users list <petsc-users at mcs.anl.gov> 
> Sent: Monday, April 2, 2012 3:08 PM
> Subject: Re: [petsc-users] transfer vector data diagonally on DA
> 
> On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <khalid_eee at yahoo.com> wrote:
> Thanks Matt. It works now. 
> 
> I have another problem. I am writing this program. 
> 
> 1) Always test on 2 procs once
> 
> 2) I have no idea what you actually want to do. However, it really simple to just print out
>     the indices you are using in row[] and the values in bodyFx[] and check that VecSetValues()
>     is working as you expect.
> 
>    Matt
>  
>  
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> {
> bodyFx[0]=1;
> bodyFx[1]=-0.3;
> bodyFx[2]=-0.3;
> bodyFx[3]=-0.3;
> row[0]=k*mx*my+j*mx+i;
> row[1]=k*mx*my+j*mx+i+1;
> row[2]=k*mx*my+(j+1)*mx+i;
> row[3]=(k+1)*mx*my+j*mx+i;
> VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);
> }
> VecAssemblyBegin(fx_test);
> VecAssemblyEnd(fx_test);
> 
> Output: Print fx_test
> 
>  Here fx_test is a global vector. This program gives correct output on 1 proc. but on 4 processors,
> the program runs but the outputs are wrong.
> 
> I also tried
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
>  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> {
> bodyFx[0]=1;
> bodyFx[1]=-0.3;
> bodyFx[2]=-0.3;
> bodyFx[3]=-0.3;
>  fx1_localptr[k][j][i+1]+=bodyFx[4];
>  fx1_localptr[k][j+1][i]+=bodyFx[3];
>  fx1_localptr[k+1][j][i]+=bodyFx[1];
>  fx1_localptr[k][j][i]+=bodyFx[0];
> }
> Here, fx1_localptr is an array from a local vector that is communicated from the global vector fx_test.
> In this case as well, I get similar errors. 
> 
> Thanks.
> Khalid
> 
> Some final numbers of the vector with 8*8*6 grid, 4 proc,  
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 4.0000000000000002e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> 1.0000000000000009e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 
> 
> Some final numbers of the vector with 8*8*6 grid, 1 proc,  
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> -2.9999999999999999e-01
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 0.0000000000000000e+00
> 
>                                
> 
> From: Matthew Knepley <knepley at gmail.com>
> To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
> Sent: Monday, April 2, 2012 5:41 AM
> Subject: Re: [petsc-users] transfer vector data diagonally on DA
> 
> On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com> wrote:
> Hi Jed,
> 
> I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h"
> and DACreate3D. 
> unew_localptr[][][] is from a global vector using VecGetArray
> u_localptr[][][] is from a local vector communicated from a global vector.
> I tried 
> unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
> but it gives Segmentation Violation error. 
> 
> I would guess (because you provide almost no information about what you are doing), that
> this is a domain on the edge. If you truly mean to have a periodic domain, you must set that in the
> creation call. Then the local vector will also ghost regions outside the domain boundary.
> 
>    Matt
>  
> I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but 
> unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
> does not work. I am running the program on4 processors.
> 
> Thanks.
> 
> 
>   ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
>                     PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
>                     &appctx.da);CHKERRQ(ierr);
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
> 
>   if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
> 
>   unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
> }}}
> From: Jed Brown <jedbrown at mcs.anl.gov>
> To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
> Sent: Sunday, April 1, 2012 10:07 PM
> Subject: Re: [petsc-users] transfer vector data diagonally on DA
> 
> On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com> wrote:
> I want to transfer vector data diagonally in the DA grid like 
>  for (k=zs; k<zs+zm; k++) {
>   for (j=ys; j<ys+ym; j++) {
>   for (i=xs; i<xs+xm; i++) {
> if(i!=mx-1 || j!=my-1 || k!=mz-1){
> u_new[k+1][j+1][i+1]=u[k][j][i];}
> }}}
> 
> Could you please suggest the best way to do it minimizing interprocessor assignments.
> 
> Both are on the same DMDA?
> 
> Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 


From jedbrown at mcs.anl.gov  Tue Apr  3 08:21:47 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 3 Apr 2012 06:21:47 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
	<CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
	<CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
Message-ID: <CAM9tzS=9A30Y_D8TUH4F3zE2JpbtCUSPHPJeVhvHuPHgX5M5yw@mail.gmail.com>

On Tue, Apr 3, 2012 at 01:02, Karin&NiKo <niko.karin at gmail.com> wrote:

> I am not sure if the definition of the near null space of the operator
> suffice in order to get an optimal preconditioner for linear elasticity.
> Considering ML, one must also set the "num_PDEs" attribute, mustn't he?
> But it seems to me that this attribute cannot be set in the PETSc interface.
> Am I wrong?
>

It is set using the "block size" of the matrix (MatSetBlockSize()). But
that is only part of the game. For robustness, you should also provide null
space information. Here is an example of using a coordinate Vec to create a
MatNullSpace defining the rigid body modes and passing it along (from
src/ksp/ksp/examples/tutorials/ex49.c).

  ierr = DMDAGetCoordinates(elas_da,&vel_coords);CHKERRQ(ierr);
  ierr = MatNullSpaceCreateRigidBody(vel_coords,&matnull);CHKERRQ(ierr);
  ierr = MatSetNearNullSpace(A,matnull);CHKERRQ(ierr);
  ierr = MatNullSpaceDestroy(&matnull);CHKERRQ(ierr);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/32ae6df8/attachment.htm>

From mark.adams at columbia.edu  Tue Apr  3 08:39:31 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Tue, 3 Apr 2012 09:39:31 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
	<CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
	<CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
Message-ID: <C828B841-F00D-406C-AA3C-D42A55995228@columbia.edu>

The null space (6 for 3D elasticity) and the block size (3 for 3D) is all you need for ML (and GAMG).

Mark

On Apr 3, 2012, at 4:02 AM, Karin&NiKo wrote:

> Dear all,
> 
> I am not sure if the definition of the near null space of the operator suffice in order to get an optimal preconditioner for linear elasticity. 
> Considering ML, one must also set the "num_PDEs" attribute, mustn't he? But it seems to me that this attribute cannot be set in the PETSc interface.
> Am I wrong?
> 
> Regards,
> Nicolas
> 
> 2012/4/3 Jed Brown <jedbrown at mcs.anl.gov>
> Well, the KSP can diagonally scale it and we should fix up the near null space. The PC should not need to find out about diagonal scaling. (My opinion.)
> 
> On Apr 2, 2012 4:50 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:
> Oh, OK, so you do not explicitly scale the matrix good.  We had a user (John) recently that was using diagonal scaling and was getting failure with GAMG and then GAMG worked and he was no longer using diagonal scaling so I was thinking this was the problem.
> 
> Mark
> 
> On Apr 2, 2012, at 7:40 PM, Barry Smith wrote:
> 
> >
> >  Mark,
> >
> >   Who would be doing the "diagonal scaling". Under normal conditions we never want to be passing to any preconditioner a diagonally scaled matrix (for exactly the issue you point out) so why are you worried about that case?
> >
> >   Barry
> >
> > On Apr 2, 2012, at 6:40 PM, Mark F. Adams wrote:
> >
> >>
> >> On Apr 2, 2012, at 7:24 PM, Jed Brown wrote:
> >>
> >>> On Mon, Apr 2, 2012 at 07:44, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> >>> The block size is used if you don't provide any other information. The preferred approach with petsc-dev is to use MatSetNearNullSpace(). (I'll make sure this is currently working with ML and reply to this message.)
> >>>
> >>> I pushed ML support for user-provided null spaces to petsc-dev. Call MatSetNearNullSpace() to provide the information. You can use MatNullSpaceCreateRigidBody() if you have nodal coordinates.
> >>>
> >>>
> >>> Mark, what is the best way to make PCGAMG use this interface?
> >>
> >> It looks like you are adding this to the Mat, so I can check to see if coordinates have been set and if not check the matrix.  This should be easy to do; I'll look at the ML code to clone the API.
> >>
> >> One thing to keep in mind is that diagonal scaling breaks the null space (ie, the rigid body modes have to be scaled appropriately).  Who owns the diagonal scaling? If it is Mat then we might want MatSetNearNullSpace to do this, otherwise we should think of a good way to deal with this.  It is very error prone to not do the right thing here, we should at least throw an error.
> >>
> >>
> >
> >
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/24881e71/attachment.htm>

From mark.adams at columbia.edu  Tue Apr  3 09:21:33 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Tue, 3 Apr 2012 10:21:33 -0400
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <CAM9tzS=9A30Y_D8TUH4F3zE2JpbtCUSPHPJeVhvHuPHgX5M5yw@mail.gmail.com>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
	<CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
	<CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
	<CAM9tzS=9A30Y_D8TUH4F3zE2JpbtCUSPHPJeVhvHuPHgX5M5yw@mail.gmail.com>
Message-ID: <8F0563B9-F2BE-4D62-91DD-E0CAF452846A@columbia.edu>

Jed,

I've added the appended code to get the null space from the Mat.  I did not see:

 ierr = MatNullSpaceDestroy(&matnull);CHKERRQ(ierr);

in your ML code ... shouldn't that be here?

Mark


  PetscFunctionBegin;
  ierr = MatGetNearNullSpace( a_A, &mnull ); CHKERRQ(ierr);
  if( !mnull ) {
    ierr = PCSetCoordinates_AGG( pc, -1, PETSC_NULL ); CHKERRQ(ierr);
  }
  else {
    PetscReal *nullvec;
    PetscBool has_const;
    PetscInt i,j,mlocal,nvec,bs;
    const Vec *vecs; const PetscScalar *v;
    ierr = MatGetLocalSize(a_A,&mlocal,PETSC_NULL);CHKERRQ(ierr);
    ierr = MatNullSpaceGetVecs(mnull,&has_const,&nvec,&vecs);CHKERRQ(ierr);
     ierr  = MatGetBlockSize( a_A, &bs );               CHKERRQ( ierr );
    ierr = PetscMalloc((nvec+!!has_const)*mlocal*sizeof *nullvec,&nullvec);CHKERRQ(ierr);
    if (has_const) for (i=0; i<mlocal; i++) nullvec[i] = 1.0;
    for (i=0; i<nvec; i++) {
      ierr = VecGetArrayRead(vecs[i],&v);CHKERRQ(ierr);
      for (j=0; j<mlocal; j++) nullvec[(i+!!has_const)*mlocal + j] = PetscRealPart(v[j]);
      ierr = VecRestoreArrayRead(vecs[i],&v);CHKERRQ(ierr);
    }
    pc_gamg->data = nullvec;
    pc_gamg->data_cell_cols = (nvec+!!has_const);
    pc_gamg->data_cell_rows = bs;
  }
  PetscFunctionReturn(0);
}


On Apr 3, 2012, at 9:21 AM, Jed Brown wrote:

> On Tue, Apr 3, 2012 at 01:02, Karin&NiKo <niko.karin at gmail.com> wrote:
> I am not sure if the definition of the near null space of the operator suffice in order to get an optimal preconditioner for linear elasticity. 
> Considering ML, one must also set the "num_PDEs" attribute, mustn't he? But it seems to me that this attribute cannot be set in the PETSc interface.
> Am I wrong?
> 
> It is set using the "block size" of the matrix (MatSetBlockSize()). But that is only part of the game. For robustness, you should also provide null space information. Here is an example of using a coordinate Vec to create a MatNullSpace defining the rigid body modes and passing it along (from src/ksp/ksp/examples/tutorials/ex49.c).
> 
>   ierr = DMDAGetCoordinates(elas_da,&vel_coords);CHKERRQ(ierr);
>   ierr = MatNullSpaceCreateRigidBody(vel_coords,&matnull);CHKERRQ(ierr);
>   ierr = MatSetNearNullSpace(A,matnull);CHKERRQ(ierr);
>   ierr = MatNullSpaceDestroy(&matnull);CHKERRQ(ierr);
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/6ff7e507/attachment.htm>

From jedbrown at mcs.anl.gov  Tue Apr  3 09:30:08 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 3 Apr 2012 07:30:08 -0700
Subject: [petsc-users] ML and linear elasticity
In-Reply-To: <8F0563B9-F2BE-4D62-91DD-E0CAF452846A@columbia.edu>
References: <CA+gX-L-OGHj0cu548H5riyPDdj2LJvQ7V45xT6YS833QjYoFFg@mail.gmail.com>
	<CAM9tzSkGJtiibv=xUQDrpYn-6t0qQgTiWAkR27YnQo-i3amhiA@mail.gmail.com>
	<CAM9tzSnG9_yqNVn4KoUcFep4qeuJNq-JqS0GNnu9MO3UVd5EOw@mail.gmail.com>
	<7620EBE0-4204-487E-A132-278C71C4ECB4@columbia.edu>
	<8F574F31-F6CC-4064-AC0F-5CF2EA4B8452@mcs.anl.gov>
	<6D4FC434-FF4C-4027-9624-AC73B1CEA873@columbia.edu>
	<CAM9tzSkKcTTniJVO7bqi5qPRbfKbqB63P9coVfKnjdsg=WtVfg@mail.gmail.com>
	<CA+gX-L9Nebkut5pt09r_7SC2dUCmkmJfg4S95dFg6gO-pEqeZg@mail.gmail.com>
	<CAM9tzS=9A30Y_D8TUH4F3zE2JpbtCUSPHPJeVhvHuPHgX5M5yw@mail.gmail.com>
	<8F0563B9-F2BE-4D62-91DD-E0CAF452846A@columbia.edu>
Message-ID: <CAM9tzSnyKN+W6=RHXcHXsSSYipmnZjJXVpN7ephAepKjEE4RNw@mail.gmail.com>

On Tue, Apr 3, 2012 at 07:21, Mark F. Adams <mark.adams at columbia.edu> wrote:

> I've added the appended code to get the null space from the Mat.  I did
> not see:
>
>  ierr = MatNullSpaceDestroy(&matnull);CHKERRQ(ierr);
>
> in your ML code ... shouldn't that be here?
>

MatGetNearNullSpace() is a lightweight accessor, it doesn't increment the
reference count and you don't have to restore it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/98444c51/attachment-0001.htm>

From balay at mcs.anl.gov  Tue Apr  3 10:02:47 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 3 Apr 2012 10:02:47 -0500 (CDT)
Subject: [petsc-users] Debugger question
In-Reply-To: <4F7AA0DD.6000109@uni-mainz.de>
References: <4F7A4DF3.9090907@geology.wisc.edu>
	<F199E031-D839-4535-BA0A-A2BF43C54436@mcs.anl.gov>
	<4F7AA0DD.6000109@uni-mainz.de>
Message-ID: <alpine.LFD.2.02.1204031000490.2087@asterix>

On Tue, 3 Apr 2012, Anton Popov wrote:

> I support 100% what Barry said. Just get the work done. Cray and IBM Linux
> systems do not support ALL the systems calls that PETSc uses. So it's always
> kind of problem to purge manually petscconf.h in between of "configure" and
> "make" on their machines. I wander how you could install any PETSc without
> modifying petscconf.h.

You shouldn't have to manually modify petscconf.h on these machines. There
could still be some warnings at link time - but that shouldn't mean breakages
at tuntime.

> If you just don't care, usually you get segfaults right
> at PetscInitialize() step.

If this is the case - then it should be verifyiable with PETSc examples.

Satish

> Literally it means, there is no way you can debug
> anything, they should reinstall PETSc, keeping in mind the exact list of
> system calls they support, and PETSc requirements.
> 
> By the way, the times when GNU compilers were "order of magnitude" slower than
> "vendor compilers" have passed long ago. Just give it a try, compile some
> simple computationally intensive code with gcc and something from "vendor"
> with aggressive optimization, and check execution time on a large data set.
> I'm sure you'll be surprised.
> 
> Cheers,
> Anton
> 
> On 4/3/12 3:57 AM, Barry Smith wrote:
> > On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:
> > 
> > > Hello
> > > 
> > > I am trying to debug a program using the switch
> > > '-on_error_attach_debugger' but  the vendor/sysadmin built PETSc 3.2.00 is
> > > unable to start the debugger in xterm (see text below). But xterm is
> > > installed. What am I doing wrong?
> > > 
> > > Btw the segfault happens during a call to MatMult but only with
> > > vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and
> > > _not_ with CRAY or GNU compilers.
> >     My advice, blow off "the vendor/sysadmin supplied PETSc 3.2" and just
> > built it yourself so you can get real work done instead of trying to debug
> > their mess.   I promise the vendor one is not like a billion times faster or
> > anything.
> > 
> >     Barry
> > 
> > 
> > 
> > > I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel
> > > compilers.
> > > 
> > > Any ideas on how to diagnose the problem? Unfortunately I cannot seem to
> > > run valgrind on this particular machine.
> > > 
> > > Thanks in advance.
> > > 
> > > Tabrez
> > > 
> > > ---
> > > 
> > > stali at krakenpf1:~/meshes>  which xterm
> > > /usr/bin/xterm
> > > stali at krakenpf1:~/meshes>  aprun -n 1 ./defmod -f
> > > 2d_point_load_dyn_abc.inp -on_error_attach_debugger
> > > ...
> > > ...
> > > ...
> > > [0]PETSC ERROR:
> > > ------------------------------------------------------------------------
> > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > > probably memory access out of range
> > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > > [0]PETSC ERROR: or see
> > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> > > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> > > memory corruption errors
> > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> > > run
> > > [0]PETSC ERROR: to get more information on the crash.
> > > [0]PETSC ERROR: User provided function() line 0 in unknown directory
> > > unknown file
> > > [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display
> > > localhost:20.0 on machine nid10649
> > > Unable to start debugger in xterm: No such file or directory
> > > aborting job:
> > > application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
> > > _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48
> > > 2012] PE 0 exit signal Aborted
> > > Application 133198 exit codes: 134
> > > Application 133198 resources: utime ~1s, stime ~0s
> 
> 


From knepley at gmail.com  Tue Apr  3 10:01:28 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 3 Apr 2012 10:01:28 -0500
Subject: [petsc-users] VecSetValues
In-Reply-To: <C7498857-607E-4F8E-970A-599930559C9C@mcs.anl.gov>
References: <1333342915.87694.YahooMailNeo@web140401.mail.bf1.yahoo.com>
	<CAM9tzS=d7L_T_5ZKyF1hOCruURVofNJzp3hbNQnEv0CBNXVj-g@mail.gmail.com>
	<1333355786.15869.YahooMailNeo@web140406.mail.bf1.yahoo.com>
	<CAMYG4GmHBV_SUbvfdhbuqPfUrF4zD4YPeuiXnZ7kOSa5Dqn4Pg@mail.gmail.com>
	<1333403588.68476.YahooMailNeo@web140403.mail.bf1.yahoo.com>
	<CAMYG4GkVuCvqiZjYCwTWoQs3RUPgRGZ0sdHgkSAR_CCEhAZH8g@mail.gmail.com>
	<1333442773.82597.YahooMailNeo@web140413.mail.bf1.yahoo.com>
	<C7498857-607E-4F8E-970A-599930559C9C@mcs.anl.gov>
Message-ID: <CAMYG4G=XUGy6X_b16cg4sxUP-agrSBCTegGZfjy2B6p7zLtaKg@mail.gmail.com>

On Tue, Apr 3, 2012 at 7:40 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I think you are mis-understanding the global ordering when using DA
> vectors in parallel. You might consider using VecSetValuesLocal() or
> VecSetValuesStencil() instead of VecSetValues(). Read up on the users
> manual and examples about PETSc global ordering.


More explicitly, you do set the 5th value of the vector each time, but the
layout is different when created with the DA
since it is splitting the grid somehow. When you call VecView() it reorders
the Vec to match the grid before output.

    Matt


>
>   Barry
>
> On Apr 3, 2012, at 3:46 AM, khalid ashraf wrote:
>
> > Hi Matt, I ran the program on two processors. I am trying to enter value
> into a
> > vector associated with DA. I am using VecSetValues since I may need to
> assign nonlocal
> > values to the vector. However, I find that on 2 procs, a general
> parallel vector works but
> > the DAVector indexing is not working. Do you see any problem with the DA
> code ?
> >
> > Thanks.
> >
> > Khalid
> >
> > THIS WORKS: 30 is written at index 5
> > -------------------------------
> > ppy=5; ppx=30;
> > VecCreate(PETSC_COMM_WORLD,&testVec);
> > VecSetFromOptions(testVec);
> > VecSetSizes(testVec,PETSC_DECIDE,384);
> >
> > VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
> > VecAssemblyBegin(testVec);
> > VecAssemblyEnd(testVec);
> > =====================================
> >
> > THIS DOES NOT WORK: puts 30 at index 9 instead of 5 when run on two
> procs.
> > ---------------------------------
> > ppy=5; ppx=30;
> > ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,8,8,6,
> >
> PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
> >                     &appctx.da);CHKERRQ(ierr);
> >  ierr = DACreateGlobalVector(da,&testVec);CHKERRQ(ierr)
> > VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);
> > VecAssemblyBegin(testVec);
> > VecAssemblyEnd(testVec);
> >
> > From: Matthew Knepley <knepley at gmail.com>
> > To: khalid ashraf <khalid_eee at yahoo.com>
> > Cc: PETSc users list <petsc-users at mcs.anl.gov>
> > Sent: Monday, April 2, 2012 3:08 PM
> > Subject: Re: [petsc-users] transfer vector data diagonally on DA
> >
> > On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <khalid_eee at yahoo.com>
> wrote:
> > Thanks Matt. It works now.
> >
> > I have another problem. I am writing this program.
> >
> > 1) Always test on 2 procs once
> >
> > 2) I have no idea what you actually want to do. However, it really
> simple to just print out
> >     the indices you are using in row[] and the values in bodyFx[] and
> check that VecSetValues()
> >     is working as you expect.
> >
> >    Matt
> >
> >
> >  for (k=zs; k<zs+zm; k++) {
> >   for (j=ys; j<ys+ym; j++) {
> >   for (i=xs; i<xs+xm; i++) {
> >  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> > {
> > bodyFx[0]=1;
> > bodyFx[1]=-0.3;
> > bodyFx[2]=-0.3;
> > bodyFx[3]=-0.3;
> > row[0]=k*mx*my+j*mx+i;
> > row[1]=k*mx*my+j*mx+i+1;
> > row[2]=k*mx*my+(j+1)*mx+i;
> > row[3]=(k+1)*mx*my+j*mx+i;
> > VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);
> > }
> > VecAssemblyBegin(fx_test);
> > VecAssemblyEnd(fx_test);
> >
> > Output: Print fx_test
> >
> >  Here fx_test is a global vector. This program gives correct output on 1
> proc. but on 4 processors,
> > the program runs but the outputs are wrong.
> >
> > I also tried
> >  for (k=zs; k<zs+zm; k++) {
> >   for (j=ys; j<ys+ym; j++) {
> >   for (i=xs; i<xs+xm; i++) {
> >  if ( i!=(mx-1) || j!=my-1 || k!=mz-1)
> > {
> > bodyFx[0]=1;
> > bodyFx[1]=-0.3;
> > bodyFx[2]=-0.3;
> > bodyFx[3]=-0.3;
> >  fx1_localptr[k][j][i+1]+=bodyFx[4];
> >  fx1_localptr[k][j+1][i]+=bodyFx[3];
> >  fx1_localptr[k+1][j][i]+=bodyFx[1];
> >  fx1_localptr[k][j][i]+=bodyFx[0];
> > }
> > Here, fx1_localptr is an array from a local vector that is communicated
> from the global vector fx_test.
> > In this case as well, I get similar errors.
> >
> > Thanks.
> > Khalid
> >
> > Some final numbers of the vector with 8*8*6 grid, 4 proc,
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > 0.0000000000000000e+00
> > 4.0000000000000002e-01
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > 1.0000000000000009e-01
> > -2.9999999999999999e-01
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> >
> >
> > Some final numbers of the vector with 8*8*6 grid, 1 proc,
> > 0.0000000000000000e+00
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > 0.0000000000000000e+00
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > 0.0000000000000000e+00
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > -2.9999999999999999e-01
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> > 0.0000000000000000e+00
> >
> >
> >
> > From: Matthew Knepley <knepley at gmail.com>
> > To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <
> petsc-users at mcs.anl.gov>
> > Sent: Monday, April 2, 2012 5:41 AM
> > Subject: Re: [petsc-users] transfer vector data diagonally on DA
> >
> > On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <khalid_eee at yahoo.com>
> wrote:
> > Hi Jed,
> >
> > I am using petsc/3.1 and the include file "petscdmda.h" is not working.
> I am using the "petscda.h"
> > and DACreate3D.
> > unew_localptr[][][] is from a global vector using VecGetArray
> > u_localptr[][][] is from a local vector communicated from a global
> vector.
> > I tried
> > unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]
> > but it gives Segmentation Violation error.
> >
> > I would guess (because you provide almost no information about what you
> are doing), that
> > this is a domain on the edge. If you truly mean to have a periodic
> domain, you must set that in the
> > creation call. Then the local vector will also ghost regions outside the
> domain boundary.
> >
> >    Matt
> >
> > I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works
> but
> > unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or
> unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]
> > does not work. I am running the program on4 processors.
> >
> > Thanks.
> >
> >
> >   ierr =
> DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
> >
> PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
> >                     &appctx.da);CHKERRQ(ierr);
> >  for (k=zs; k<zs+zm; k++) {
> >   for (j=ys; j<ys+ym; j++) {
> >   for (i=xs; i<xs+xm; i++) {
> >
> >   if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
> >
> >   unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];
> > }}}
> > From: Jed Brown <jedbrown at mcs.anl.gov>
> > To: khalid ashraf <khalid_eee at yahoo.com>; PETSc users list <
> petsc-users at mcs.anl.gov>
> > Sent: Sunday, April 1, 2012 10:07 PM
> > Subject: Re: [petsc-users] transfer vector data diagonally on DA
> >
> > On Sun, Apr 1, 2012 at 22:01, khalid ashraf <khalid_eee at yahoo.com>
> wrote:
> > I want to transfer vector data diagonally in the DA grid like
> >  for (k=zs; k<zs+zm; k++) {
> >   for (j=ys; j<ys+ym; j++) {
> >   for (i=xs; i<xs+xm; i++) {
> > if(i!=mx-1 || j!=my-1 || k!=mz-1){
> > u_new[k+1][j+1][i+1]=u[k][j][i];}
> > }}}
> >
> > Could you please suggest the best way to do it minimizing interprocessor
> assignments.
> >
> > Both are on the same DMDA?
> >
> > Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil
> with width at least 1, get the global array u_new[][][] from UGlobalNew and
> the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] =
> u[k-1][j-1][i-1].
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/3cd2f82e/attachment.htm>

From fuentesdt at gmail.com  Tue Apr  3 12:36:02 2012
From: fuentesdt at gmail.com (David Fuentes)
Date: Tue, 3 Apr 2012 12:36:02 -0500
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
Message-ID: <CAKnkmZwQ2+TXzFO=vauyJDmEvH00JXH_jqv8F35N6QsWa0nwGQ@mail.gmail.com>

thank you Matt,
David

On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>
>>> Yes. This would work.
>>> I had trouble compiling in single precision using the some of the
>>> external package options I was using for double.
>>>
>>
>> Okay,  getting on it now.
>>
>
> Okay, I have tested using PetscReal=double with floats on the GPU. If you
> want doubles on the GPU, you can
> currently change PETSc/FEM.py:434. I am deciding what the best way to
> specify this type is.
>
>   Thanks,
>
>      Matt
>
>
>>    Matt
>>
>>
>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>
>>>>> works!
>>>>>
>>>>
>>>> Excellent. Now, my thinking was that GPUs are most useful doing single
>>>> work, but
>>>> I can see the utility of double accuracy for a residual.
>>>>
>>>> My inclination is to define another type, say GPUReal, and use it for
>>>> all kernels.
>>>> Would that do what you want?
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> SCRGP2$ make ex52
>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>> declared but never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>> declared but never referenced
>>>>>
>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>  -lpetsc
>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>
>>>>>
>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>  N_t: 3, N_cb: 1
>>>>> Residual:
>>>>> Vector Object: 1 MPI processes
>>>>>   type: seq
>>>>> -0.25
>>>>> -0.5
>>>>> 0.25
>>>>> -0.5
>>>>> -1
>>>>> 0.5
>>>>> 0.25
>>>>> 0.5
>>>>> 0.75
>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>> Residual:
>>>>> Vector Object: 1 MPI processes
>>>>>   type: seq
>>>>> -0.25
>>>>> -0.5
>>>>> 0.25
>>>>> -0.5
>>>>> -1
>>>>> 0.5
>>>>> 0.25
>>>>> 0.5
>>>>> 0.75
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>
>>>>>> sure. will do.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>>
>>>>>>>> thanks! its running, but I seem to be getting different answer for
>>>>>>>> cpu/gpu ?
>>>>>>>> i had some floating point problems on this Tesla M2070 gpu before,
>>>>>>>> but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>
>>>>>>>>
>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>
>>>>>>>
>>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>>> have not decided exactly how to handle it. Can you
>>>>>>> make another ARCH --with-precision=single and make sure it works,
>>>>>>> and then we can fix the double issue?
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> 0
>>>>>>>> 755712
>>>>>>>> 0
>>>>>>>> -58720
>>>>>>>> -2953.13
>>>>>>>> 0.375
>>>>>>>> 1.50323e+07
>>>>>>>> 0.875
>>>>>>>> 0
>>>>>>>> SCRGP2$
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> -0.25
>>>>>>>> -0.5
>>>>>>>> 0.25
>>>>>>>> -0.5
>>>>>>>> -1
>>>>>>>> 0.5
>>>>>>>> 0.25
>>>>>>>> 0.5
>>>>>>>> 0.75
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>
>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>
>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>> sizeof(float));CHKERRQ(ierr);
>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ python
>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>> laplacian ex52.h
>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>> []}}
>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>> []}}
>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>>> declared but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>>> declared but never referenced
>>>>>>>>>>
>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>> Residual:
>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>   type: seq
>>>>>>>>>> -0.25
>>>>>>>>>> -0.5
>>>>>>>>>> 0.25
>>>>>>>>>> -0.5
>>>>>>>>>> -1
>>>>>>>>>> 0.5
>>>>>>>>>> 0.25
>>>>>>>>>> 0.5
>>>>>>>>>> 0.75
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>> If so, applications like Preview can lock up the GPU. I know of no
>>>>>>>>> way to test this in CUDA while running. I just close
>>>>>>>>> apps until it runs.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <bourdin at lsu.edu
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>>> using the Python build system
>>>>>>>>>>>>
>>>>>>>>>>>>   ./config/builder2.py check src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>
>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Blaise
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>      Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> David
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  --
>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>> Technology
>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/e6f99f00/attachment-0001.htm>

From ywu at culink.Colorado.EDU  Tue Apr  3 14:25:36 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Tue,  3 Apr 2012 13:25:36 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
Message-ID: <20120403132536.APX82674@batman.int.colorado.edu>

Dear All,

I want to create two grid preconditioner for the linear Jacobian solve for the nonlinear problem. I am trying to use the inexact Newton as the nonlinear solver, and the fGMRES as the linear solve. For the preconditioner for the linear solve, I want to create a two level ASM preconditioner by using PCMG. 

I use the additive Schwarz (ASM) preconditioned Richardson iteration as the smoother on the fine grid. And use the ASM preconditioned GMRES as the coarse solve. 

But I have the follow questions in setting up the preconditioer,

1. When I use the command PCMGGetSmoother to setup the fine grid smoother, it only effective on the down solver but not the up solver. Although I want to use same smoother for up and down procedure, I need to call PCMGGetSmootherUp and PCMGGetSmootherDown to setup the smoother seperately. Do you have any ideas about this issue?

2. In my two level preconditioner, I need three ASM preconditioner, two for the up and down smoother, and one for the coarse solve. If I want to use LU factorization as the subdomain solve for both the up and down smoother, but I don't want to redo the LU factorization for the up smoother. How can I keep the LU factorization for the down smoother in order to use for the up smoother?

3. I run my code with the my twolevel preconditioner. The SNES converges in two iteration.
 
  0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00.
  1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
  2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.

In the log summary, I check the number of the factorization in the problem

MatLUFactorSym         4 1.0 1.1232e+00 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01  2  0  0  0  1   2  0  0  0  1     0
MatLUFactorNum         4 1.0 1.3245e+01 2.2 1.22e+09 3.0 0.0e+00 0.0e+00 0.0e+00 28 86  0  0  0  30 86  0  0  0   553

Do you have any ideas that why the number of MatLUFactorSym is 4? And is there any approach that I can find out how much LU factorization are done for the coarse solve, and how much LU factorization are done for the up and down smoother? I believe the number should be six, in each SNES iteration, I have need to do two LU for the up and down smoother, and one LU for the coarse solve. Because we have two SNES iteration now, the number of LU factorization should be 6 instead of 4.

Thank you so much for your help. Below is the output form the SNESView of my program, and the setup of my preconditioner.

Best

Yuqi Wu


/* SNES view */
SNES Object: 8 MPI processes
  type: ls
    line search variant: SNESLineSearchCubic
    alpha=1.000000000000e-04, maxstep=1.000000000000e+08, minlambda=1.000000000000e-12
  maximum iterations=10, maximum function evaluations=10000
  tolerances: relative=1e-07, absolute=1e-50, solution=1e-08
  total number of linear solver iterations=9
  total number of function evaluations=3
  KSP Object:   8 MPI processes  
    type: fgmres
      GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      GMRES: happy breakdown tolerance 1e-30
    maximum iterations=1000, initial guess is zero
    tolerances:  relative=1e-06, absolute=1e-14, divergence=10000
    right preconditioning
    using UNPRECONDITIONED norm type for convergence test
  PC Object:   8 MPI processes  
    type: mg
      MG: type is MULTIPLICATIVE, levels=2 cycles=v
        Cycles per PCApply=1
        Using Galerkin computed coarse grid matrices
    Coarse grid solver -- level -------------------------------
      KSP Object:      (coarse_)       8 MPI processes      
        type: gmres
          GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
          GMRES: happy breakdown tolerance 1e-30
        maximum iterations=1000, initial guess is zero
        tolerances:  relative=0.001, absolute=1e-50, divergence=10000
        right preconditioning
        using UNPRECONDITIONED norm type for convergence test
      PC Object:      (coarse_)       8 MPI processes      
        type: asm
          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
          Additive Schwarz: restriction/interpolation type - RESTRICT
          Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object:        (coarse_sub_)         1 MPI processes        
          type: preonly
          maximum iterations=10000, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (coarse_sub_)         1 MPI processes        
          type: lu
            LU: out-of-place factorization
            tolerance for zero pivot 1e-12
            matrix ordering: nd
            factor fill ratio given 5, needed 3.73447
              Factored matrix follows:
                Matrix Object:                 1 MPI processes                
                  type: seqaij
                  rows=840, cols=840
                  package used to perform factorization: petsc
                  total: nonzeros=384486, allocated nonzeros=384486
                  total number of mallocs used during MatSetValues calls =0
                    using I-node routines: found 349 nodes, limit used is 5
          linear system matrix = precond matrix:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=840, cols=840
            total: nonzeros=102956, allocated nonzeros=102956
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 360 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         8 MPI processes        
          type: mpiaij
          rows=4186, cols=4186
          total: nonzeros=656174, allocated nonzeros=656174
          total number of mallocs used during MatSetValues calls =0
            using I-node (on process 0) routines: found 315 nodes, limit used is 5
    Down solver (pre-smoother) on level 1 -------------------------------
      KSP Object:      (mg_levels_1_)       8 MPI processes      
        type: richardson
          Richardson: damping factor=1
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using PRECONDITIONED norm type for convergence test
      PC Object:      (fine_)       8 MPI processes      
        type: asm
          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
          Additive Schwarz: restriction/interpolation type - RESTRICT
          Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object:        (fine_sub_)         1 MPI processes        
          type: preonly
          maximum iterations=10000, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (fine_sub_)         1 MPI processes        
          type: lu
            LU: out-of-place factorization
            tolerance for zero pivot 1e-12
            matrix ordering: nd
            factor fill ratio given 5, needed 5.49559
              Factored matrix follows:
                Matrix Object:                 1 MPI processes                
                  type: seqaij
                  rows=2100, cols=2100
                  package used to perform factorization: petsc
                  total: nonzeros=401008, allocated nonzeros=401008
                  total number of mallocs used during MatSetValues calls =0
                    using I-node routines: found 1491 nodes, limit used is 5
          linear system matrix = precond matrix:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=2100, cols=2100
            total: nonzeros=72969, allocated nonzeros=72969
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 1532 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         8 MPI processes        
          type: mpiaij
          rows=11585, cols=11585
          total: nonzeros=458097, allocated nonzeros=3026770
          total number of mallocs used during MatSetValues calls =978
            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
    Up solver (post-smoother) on level 1 -------------------------------
      KSP Object:      (mg_levels_1_)       8 MPI processes      
        type: richardson
          Richardson: damping factor=1
        maximum iterations=1
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using nonzero initial guess
        using PRECONDITIONED norm type for convergence test
      PC Object:      (fine_)       8 MPI processes      
        type: asm
          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
          Additive Schwarz: restriction/interpolation type - RESTRICT
          Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object:        (fine_sub_)         1 MPI processes        
          type: preonly
          maximum iterations=10000, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (fine_sub_)         1 MPI processes        
          type: lu
            LU: out-of-place factorization
            tolerance for zero pivot 1e-12
            matrix ordering: nd
            factor fill ratio given 5, needed 5.49559
              Factored matrix follows:
                Matrix Object:                 1 MPI processes                
                  type: seqaij
                  rows=2100, cols=2100
                  package used to perform factorization: petsc
                  total: nonzeros=401008, allocated nonzeros=401008
                  total number of mallocs used during MatSetValues calls =0
                    using I-node routines: found 1491 nodes, limit used is 5
          linear system matrix = precond matrix:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=2100, cols=2100
            total: nonzeros=72969, allocated nonzeros=72969
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 1532 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         8 MPI processes        
          type: mpiaij
          rows=11585, cols=11585
          total: nonzeros=458097, allocated nonzeros=3026770
          total number of mallocs used during MatSetValues calls =978
            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
    linear system matrix = precond matrix:
    Matrix Object:     8 MPI processes    
      type: mpiaij
      rows=11585, cols=11585
      total: nonzeros=458097, allocated nonzeros=3026770
      total number of mallocs used during MatSetValues calls =978
        using I-node (on process 0) routines: found 1365 nodes, limit used is 5
SNES converged: CONVERGED_FNORM_RELATIVE.

//*******************************
Below setup of my preconditioner,

  /* set up the MG preconditioner */
  ierr = SNESGetKSP(snes,&fineksp);CHKERRQ(ierr);
  ierr = KSPGetPC(fineksp,&finepc);CHKERRQ(ierr);
  ierr = PCSetType(finepc,PCMG);CHKERRQ(ierr);
  ierr = PCMGSetType(finepc,PC_MG_MULTIPLICATIVE);CHKERRQ(ierr);
  ierr = PCMGSetLevels(finepc,2,PETSC_NULL);CHKERRQ(ierr);
  ierr = PCMGSetCycleType(finepc,PC_MG_CYCLE_V);CHKERRQ(ierr);
  ierr = PCMGSetNumberSmoothUp(finepc,1);CHKERRQ(ierr);
  ierr = PCMGSetNumberSmoothDown(finepc,1);CHKERRQ(ierr);
  ierr = PCMGSetGalerkin(finepc,PETSC_TRUE);CHKERRQ(ierr);
  ierr = PCMGSetResidual(finepc,1,PCMGDefaultResidual,algebra->J);CHKERRQ(ierr);

  ierr = PCMGSetInterpolation(finepc,1,ctx->Interp);CHKERRQ(ierr); 

  /* set up the coarse solve */
  ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
  ierr = KSPSetOptionsPrefix(ctx->coarseksp,"coarse_");CHKERRQ(ierr);
  ierr = KSPSetFromOptions(ctx->coarseksp);CHKERRQ(ierr);

  /* set up the fine grid smoother */
  ierr = PCMGGetSmoother(finepc,1,&kspsmooth);CHKERRQ(ierr);
  ierr = KSPSetType(kspsmooth, KSPRICHARDSON);CHKERRQ(ierr);
  ierr = KSPGetPC(kspsmooth,&asmpc);CHKERRQ(ierr);
  ierr = PCSetType(asmpc,PCASM);CHKERRQ(ierr);
  ierr = PCASMSetOverlap(asmpc,0);CHKERRQ(ierr);
  ierr = PCASMSetLocalSubdomains(asmpc,1,&grid->df_global_asm,PETSC_NULL);CHKERRQ(ierr);




 

From mark.adams at columbia.edu  Tue Apr  3 15:18:46 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Tue, 3 Apr 2012 16:18:46 -0400
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120403132536.APX82674@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
Message-ID: <4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>


On Apr 3, 2012, at 3:25 PM, Yuqi Wu wrote:

> Dear All,
> 
> I want to create two grid preconditioner for the linear Jacobian solve for the nonlinear problem. I am trying to use the inexact Newton as the nonlinear solver, and the fGMRES as the linear solve. For the preconditioner for the linear solve, I want to create a two level ASM preconditioner by using PCMG. 
> 
> I use the additive Schwarz (ASM) preconditioned Richardson iteration as the smoother on the fine grid. And use the ASM preconditioned GMRES as the coarse solve. 
> 
> But I have the follow questions in setting up the preconditioer,
> 
> 1. When I use the command PCMGGetSmoother to setup the fine grid smoother, it only effective on the down solver but not the up solver.

This should work.  I see that you have PCMGSetNumberSmoothUp in the code below.  By calling an up/down method I think PETSc will decouple the two smoothers, that is make a copy of the smoother and use one for up and one for down.  So you want to avoid and U//Down methods or command line parameters.

> Although I want to use same smoother for up and down procedure, I need to call PCMGGetSmootherUp and PCMGGetSmootherDown to setup the smoother seperately. Do you have any ideas about this issue?
> 
> 2. In my two level preconditioner, I need three ASM preconditioner, two for the up and down smoother, and one for the coarse solve. If I want to use LU factorization as the subdomain solve for both the up and down smoother, but I don't want to redo the LU factorization for the up smoother. How can I keep the LU factorization for the down smoother in order to use for the up smoother?
> 
> 3. I run my code with the my twolevel preconditioner. The SNES converges in two iteration.
> 
>  0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00.
>  1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>  2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
> 
> In the log summary, I check the number of the factorization in the problem
> 
> MatLUFactorSym         4 1.0 1.1232e+00 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01  2  0  0  0  1   2  0  0  0  1     0
> MatLUFactorNum         4 1.0 1.3245e+01 2.2 1.22e+09 3.0 0.0e+00 0.0e+00 0.0e+00 28 86  0  0  0  30 86  0  0  0   553
> 
> Do you have any ideas that why the number of MatLUFactorSym is 4? And is there any approach that I can find out how much LU factorization are done for the coarse solve, and how much LU factorization are done for the up and down smoother?

-pc_mg_log will give some level information but I'm not sure if it does setup stuff.  

Running with -info gives verbose output (I would run with one processor) and you can see the sizes of the factorizations, along with fill stats, etc.

Mark

> I believe the number should be six, in each SNES iteration, I have need to do two LU for the up and down smoother, and one LU for the coarse solve. Because we have two SNES iteration now, the number of LU factorization should be 6 instead of 4.
> 
> Thank you so much for your help. Below is the output form the SNESView of my program, and the setup of my preconditioner.
> 
> Best
> 
> Yuqi Wu
> 
> 
> /* SNES view */
> SNES Object: 8 MPI processes
>  type: ls
>    line search variant: SNESLineSearchCubic
>    alpha=1.000000000000e-04, maxstep=1.000000000000e+08, minlambda=1.000000000000e-12
>  maximum iterations=10, maximum function evaluations=10000
>  tolerances: relative=1e-07, absolute=1e-50, solution=1e-08
>  total number of linear solver iterations=9
>  total number of function evaluations=3
>  KSP Object:   8 MPI processes  
>    type: fgmres
>      GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>      GMRES: happy breakdown tolerance 1e-30
>    maximum iterations=1000, initial guess is zero
>    tolerances:  relative=1e-06, absolute=1e-14, divergence=10000
>    right preconditioning
>    using UNPRECONDITIONED norm type for convergence test
>  PC Object:   8 MPI processes  
>    type: mg
>      MG: type is MULTIPLICATIVE, levels=2 cycles=v
>        Cycles per PCApply=1
>        Using Galerkin computed coarse grid matrices
>    Coarse grid solver -- level -------------------------------
>      KSP Object:      (coarse_)       8 MPI processes      
>        type: gmres
>          GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>          GMRES: happy breakdown tolerance 1e-30
>        maximum iterations=1000, initial guess is zero
>        tolerances:  relative=0.001, absolute=1e-50, divergence=10000
>        right preconditioning
>        using UNPRECONDITIONED norm type for convergence test
>      PC Object:      (coarse_)       8 MPI processes      
>        type: asm
>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>          Additive Schwarz: restriction/interpolation type - RESTRICT
>          Local solve is same for all blocks, in the following KSP and PC objects:
>        KSP Object:        (coarse_sub_)         1 MPI processes        
>          type: preonly
>          maximum iterations=10000, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>        PC Object:        (coarse_sub_)         1 MPI processes        
>          type: lu
>            LU: out-of-place factorization
>            tolerance for zero pivot 1e-12
>            matrix ordering: nd
>            factor fill ratio given 5, needed 3.73447
>              Factored matrix follows:
>                Matrix Object:                 1 MPI processes                
>                  type: seqaij
>                  rows=840, cols=840
>                  package used to perform factorization: petsc
>                  total: nonzeros=384486, allocated nonzeros=384486
>                  total number of mallocs used during MatSetValues calls =0
>                    using I-node routines: found 349 nodes, limit used is 5
>          linear system matrix = precond matrix:
>          Matrix Object:           1 MPI processes          
>            type: seqaij
>            rows=840, cols=840
>            total: nonzeros=102956, allocated nonzeros=102956
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 360 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Matrix Object:         8 MPI processes        
>          type: mpiaij
>          rows=4186, cols=4186
>          total: nonzeros=656174, allocated nonzeros=656174
>          total number of mallocs used during MatSetValues calls =0
>            using I-node (on process 0) routines: found 315 nodes, limit used is 5
>    Down solver (pre-smoother) on level 1 -------------------------------
>      KSP Object:      (mg_levels_1_)       8 MPI processes      
>        type: richardson
>          Richardson: damping factor=1
>        maximum iterations=1, initial guess is zero
>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>        left preconditioning
>        using PRECONDITIONED norm type for convergence test
>      PC Object:      (fine_)       8 MPI processes      
>        type: asm
>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>          Additive Schwarz: restriction/interpolation type - RESTRICT
>          Local solve is same for all blocks, in the following KSP and PC objects:
>        KSP Object:        (fine_sub_)         1 MPI processes        
>          type: preonly
>          maximum iterations=10000, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>        PC Object:        (fine_sub_)         1 MPI processes        
>          type: lu
>            LU: out-of-place factorization
>            tolerance for zero pivot 1e-12
>            matrix ordering: nd
>            factor fill ratio given 5, needed 5.49559
>              Factored matrix follows:
>                Matrix Object:                 1 MPI processes                
>                  type: seqaij
>                  rows=2100, cols=2100
>                  package used to perform factorization: petsc
>                  total: nonzeros=401008, allocated nonzeros=401008
>                  total number of mallocs used during MatSetValues calls =0
>                    using I-node routines: found 1491 nodes, limit used is 5
>          linear system matrix = precond matrix:
>          Matrix Object:           1 MPI processes          
>            type: seqaij
>            rows=2100, cols=2100
>            total: nonzeros=72969, allocated nonzeros=72969
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 1532 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Matrix Object:         8 MPI processes        
>          type: mpiaij
>          rows=11585, cols=11585
>          total: nonzeros=458097, allocated nonzeros=3026770
>          total number of mallocs used during MatSetValues calls =978
>            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
>    Up solver (post-smoother) on level 1 -------------------------------
>      KSP Object:      (mg_levels_1_)       8 MPI processes      
>        type: richardson
>          Richardson: damping factor=1
>        maximum iterations=1
>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>        left preconditioning
>        using nonzero initial guess
>        using PRECONDITIONED norm type for convergence test
>      PC Object:      (fine_)       8 MPI processes      
>        type: asm
>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>          Additive Schwarz: restriction/interpolation type - RESTRICT
>          Local solve is same for all blocks, in the following KSP and PC objects:
>        KSP Object:        (fine_sub_)         1 MPI processes        
>          type: preonly
>          maximum iterations=10000, initial guess is zero
>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>          left preconditioning
>          using NONE norm type for convergence test
>        PC Object:        (fine_sub_)         1 MPI processes        
>          type: lu
>            LU: out-of-place factorization
>            tolerance for zero pivot 1e-12
>            matrix ordering: nd
>            factor fill ratio given 5, needed 5.49559
>              Factored matrix follows:
>                Matrix Object:                 1 MPI processes                
>                  type: seqaij
>                  rows=2100, cols=2100
>                  package used to perform factorization: petsc
>                  total: nonzeros=401008, allocated nonzeros=401008
>                  total number of mallocs used during MatSetValues calls =0
>                    using I-node routines: found 1491 nodes, limit used is 5
>          linear system matrix = precond matrix:
>          Matrix Object:           1 MPI processes          
>            type: seqaij
>            rows=2100, cols=2100
>            total: nonzeros=72969, allocated nonzeros=72969
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 1532 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Matrix Object:         8 MPI processes        
>          type: mpiaij
>          rows=11585, cols=11585
>          total: nonzeros=458097, allocated nonzeros=3026770
>          total number of mallocs used during MatSetValues calls =978
>            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
>    linear system matrix = precond matrix:
>    Matrix Object:     8 MPI processes    
>      type: mpiaij
>      rows=11585, cols=11585
>      total: nonzeros=458097, allocated nonzeros=3026770
>      total number of mallocs used during MatSetValues calls =978
>        using I-node (on process 0) routines: found 1365 nodes, limit used is 5
> SNES converged: CONVERGED_FNORM_RELATIVE.
> 
> //*******************************
> Below setup of my preconditioner,
> 
>  /* set up the MG preconditioner */
>  ierr = SNESGetKSP(snes,&fineksp);CHKERRQ(ierr);
>  ierr = KSPGetPC(fineksp,&finepc);CHKERRQ(ierr);
>  ierr = PCSetType(finepc,PCMG);CHKERRQ(ierr);
>  ierr = PCMGSetType(finepc,PC_MG_MULTIPLICATIVE);CHKERRQ(ierr);
>  ierr = PCMGSetLevels(finepc,2,PETSC_NULL);CHKERRQ(ierr);
>  ierr = PCMGSetCycleType(finepc,PC_MG_CYCLE_V);CHKERRQ(ierr);
>  ierr = PCMGSetNumberSmoothUp(finepc,1);CHKERRQ(ierr);
>  ierr = PCMGSetNumberSmoothDown(finepc,1);CHKERRQ(ierr);
>  ierr = PCMGSetGalerkin(finepc,PETSC_TRUE);CHKERRQ(ierr);
>  ierr = PCMGSetResidual(finepc,1,PCMGDefaultResidual,algebra->J);CHKERRQ(ierr);
> 
>  ierr = PCMGSetInterpolation(finepc,1,ctx->Interp);CHKERRQ(ierr); 
> 
>  /* set up the coarse solve */
>  ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>  ierr = KSPSetOptionsPrefix(ctx->coarseksp,"coarse_");CHKERRQ(ierr);
>  ierr = KSPSetFromOptions(ctx->coarseksp);CHKERRQ(ierr);
> 
>  /* set up the fine grid smoother */
>  ierr = PCMGGetSmoother(finepc,1,&kspsmooth);CHKERRQ(ierr);
>  ierr = KSPSetType(kspsmooth, KSPRICHARDSON);CHKERRQ(ierr);
>  ierr = KSPGetPC(kspsmooth,&asmpc);CHKERRQ(ierr);
>  ierr = PCSetType(asmpc,PCASM);CHKERRQ(ierr);
>  ierr = PCASMSetOverlap(asmpc,0);CHKERRQ(ierr);
>  ierr = PCASMSetLocalSubdomains(asmpc,1,&grid->df_global_asm,PETSC_NULL);CHKERRQ(ierr);
> 
> 
> 
> 
> 
> 


From renzhengyong at gmail.com  Tue Apr  3 16:25:31 2012
From: renzhengyong at gmail.com (RenZhengYong)
Date: Tue, 3 Apr 2012 23:25:31 +0200
Subject: [petsc-users] precondition for matrix-free GMRES
Message-ID: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>

Dear petsc users,

I have a question to ask for your suggestions which is to solve Ax=b using
GMRES,
here A is partially dense.  Using petsc, I successfully used the
matrix-free approach to solve
it so that the expensive storage of A is avoided.  My question is could I
offer an matrix B
(which is sparse matrix and good approximation to A) so that the
convergence rate of
GMRES can be speed up.

Best wishes
Zhengyong


-- 
Zhengyong Ren
AUG Group, Institute of Geophysics
Department of Geosciences, ETH Zurich
NO H 47 Sonneggstrasse 5
CH-8092, Z?rich, Switzerland
Tel: +41 44 633 37561
e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
Gmail: renzhengyong at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/18eb0fae/attachment.htm>

From jedbrown at mcs.anl.gov  Tue Apr  3 16:28:01 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 3 Apr 2012 14:28:01 -0700
Subject: [petsc-users] precondition for matrix-free GMRES
In-Reply-To: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>
References: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>
Message-ID: <CAM9tzS=nonfkjSV8pX-RbCOrtg2MVCGbts11ZYVy8Kid9w4Law@mail.gmail.com>

On Tue, Apr 3, 2012 at 14:25, RenZhengYong <renzhengyong at gmail.com> wrote:

> I have a question to ask for your suggestions which is to solve Ax=b using
> GMRES,
> here A is partially dense.  Using petsc, I successfully used the
> matrix-free approach to solve
> it so that the expensive storage of A is avoided.  My question is could I
> offer an matrix B
> (which is sparse matrix and good approximation to A) so that the
> convergence rate of
> GMRES can be speed up.
>

Certainly, but the challenge is to find this other operator. You could try
sparse approximate methods to approximate either the operator or its
inverse. Other approaches would typically involve further knowledge of your
problem.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/277a75d5/attachment-0001.htm>

From renzhengyong at gmail.com  Tue Apr  3 16:40:26 2012
From: renzhengyong at gmail.com (RenZhengYong)
Date: Tue, 3 Apr 2012 23:40:26 +0200
Subject: [petsc-users] precondition for matrix-free GMRES
In-Reply-To: <CAM9tzS=nonfkjSV8pX-RbCOrtg2MVCGbts11ZYVy8Kid9w4Law@mail.gmail.com>
References: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>
	<CAM9tzS=nonfkjSV8pX-RbCOrtg2MVCGbts11ZYVy8Kid9w4Law@mail.gmail.com>
Message-ID: <CA+OnyrDyQSg15nDJM1u5GsjSUvt0X5ichHFi5V21-f-ajGZArQ@mail.gmail.com>

Hi, Jed,
Thanks for your quick reply and have a nice day.

My problem is to solve a hybrid FEM-BEM problem, so one sub-matrix from BEM
is fully dense and one from BEM-FEM is partially dense. I managed to used a
multi-level fast multpole method to compute the product of my entire system
matrix with a given vector. And I can find a quite good problem dependent
approximation B to my system matrix *A*.

My next purpose to speed up the convergence rate of GMRES using this
operator *B*.  In my code, the expensive of storage of the far field
interaction from BEM part is avoided. And I can explicitly form the sparse
matrix *B*.  So, could I ask you:
how to give the sparse matrix *B* to petsc?. Because *B* is sparse, I
prefer to fully LU decomposition to solve the preconditioned problem *B*z=c.


Best wishes
Zhengyong

On Tue, Apr 3, 2012 at 11:28 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Tue, Apr 3, 2012 at 14:25, RenZhengYong <renzhengyong at gmail.com> wrote:
>
>> I have a question to ask for your suggestions which is to solve Ax=b
>> using GMRES,
>> here A is partially dense.  Using petsc, I successfully used the
>> matrix-free approach to solve
>> it so that the expensive storage of A is avoided.  My question is could I
>> offer an matrix B
>> (which is sparse matrix and good approximation to A) so that the
>> convergence rate of
>> GMRES can be speed up.
>>
>
> Certainly, but the challenge is to find this other operator. You could try
> sparse approximate methods to approximate either the operator or its
> inverse. Other approaches would typically involve further knowledge of your
> problem.
>



-- 
Zhengyong Ren
AUG Group, Institute of Geophysics
Department of Geosciences, ETH Zurich
NO H 47 Sonneggstrasse 5
CH-8092, Z?rich, Switzerland
Tel: +41 44 633 37561
e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
Gmail: renzhengyong at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/6ad0a0c3/attachment.htm>

From knepley at gmail.com  Tue Apr  3 16:44:39 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 3 Apr 2012 16:44:39 -0500
Subject: [petsc-users] precondition for matrix-free GMRES
In-Reply-To: <CA+OnyrDyQSg15nDJM1u5GsjSUvt0X5ichHFi5V21-f-ajGZArQ@mail.gmail.com>
References: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>
	<CAM9tzS=nonfkjSV8pX-RbCOrtg2MVCGbts11ZYVy8Kid9w4Law@mail.gmail.com>
	<CA+OnyrDyQSg15nDJM1u5GsjSUvt0X5ichHFi5V21-f-ajGZArQ@mail.gmail.com>
Message-ID: <CAMYG4G=Qs+VCmcLzYVCFg-jtXz2UTyVebGGVOJwAGzRc6j7VgA@mail.gmail.com>

On Tue, Apr 3, 2012 at 4:40 PM, RenZhengYong <renzhengyong at gmail.com> wrote:

> Hi, Jed,
> Thanks for your quick reply and have a nice day.
>
> My problem is to solve a hybrid FEM-BEM problem, so one sub-matrix from
> BEM is fully dense and one from BEM-FEM is partially dense. I managed to
> used a multi-level fast multpole method to compute the product of my entire
> system matrix with a given vector. And I can find a quite good problem
> dependent approximation B to my system matrix *A*.
>
> My next purpose to speed up the convergence rate of GMRES using this
> operator *B*.  In my code, the expensive of storage of the far field
> interaction from BEM part is avoided. And I can explicitly form the sparse
> matrix *B*.  So, could I ask you:
> how to give the sparse matrix *B* to petsc?. Because *B* is sparse, I
> prefer to fully LU decomposition to solve the preconditioned problem *B*z=c.
>
>

Give B as the preconditioner matrix,

  KSPSetOperators(ksp, A, B, ...)

    Matt


> Best wishes
> Zhengyong
>
> On Tue, Apr 3, 2012 at 11:28 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Tue, Apr 3, 2012 at 14:25, RenZhengYong <renzhengyong at gmail.com>wrote:
>>
>>> I have a question to ask for your suggestions which is to solve Ax=b
>>> using GMRES,
>>> here A is partially dense.  Using petsc, I successfully used the
>>> matrix-free approach to solve
>>> it so that the expensive storage of A is avoided.  My question is could
>>> I offer an matrix B
>>> (which is sparse matrix and good approximation to A) so that the
>>> convergence rate of
>>> GMRES can be speed up.
>>>
>>
>> Certainly, but the challenge is to find this other operator. You could
>> try sparse approximate methods to approximate either the operator or its
>> inverse. Other approaches would typically involve further knowledge of your
>> problem.
>>
>
>
>
> --
> Zhengyong Ren
> AUG Group, Institute of Geophysics
> Department of Geosciences, ETH Zurich
> NO H 47 Sonneggstrasse 5
> CH-8092, Z?rich, Switzerland
> Tel: +41 44 633 37561
> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
> Gmail: renzhengyong at gmail.com
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/86db6e31/attachment.htm>

From renzhengyong at gmail.com  Tue Apr  3 16:49:52 2012
From: renzhengyong at gmail.com (RenZhengYong)
Date: Tue, 3 Apr 2012 23:49:52 +0200
Subject: [petsc-users] precondition for matrix-free GMRES
In-Reply-To: <CAMYG4G=Qs+VCmcLzYVCFg-jtXz2UTyVebGGVOJwAGzRc6j7VgA@mail.gmail.com>
References: <CA+OnyrA5cmQ_yfo=Y-O1Ur2boOFATdCrdm6SEt3-=hX=WyL4wQ@mail.gmail.com>
	<CAM9tzS=nonfkjSV8pX-RbCOrtg2MVCGbts11ZYVy8Kid9w4Law@mail.gmail.com>
	<CA+OnyrDyQSg15nDJM1u5GsjSUvt0X5ichHFi5V21-f-ajGZArQ@mail.gmail.com>
	<CAMYG4G=Qs+VCmcLzYVCFg-jtXz2UTyVebGGVOJwAGzRc6j7VgA@mail.gmail.com>
Message-ID: <CA+OnyrC13mnSR=w_Qo0m0pFDysF6b7vOv452rMabm_zzGJCZtQ@mail.gmail.com>

Hi, Matt,
Thanks for the help. I got it.

Have a nice day
Best wishes
Zhengyong

On Tue, Apr 3, 2012 at 11:44 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Apr 3, 2012 at 4:40 PM, RenZhengYong <renzhengyong at gmail.com>wrote:
>
>> Hi, Jed,
>> Thanks for your quick reply and have a nice day.
>>
>> My problem is to solve a hybrid FEM-BEM problem, so one sub-matrix from
>> BEM is fully dense and one from BEM-FEM is partially dense. I managed to
>> used a multi-level fast multpole method to compute the product of my entire
>> system matrix with a given vector. And I can find a quite good problem
>> dependent approximation B to my system matrix *A*.
>>
>> My next purpose to speed up the convergence rate of GMRES using this
>> operator *B*.  In my code, the expensive of storage of the far field
>> interaction from BEM part is avoided. And I can explicitly form the sparse
>> matrix *B*.  So, could I ask you:
>> how to give the sparse matrix *B* to petsc?. Because *B* is sparse, I
>> prefer to fully LU decomposition to solve the preconditioned problem *B*z=c.
>>
>>
>
> Give B as the preconditioner matrix,
>
>   KSPSetOperators(ksp, A, B, ...)
>
>     Matt
>
>
>> Best wishes
>> Zhengyong
>>
>> On Tue, Apr 3, 2012 at 11:28 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>> On Tue, Apr 3, 2012 at 14:25, RenZhengYong <renzhengyong at gmail.com>wrote:
>>>
>>>> I have a question to ask for your suggestions which is to solve Ax=b
>>>> using GMRES,
>>>> here A is partially dense.  Using petsc, I successfully used the
>>>> matrix-free approach to solve
>>>> it so that the expensive storage of A is avoided.  My question is could
>>>> I offer an matrix B
>>>> (which is sparse matrix and good approximation to A) so that the
>>>> convergence rate of
>>>> GMRES can be speed up.
>>>>
>>>
>>> Certainly, but the challenge is to find this other operator. You could
>>> try sparse approximate methods to approximate either the operator or its
>>> inverse. Other approaches would typically involve further knowledge of your
>>> problem.
>>>
>>
>>
>>
>> --
>> Zhengyong Ren
>> AUG Group, Institute of Geophysics
>> Department of Geosciences, ETH Zurich
>> NO H 47 Sonneggstrasse 5
>> CH-8092, Z?rich, Switzerland
>> Tel: +41 44 633 37561
>> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
>> Gmail: renzhengyong at gmail.com
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Zhengyong Ren
AUG Group, Institute of Geophysics
Department of Geosciences, ETH Zurich
NO H 47 Sonneggstrasse 5
CH-8092, Z?rich, Switzerland
Tel: +41 44 633 37561
e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
Gmail: renzhengyong at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/3c60b66b/attachment.htm>

From ywu at culink.Colorado.EDU  Tue Apr  3 18:56:28 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Tue,  3 Apr 2012 17:56:28 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
Message-ID: <20120403175628.APY01823@batman.int.colorado.edu>

Hi, Mark,

Thank you so much for your suggestion.

The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 

But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.

As I mentioned before, I am solving a nonlinear problem, and use inexact Newton with linear search as the nonlinear solve, the fGMRES as the linear solve and two level multi-grid type ASM preconditioner as the preconditioner. For example, if solving the nonlinear problem needs two SNES iteration. In my opinion, in each SNES iteration, we need to setup the ASM preconditioner for the up and down smoother, and we need to setup the coarse solve (ASM preconditioned GMRES) every KSP iteration. That means we need to have 2 LU factorization for setting up the up and down smoother in every SNES iteration, and at least one LU factorization for the coarse solve. Since the up and down smoother are the same, LU factorization can be saved in the up smoother stage. That is at each SNES iteration, it needs at least 2 LU factorization, one for down smoother and one for coarse solve. So if my program converged in 2 SNES iterations, it should need at least 4 LU factorizations, 2 for the down sm
 o!
other, and 2 for the coarse solve. But in my -info output, it shows that there are only 3 LU factorization are carried on in my program. One MatLUFactorSymbolic is used in the down smoother of the first SNES iteration, one MatLUFactorSymbolic is used in the coarse solve of the first SNES iteration, and one MatLUFactorSymbolic is used in the coarse solve of the second SNES iteration. It seems that PETSC didn't call MatLUFactorSymbolic for the down smoother in the second SNES iteration. Do you have any ideas about this issue? It seems that in the second SNES iteration, the smoothers still use the LU factorization of the previous SNES iteration. I have enclosed the -info output as an attachment in this email. In the -info output file, it has the following lines

[0] PCSetUp_MG(): Using outer operators to define finest grid operator because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.

It seems that the KSP for the smoother is using the SAME_PRECONDITIONER flag, so that the down smoother in second SNES iteration is not called MatLUFactorSymbolic any more.

Can I also ask another questions related to the twolevel preconditioner? I have also try to use PCComposite to create two level ASM preconditioner instead of using PCMG. Since I am not able to use DA, which way is better to use in create a two level preconditioner? 

If I use the PCMG approach, it seems that the LU factorization in the up smoother can be avoided, only the LU factorization in the down smoother are need to be computed. But if I have the following precondition created with PCComposite

 ierr = PCSetType(finepc,PCCOMPOSITE);CHKERRQ(ierr);
 /* smooth down preconditioner */
 ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);
 /* coarse solve */
 ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);
 /* smooth up preconditioner */
 ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);

How can I save the LU factorization in the smooth up stage, if I want to use the same precondition in smooth up and smooth down?

Thank you.


Yuqi 



---- Original message ----
>Date: Tue, 3 Apr 2012 16:18:46 -0400
>From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 3, 2012, at 3:25 PM, Yuqi Wu wrote:
>
>> Dear All,
>> 
>> I want to create two grid preconditioner for the linear Jacobian solve for the nonlinear problem. I am trying to use the inexact Newton as the nonlinear solver, and the fGMRES as the linear solve. For the preconditioner for the linear solve, I want to create a two level ASM preconditioner by using PCMG. 
>> 
>> I use the additive Schwarz (ASM) preconditioned Richardson iteration as the smoother on the fine grid. And use the ASM preconditioned GMRES as the coarse solve. 
>> 
>> But I have the follow questions in setting up the preconditioer,
>> 
>> 1. When I use the command PCMGGetSmoother to setup the fine grid smoother, it only effective on the down solver but not the up solver.
>
>This should work.  I see that you have PCMGSetNumberSmoothUp in the code below.  By calling an up/down method I think PETSc will decouple the two smoothers, that is make a copy of the smoother and use one for up and one for down.  So you want to avoid and U//Down methods or command line parameters.
>
>> Although I want to use same smoother for up and down procedure, I need to call PCMGGetSmootherUp and PCMGGetSmootherDown to setup the smoother seperately. Do you have any ideas about this issue?
>> 
>> 2. In my two level preconditioner, I need three ASM preconditioner, two for the up and down smoother, and one for the coarse solve. If I want to use LU factorization as the subdomain solve for both the up and down smoother, but I don't want to redo the LU factorization for the up smoother. How can I keep the LU factorization for the down smoother in order to use for the up smoother?
>> 
>> 3. I run my code with the my twolevel preconditioner. The SNES converges in two iteration.
>> 
>>  0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00.
>>  1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>  2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>> 
>> In the log summary, I check the number of the factorization in the problem
>> 
>> MatLUFactorSym         4 1.0 1.1232e+00 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01  2  0  0  0  1   2  0  0  0  1     0
>> MatLUFactorNum         4 1.0 1.3245e+01 2.2 1.22e+09 3.0 0.0e+00 0.0e+00 0.0e+00 28 86  0  0  0  30 86  0  0  0   553
>> 
>> Do you have any ideas that why the number of MatLUFactorSym is 4? And is there any approach that I can find out how much LU factorization are done for the coarse solve, and how much LU factorization are done for the up and down smoother?
>
>-pc_mg_log will give some level information but I'm not sure if it does setup stuff.  
>
>Running with -info gives verbose output (I would run with one processor) and you can see the sizes of the factorizations, along with fill stats, etc.
>
>Mark
>
>> I believe the number should be six, in each SNES iteration, I have need to do two LU for the up and down smoother, and one LU for the coarse solve. Because we have two SNES iteration now, the number of LU factorization should be 6 instead of 4.
>> 
>> Thank you so much for your help. Below is the output form the SNESView of my program, and the setup of my preconditioner.
>> 
>> Best
>> 
>> Yuqi Wu
>> 
>> 
>> /* SNES view */
>> SNES Object: 8 MPI processes
>>  type: ls
>>    line search variant: SNESLineSearchCubic
>>    alpha=1.000000000000e-04, maxstep=1.000000000000e+08, minlambda=1.000000000000e-12
>>  maximum iterations=10, maximum function evaluations=10000
>>  tolerances: relative=1e-07, absolute=1e-50, solution=1e-08
>>  total number of linear solver iterations=9
>>  total number of function evaluations=3
>>  KSP Object:   8 MPI processes  
>>    type: fgmres
>>      GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>      GMRES: happy breakdown tolerance 1e-30
>>    maximum iterations=1000, initial guess is zero
>>    tolerances:  relative=1e-06, absolute=1e-14, divergence=10000
>>    right preconditioning
>>    using UNPRECONDITIONED norm type for convergence test
>>  PC Object:   8 MPI processes  
>>    type: mg
>>      MG: type is MULTIPLICATIVE, levels=2 cycles=v
>>        Cycles per PCApply=1
>>        Using Galerkin computed coarse grid matrices
>>    Coarse grid solver -- level -------------------------------
>>      KSP Object:      (coarse_)       8 MPI processes      
>>        type: gmres
>>          GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>          GMRES: happy breakdown tolerance 1e-30
>>        maximum iterations=1000, initial guess is zero
>>        tolerances:  relative=0.001, absolute=1e-50, divergence=10000
>>        right preconditioning
>>        using UNPRECONDITIONED norm type for convergence test
>>      PC Object:      (coarse_)       8 MPI processes      
>>        type: asm
>>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>>          Additive Schwarz: restriction/interpolation type - RESTRICT
>>          Local solve is same for all blocks, in the following KSP and PC objects:
>>        KSP Object:        (coarse_sub_)         1 MPI processes        
>>          type: preonly
>>          maximum iterations=10000, initial guess is zero
>>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>          left preconditioning
>>          using NONE norm type for convergence test
>>        PC Object:        (coarse_sub_)         1 MPI processes        
>>          type: lu
>>            LU: out-of-place factorization
>>            tolerance for zero pivot 1e-12
>>            matrix ordering: nd
>>            factor fill ratio given 5, needed 3.73447
>>              Factored matrix follows:
>>                Matrix Object:                 1 MPI processes                
>>                  type: seqaij
>>                  rows=840, cols=840
>>                  package used to perform factorization: petsc
>>                  total: nonzeros=384486, allocated nonzeros=384486
>>                  total number of mallocs used during MatSetValues calls =0
>>                    using I-node routines: found 349 nodes, limit used is 5
>>          linear system matrix = precond matrix:
>>          Matrix Object:           1 MPI processes          
>>            type: seqaij
>>            rows=840, cols=840
>>            total: nonzeros=102956, allocated nonzeros=102956
>>            total number of mallocs used during MatSetValues calls =0
>>              using I-node routines: found 360 nodes, limit used is 5
>>        linear system matrix = precond matrix:
>>        Matrix Object:         8 MPI processes        
>>          type: mpiaij
>>          rows=4186, cols=4186
>>          total: nonzeros=656174, allocated nonzeros=656174
>>          total number of mallocs used during MatSetValues calls =0
>>            using I-node (on process 0) routines: found 315 nodes, limit used is 5
>>    Down solver (pre-smoother) on level 1 -------------------------------
>>      KSP Object:      (mg_levels_1_)       8 MPI processes      
>>        type: richardson
>>          Richardson: damping factor=1
>>        maximum iterations=1, initial guess is zero
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>        left preconditioning
>>        using PRECONDITIONED norm type for convergence test
>>      PC Object:      (fine_)       8 MPI processes      
>>        type: asm
>>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>>          Additive Schwarz: restriction/interpolation type - RESTRICT
>>          Local solve is same for all blocks, in the following KSP and PC objects:
>>        KSP Object:        (fine_sub_)         1 MPI processes        
>>          type: preonly
>>          maximum iterations=10000, initial guess is zero
>>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>          left preconditioning
>>          using NONE norm type for convergence test
>>        PC Object:        (fine_sub_)         1 MPI processes        
>>          type: lu
>>            LU: out-of-place factorization
>>            tolerance for zero pivot 1e-12
>>            matrix ordering: nd
>>            factor fill ratio given 5, needed 5.49559
>>              Factored matrix follows:
>>                Matrix Object:                 1 MPI processes                
>>                  type: seqaij
>>                  rows=2100, cols=2100
>>                  package used to perform factorization: petsc
>>                  total: nonzeros=401008, allocated nonzeros=401008
>>                  total number of mallocs used during MatSetValues calls =0
>>                    using I-node routines: found 1491 nodes, limit used is 5
>>          linear system matrix = precond matrix:
>>          Matrix Object:           1 MPI processes          
>>            type: seqaij
>>            rows=2100, cols=2100
>>            total: nonzeros=72969, allocated nonzeros=72969
>>            total number of mallocs used during MatSetValues calls =0
>>              using I-node routines: found 1532 nodes, limit used is 5
>>        linear system matrix = precond matrix:
>>        Matrix Object:         8 MPI processes        
>>          type: mpiaij
>>          rows=11585, cols=11585
>>          total: nonzeros=458097, allocated nonzeros=3026770
>>          total number of mallocs used during MatSetValues calls =978
>>            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
>>    Up solver (post-smoother) on level 1 -------------------------------
>>      KSP Object:      (mg_levels_1_)       8 MPI processes      
>>        type: richardson
>>          Richardson: damping factor=1
>>        maximum iterations=1
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>        left preconditioning
>>        using nonzero initial guess
>>        using PRECONDITIONED norm type for convergence test
>>      PC Object:      (fine_)       8 MPI processes      
>>        type: asm
>>          Additive Schwarz: total subdomain blocks = 8, user-defined overlap
>>          Additive Schwarz: restriction/interpolation type - RESTRICT
>>          Local solve is same for all blocks, in the following KSP and PC objects:
>>        KSP Object:        (fine_sub_)         1 MPI processes        
>>          type: preonly
>>          maximum iterations=10000, initial guess is zero
>>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>          left preconditioning
>>          using NONE norm type for convergence test
>>        PC Object:        (fine_sub_)         1 MPI processes        
>>          type: lu
>>            LU: out-of-place factorization
>>            tolerance for zero pivot 1e-12
>>            matrix ordering: nd
>>            factor fill ratio given 5, needed 5.49559
>>              Factored matrix follows:
>>                Matrix Object:                 1 MPI processes                
>>                  type: seqaij
>>                  rows=2100, cols=2100
>>                  package used to perform factorization: petsc
>>                  total: nonzeros=401008, allocated nonzeros=401008
>>                  total number of mallocs used during MatSetValues calls =0
>>                    using I-node routines: found 1491 nodes, limit used is 5
>>          linear system matrix = precond matrix:
>>          Matrix Object:           1 MPI processes          
>>            type: seqaij
>>            rows=2100, cols=2100
>>            total: nonzeros=72969, allocated nonzeros=72969
>>            total number of mallocs used during MatSetValues calls =0
>>              using I-node routines: found 1532 nodes, limit used is 5
>>        linear system matrix = precond matrix:
>>        Matrix Object:         8 MPI processes        
>>          type: mpiaij
>>          rows=11585, cols=11585
>>          total: nonzeros=458097, allocated nonzeros=3026770
>>          total number of mallocs used during MatSetValues calls =978
>>            using I-node (on process 0) routines: found 1365 nodes, limit used is 5
>>    linear system matrix = precond matrix:
>>    Matrix Object:     8 MPI processes    
>>      type: mpiaij
>>      rows=11585, cols=11585
>>      total: nonzeros=458097, allocated nonzeros=3026770
>>      total number of mallocs used during MatSetValues calls =978
>>        using I-node (on process 0) routines: found 1365 nodes, limit used is 5
>> SNES converged: CONVERGED_FNORM_RELATIVE.
>> 
>> //*******************************
>> Below setup of my preconditioner,
>> 
>>  /* set up the MG preconditioner */
>>  ierr = SNESGetKSP(snes,&fineksp);CHKERRQ(ierr);
>>  ierr = KSPGetPC(fineksp,&finepc);CHKERRQ(ierr);
>>  ierr = PCSetType(finepc,PCMG);CHKERRQ(ierr);
>>  ierr = PCMGSetType(finepc,PC_MG_MULTIPLICATIVE);CHKERRQ(ierr);
>>  ierr = PCMGSetLevels(finepc,2,PETSC_NULL);CHKERRQ(ierr);
>>  ierr = PCMGSetCycleType(finepc,PC_MG_CYCLE_V);CHKERRQ(ierr);
>>  ierr = PCMGSetNumberSmoothUp(finepc,1);CHKERRQ(ierr);
>>  ierr = PCMGSetNumberSmoothDown(finepc,1);CHKERRQ(ierr);
>>  ierr = PCMGSetGalerkin(finepc,PETSC_TRUE);CHKERRQ(ierr);
>>  ierr = PCMGSetResidual(finepc,1,PCMGDefaultResidual,algebra->J);CHKERRQ(ierr);
>> 
>>  ierr = PCMGSetInterpolation(finepc,1,ctx->Interp);CHKERRQ(ierr); 
>> 
>>  /* set up the coarse solve */
>>  ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>  ierr = KSPSetOptionsPrefix(ctx->coarseksp,"coarse_");CHKERRQ(ierr);
>>  ierr = KSPSetFromOptions(ctx->coarseksp);CHKERRQ(ierr);
>> 
>>  /* set up the fine grid smoother */
>>  ierr = PCMGGetSmoother(finepc,1,&kspsmooth);CHKERRQ(ierr);
>>  ierr = KSPSetType(kspsmooth, KSPRICHARDSON);CHKERRQ(ierr);
>>  ierr = KSPGetPC(kspsmooth,&asmpc);CHKERRQ(ierr);
>>  ierr = PCSetType(asmpc,PCASM);CHKERRQ(ierr);
>>  ierr = PCASMSetOverlap(asmpc,0);CHKERRQ(ierr);
>>  ierr = PCASMSetLocalSubdomains(asmpc,1,&grid->df_global_asm,PETSC_NULL);CHKERRQ(ierr);
>> 
>> 
>> 
>> 
>> 
>> 
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mg_info.txt
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/4aac5bbd/attachment-0001.txt>

From bsmith at mcs.anl.gov  Tue Apr  3 19:19:23 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Apr 2012 19:19:23 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120403175628.APY01823@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
Message-ID: <99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>


  -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?

   Barry

On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:

> Hi, Mark,
> 
> Thank you so much for your suggestion.
> 
> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
> 
> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.


From ywu at culink.Colorado.EDU  Tue Apr  3 19:57:32 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Tue,  3 Apr 2012 18:57:32 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
Message-ID: <20120403185732.APY04392@batman.int.colorado.edu>

Hi, Barry,

Thank you. If my program converges in two SNES iteration,
0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.

And -pc_mg_log shows the following output

MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37

What are the MGSmooth, MGResid, MGInterp represent for?

Best

Yuqi

---- Original message ----
>Date: Tue, 3 Apr 2012 19:19:23 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>  -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>
>   Barry
>
>On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>
>> Hi, Mark,
>> 
>> Thank you so much for your suggestion.
>> 
>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>> 
>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>

From bsmith at mcs.anl.gov  Tue Apr  3 20:08:27 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 3 Apr 2012 20:08:27 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120403185732.APY04392@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
Message-ID: <215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>


   There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.

   Barry

On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:

> Hi, Barry,
> 
> Thank you. If my program converges in two SNES iteration,
> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
> 
> And -pc_mg_log shows the following output
> 
> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
> 
> What are the MGSmooth, MGResid, MGInterp represent for?
> 
> Best
> 
> Yuqi
> 
> ---- Original message ----
>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>> 
>>  Barry
>> 
>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>> 
>>> Hi, Mark,
>>> 
>>> Thank you so much for your suggestion.
>>> 
>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>> 
>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>> 


From behzad.baghapour at gmail.com  Wed Apr  4 09:57:05 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 4 Apr 2012 19:27:05 +0430
Subject: [petsc-users] Calling function from outside of SNES
Message-ID: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>

Dear Developers,

Is there a clear way to call an independent function from outside of SNES
context within the procedure of nonlinear solution?

Regards,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/b377b072/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr  4 10:06:22 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 4 Apr 2012 08:06:22 -0700
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
Message-ID: <CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>

On Wed, Apr 4, 2012 at 07:57, behzad baghapour
<behzad.baghapour at gmail.com>wrote:

> Is there a clear way to call an independent function from outside of SNES
> context within the procedure of nonlinear solution?


What do you want to do? Monitoring, changing the system, etc?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/39af7491/attachment.htm>

From behzad.baghapour at gmail.com  Wed Apr  4 10:13:24 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 4 Apr 2012 19:43:24 +0430
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
Message-ID: <CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>

Actually I want to change the system, applying some modifications to the
solution after each Newton iteration.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/ecddd745/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr  4 10:18:36 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 4 Apr 2012 08:18:36 -0700
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
Message-ID: <CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>

On Wed, Apr 4, 2012 at 08:13, behzad baghapour
<behzad.baghapour at gmail.com>wrote:

> Actually I want to change the system, applying some modifications to the
> solution after each Newton iteration.
>

in petsc-dev

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESLineSearchSetPostCheck.html

or, in 3.2

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESLineSearchSetPostCheck.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/a677328d/attachment.htm>

From behzad.baghapour at gmail.com  Wed Apr  4 10:33:04 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 4 Apr 2012 20:03:04 +0430
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
Message-ID: <CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>

Thanks, Is it work with "-snes_ls basic" ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/7fd95564/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr  4 10:37:28 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 4 Apr 2012 08:37:28 -0700
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
	<CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
Message-ID: <CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>

On Wed, Apr 4, 2012 at 08:33, behzad baghapour
<behzad.baghapour at gmail.com>wrote:

> Thanks, Is it work with "-snes_ls basic" ?
>
>
Yes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/dce7aab6/attachment.htm>

From ywu at culink.Colorado.EDU  Wed Apr  4 12:53:20 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 11:53:20 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
Message-ID: <20120404115320.APY65194@batman.int.colorado.edu>

Thank you.

Can I ask another question? 

In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 

MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280

I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.

Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?

Best

Yuqi




---- Original message ----
>Date: Tue, 3 Apr 2012 20:08:27 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>   There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>
>   Barry
>
>On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>
>> Hi, Barry,
>> 
>> Thank you. If my program converges in two SNES iteration,
>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>> 
>> And -pc_mg_log shows the following output
>> 
>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>> 
>> What are the MGSmooth, MGResid, MGInterp represent for?
>> 
>> Best
>> 
>> Yuqi
>> 
>> ---- Original message ----
>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>> 
>>>  Barry
>>> 
>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Mark,
>>>> 
>>>> Thank you so much for your suggestion.
>>>> 
>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>> 
>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>> 
>

From bsmith at mcs.anl.gov  Wed Apr  4 13:15:45 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Apr 2012 13:15:45 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404115320.APY65194@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
Message-ID: <8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>


On Apr 4, 2012, at 12:53 PM, Yuqi Wu wrote:

> Thank you.
> 
> Can I ask another question? 
> 
> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
> 
> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
> 
> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
> 
> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?

   By default PETSc's multigrid uses ILU smoothing. You can change to SOR with -mg_levels_pc_type sor

   The factorization only needs to done once then triangular solves (MatSolve) are done for each smoothing step. Since there are only two different matrices for the fine level (from the first and second SNES iteration) there only need to be two factorizations. The smoother is run 9 times because there are 9 multigrid iterations.

   Barry

> 
> Best
> 
> Yuqi
> 
> 
> 
> 
> ---- Original message ----
>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>>  There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>> 
>>  Barry
>> 
>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>> 
>>> Hi, Barry,
>>> 
>>> Thank you. If my program converges in two SNES iteration,
>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>> 
>>> And -pc_mg_log shows the following output
>>> 
>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>> 
>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> ---- Original message ----
>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>> 
>>>> Barry
>>>> 
>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Mark,
>>>>> 
>>>>> Thank you so much for your suggestion.
>>>>> 
>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>> 
>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>> 
>> 


From mark.adams at columbia.edu  Wed Apr  4 13:24:28 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Wed, 4 Apr 2012 14:24:28 -0400
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404115320.APY65194@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
Message-ID: <63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>

I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.

Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.

Mark

On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:

> Thank you.
> 
> Can I ask another question? 
> 
> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
> 
> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
> 
> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
> 
> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
> 
> Best
> 
> Yuqi
> 
> 
> 
> 
> ---- Original message ----
>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>>  There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>> 
>>  Barry
>> 
>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>> 
>>> Hi, Barry,
>>> 
>>> Thank you. If my program converges in two SNES iteration,
>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>> 
>>> And -pc_mg_log shows the following output
>>> 
>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>> 
>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> ---- Original message ----
>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>> 
>>>> Barry
>>>> 
>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Mark,
>>>>> 
>>>>> Thank you so much for your suggestion.
>>>>> 
>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>> 
>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>> 
>> 
> 


From ywu at culink.Colorado.EDU  Wed Apr  4 13:28:18 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 12:28:18 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>
Message-ID: <20120404122818.APY68211@batman.int.colorado.edu>

Thank you Barry.

In my application, I set the smoothers to be Richardson iteration with ASM preconditioner, and use exact LU in the ASM subdomain solve. And for the coarse solve, I use the GMRES preconditioned ASM solver, again using LU factorization as the subdomain solve.

Yes, there are two different matrices for fine level, but since I am having a coarse grid, it should be two different matrices for coarse level. So I believe it needs 4 LU factorizations. But based on my -info output, it seems only 3 LU factorization are called in the program. I am confused about that.

Best

Yuqi

---- Original message ----
>Date: Wed, 4 Apr 2012 13:15:45 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 4, 2012, at 12:53 PM, Yuqi Wu wrote:
>
>> Thank you.
>> 
>> Can I ask another question? 
>> 
>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>> 
>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>> 
>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>> 
>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>
>   By default PETSc's multigrid uses ILU smoothing. You can change to SOR with -mg_levels_pc_type sor
>
>   The factorization only needs to done once then triangular solves (MatSolve) are done for each smoothing step. Since there are only two different matrices for the fine level (from the first and second SNES iteration) there only need to be two factorizations. The smoother is run 9 times because there are 9 multigrid iterations.
>
>   Barry
>
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>>  There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>> 
>>>  Barry
>>> 
>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Barry,
>>>> 
>>>> Thank you. If my program converges in two SNES iteration,
>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>> 
>>>> And -pc_mg_log shows the following output
>>>> 
>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>> 
>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> ---- Original message ----
>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Mark,
>>>>>> 
>>>>>> Thank you so much for your suggestion.
>>>>>> 
>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>> 
>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>> 
>>> 
>

From ywu at culink.Colorado.EDU  Wed Apr  4 13:34:52 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 12:34:52 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
Message-ID: <20120404123452.APY68748@batman.int.colorado.edu>

Thanks, Adam.

Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.

In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.


[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines

[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines

[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines

So I believe that the fine grid smoother is not getting refactored in the second SNES solve.

Best

Yuqi



---- Original message ----
>Date: Wed, 4 Apr 2012 14:24:28 -0400
>From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>
>Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>
>Mark
>
>On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>
>> Thank you.
>> 
>> Can I ask another question? 
>> 
>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>> 
>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>> 
>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>> 
>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>>  There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>> 
>>>  Barry
>>> 
>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Barry,
>>>> 
>>>> Thank you. If my program converges in two SNES iteration,
>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>> 
>>>> And -pc_mg_log shows the following output
>>>> 
>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>> 
>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> ---- Original message ----
>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Mark,
>>>>>> 
>>>>>> Thank you so much for your suggestion.
>>>>>> 
>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>> 
>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>> 
>>> 
>> 
>

From ywu at culink.Colorado.EDU  Wed Apr  4 13:40:11 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 12:40:11 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>
Message-ID: <20120404124011.APY69218@batman.int.colorado.edu>

Can I also ask another questions related to the twolevel preconditioner? I have also try to use PCComposite to create two level ASM preconditioner instead of using PCMG. Since I am not able to use DA, which way is better to use in create a two level preconditioner?

If I use the PCMG approach, it seems that the LU factorization in the up smoother can be avoided, only the LU factorization in the down smoother are need to be computed. But if I have the following precondition created with PCComposite

ierr = PCSetType(finepc,PCCOMPOSITE);CHKERRQ(ierr);
/* smooth down preconditioner */
ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);
/* coarse solve */
ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);
/* smooth up preconditioner */
ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);

How can I save the LU factorization in the smooth up stage, if I want to use the same precondition in smooth up and smooth down?

Thank you.


Yuqi

---- Original message ----
>Date: Wed, 4 Apr 2012 13:15:45 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 4, 2012, at 12:53 PM, Yuqi Wu wrote:
>
>> Thank you.
>> 
>> Can I ask another question? 
>> 
>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>> 
>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>> 
>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>> 
>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>
>   By default PETSc's multigrid uses ILU smoothing. You can change to SOR with -mg_levels_pc_type sor
>
>   The factorization only needs to done once then triangular solves (MatSolve) are done for each smoothing step. Since there are only two different matrices for the fine level (from the first and second SNES iteration) there only need to be two factorizations. The smoother is run 9 times because there are 9 multigrid iterations.
>
>   Barry
>
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>>  There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>> 
>>>  Barry
>>> 
>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Barry,
>>>> 
>>>> Thank you. If my program converges in two SNES iteration,
>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>> 
>>>> And -pc_mg_log shows the following output
>>>> 
>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>> 
>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> ---- Original message ----
>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Mark,
>>>>>> 
>>>>>> Thank you so much for your suggestion.
>>>>>> 
>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>> 
>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Wed Apr  4 13:48:14 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Apr 2012 13:48:14 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404124011.APY69218@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<8CF53F52-A33B-4BB0-B093-3D88B119DEBE@mcs.anl.gov>
	<20120404124011.APY69218@batman.int.colorado.edu>
Message-ID: <BB9DCE4E-4B7D-450A-9EDA-72AFF1E0AD49@mcs.anl.gov>


   Use the PCMG.


On Apr 4, 2012, at 1:40 PM, Yuqi Wu wrote:

> Can I also ask another questions related to the twolevel preconditioner? I have also try to use PCComposite to create two level ASM preconditioner instead of using PCMG. Since I am not able to use DA, which way is better to use in create a two level preconditioner?
> 
> If I use the PCMG approach, it seems that the LU factorization in the up smoother can be avoided, only the LU factorization in the down smoother are need to be computed. But if I have the following precondition created with PCComposite
> 
> ierr = PCSetType(finepc,PCCOMPOSITE);CHKERRQ(ierr);
> /* smooth down preconditioner */
> ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);
> /* coarse solve */
> ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);
> /* smooth up preconditioner */
> ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);
> 
> How can I save the LU factorization in the smooth up stage, if I want to use the same precondition in smooth up and smooth down?
> 
> Thank you.
> 
> 
> Yuqi
> 
> ---- Original message ----
>> Date: Wed, 4 Apr 2012 13:15:45 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> On Apr 4, 2012, at 12:53 PM, Yuqi Wu wrote:
>> 
>>> Thank you.
>>> 
>>> Can I ask another question? 
>>> 
>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>> 
>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>> 
>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>> 
>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>> 
>>  By default PETSc's multigrid uses ILU smoothing. You can change to SOR with -mg_levels_pc_type sor
>> 
>>  The factorization only needs to done once then triangular solves (MatSolve) are done for each smoothing step. Since there are only two different matrices for the fine level (from the first and second SNES iteration) there only need to be two factorizations. The smoother is run 9 times because there are 9 multigrid iterations.
>> 
>>  Barry
>> 
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>> 
>>>> Barry
>>>> 
>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Barry,
>>>>> 
>>>>> Thank you. If my program converges in two SNES iteration,
>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>> 
>>>>> And -pc_mg_log shows the following output
>>>>> 
>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>> 
>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Hi, Mark,
>>>>>>> 
>>>>>>> Thank you so much for your suggestion.
>>>>>>> 
>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>> 
>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>> 
>>>> 
>> 


From bsmith at mcs.anl.gov  Wed Apr  4 13:55:55 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Apr 2012 13:55:55 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404123452.APY68748@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
Message-ID: <64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>


  Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?

   When you run the program with -info it will print information of the form: (run on one process to make life simple)

Setting PC with identical preconditioner\
Setting up new PC
Setting up PC with same nonzero pattern\
Setting up PC with different nonzero pattern\n

   How many, and exactly what messages of this form are you getting?

    When all else fails you can run the program in the debugger to track what is happening and why. 
  
  Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.

   First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 

   Second thing, if it is then why is it not triggering the new numerical factorization.


    Barry

On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:

> Thanks, Adam.
> 
> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
> 
> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
> 
> 
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
> 
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
> 
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
> 
> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
> 
> Best
> 
> Yuqi
> 
> 
> 
> ---- Original message ----
>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>> 
>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>> 
>> Mark
>> 
>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>> 
>>> Thank you.
>>> 
>>> Can I ask another question? 
>>> 
>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>> 
>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>> 
>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>> 
>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>> 
>>>> Barry
>>>> 
>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Barry,
>>>>> 
>>>>> Thank you. If my program converges in two SNES iteration,
>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>> 
>>>>> And -pc_mg_log shows the following output
>>>>> 
>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>> 
>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Hi, Mark,
>>>>>>> 
>>>>>>> Thank you so much for your suggestion.
>>>>>>> 
>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>> 
>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>> 
>>>> 
>>> 
>> 


From ywu at culink.Colorado.EDU  Wed Apr  4 14:21:46 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 13:21:46 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
Message-ID: <20120404132146.APY73074@batman.int.colorado.edu>

I have three SYMBOLIC factorization and three SYMBOLIC factorization done in my program.

I didn't use the SAME_NONZERO_PATTERN is provided in the compute Jacobian routine. If I use the flag then it will be two SYMBOLIC factorization and three SYMBOLIC factorization done in my program.

For -info output
I got none message with "Setting PC with identical preconditioner"

I got five message with "Setting up new PC"

I got none message with "Setting up PC with same nonzero pattern"

I got three message with "Setting up PC with different nonzero pattern"


For messages with "Setting up new PC", I got the following output 

[0] PCSetUp(): Setting up new PC
[0] PCSetUp_MG(): Using outer operators to define finest grid operator 
  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.
[0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
[0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
[0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Reallocs 1; Fill ratio: given 1 needed 1.43239.
[0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Use MatPtAP(A,P,MatReuse,1.43239,&C) for best performance.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
[0] PCSetUp(): Setting up new PC
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] VecScatterCreate(): Special case: sequential vector general to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11585 X 11585; storage space: 0 unneeded,458097 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 168
[0] Mat_CheckInode(): Found 8271 nodes of 11585. Limit used: 5. Using Inode routines
[0] PCSetUp(): Setting up new PC
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] VecScatterCreate(): Special case: sequential vector general to stride
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
[0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
    0 KSP Residual norm 1.014990964599e+02 
[0] PCSetUp(): Setting up new PC
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
[0] KSPDefaultConverged(): Linear solver has converged. Residual norm 5.755920981112e-13 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 5.479558824115e+02 at iteration 1
[0] PCSetUp(): Setting up new PC
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
        Residual norms for coarse_ solve.
        0 KSP Residual norm 5.698312810532e-16 

For the message with "Setting up PC with different nonzero pattern", I got the following outputs

[0] PCSetUp(): Setting up PC with different nonzero pattern
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
[0] PCSetUp(): Setting up PC with different nonzero pattern
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
[0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
    0 KSP Residual norm 9.922628060272e-05 
[0] PCSetUp(): Setting up PC with different nonzero pattern
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
[0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
[0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
[0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
[0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
[0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines

Thank you so much for your help.

Yuqi


>Setting up new PC
>Setting up PC with same nonzero pattern\
>Setting up PC with different nonzero pattern\n


---- Original message ----
>Date: Wed, 4 Apr 2012 13:55:55 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>  Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>
>   When you run the program with -info it will print information of the form: (run on one process to make life simple)
>
>Setting PC with identical preconditioner\
>Setting up new PC
>Setting up PC with same nonzero pattern\
>Setting up PC with different nonzero pattern\n
>
>   How many, and exactly what messages of this form are you getting?
>
>    When all else fails you can run the program in the debugger to track what is happening and why. 
>  
>  Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>
>   First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>
>   Second thing, if it is then why is it not triggering the new numerical factorization.
>
>
>    Barry
>
>On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>
>> Thanks, Adam.
>> 
>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>> 
>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>> 
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>> 
>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>> 
>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>> 
>>> Mark
>>> 
>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>> 
>>>> Thank you.
>>>> 
>>>> Can I ask another question? 
>>>> 
>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>> 
>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>> 
>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>> 
>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Barry,
>>>>>> 
>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>> 
>>>>>> And -pc_mg_log shows the following output
>>>>>> 
>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>> 
>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Hi, Mark,
>>>>>>>> 
>>>>>>>> Thank you so much for your suggestion.
>>>>>>>> 
>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>> 
>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>> 
>>>>> 
>>>> 
>>> 
>

From mark.adams at columbia.edu  Wed Apr  4 15:22:45 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Wed, 4 Apr 2012 16:22:45 -0400
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404132146.APY73074@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404132146.APY73074@batman.int.colorado.edu>
Message-ID: <291C0F1C-F7C4-465F-8EC5-A1AA4B71A40C@columbia.edu>


On Apr 4, 2012, at 3:21 PM, Yuqi Wu wrote:

> I have three SYMBOLIC factorization and three SYMBOLIC factorization done in my program.
> 
> I didn't use the SAME_NONZERO_PATTERN is provided in the compute Jacobian routine. If I use the flag then it will be two SYMBOLIC factorization and three SYMBOLIC factorization done in my program.

2 SYMB and 3 (or it seems 4 to me) NUMERIC factorizations.

Mark

> 
> For -info output
> I got none message with "Setting PC with identical preconditioner"
> 
> I got five message with "Setting up new PC"
> 
> I got none message with "Setting up PC with same nonzero pattern"
> 
> I got three message with "Setting up PC with different nonzero pattern"
> 
> 
> For messages with "Setting up new PC", I got the following output 
> 
> [0] PCSetUp(): Setting up new PC
> [0] PCSetUp_MG(): Using outer operators to define finest grid operator 
>  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.
> [0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
> [0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
> [0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Reallocs 1; Fill ratio: given 1 needed 1.43239.
> [0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Use MatPtAP(A,P,MatReuse,1.43239,&C) for best performance.
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] VecScatterCreate(): Special case: sequential vector general to stride
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11585 X 11585; storage space: 0 unneeded,458097 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 168
> [0] Mat_CheckInode(): Found 8271 nodes of 11585. Limit used: 5. Using Inode routines
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] VecScatterCreate(): Special case: sequential vector general to stride
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
>    0 KSP Residual norm 1.014990964599e+02 
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
> [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 5.755920981112e-13 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 5.479558824115e+02 at iteration 1
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>        Residual norms for coarse_ solve.
>        0 KSP Residual norm 5.698312810532e-16 
> 
> For the message with "Setting up PC with different nonzero pattern", I got the following outputs
> 
> [0] PCSetUp(): Setting up PC with different nonzero pattern
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
> [0] PCSetUp(): Setting up PC with different nonzero pattern
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
>    0 KSP Residual norm 9.922628060272e-05 
> [0] PCSetUp(): Setting up PC with different nonzero pattern
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
> 
> Thank you so much for your help.
> 
> Yuqi
> 
> 
>> Setting up new PC
>> Setting up PC with same nonzero pattern\
>> Setting up PC with different nonzero pattern\n
> 
> 
> ---- Original message ----
>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>> 
>>  When you run the program with -info it will print information of the form: (run on one process to make life simple)
>> 
>> Setting PC with identical preconditioner\
>> Setting up new PC
>> Setting up PC with same nonzero pattern\
>> Setting up PC with different nonzero pattern\n
>> 
>>  How many, and exactly what messages of this form are you getting?
>> 
>>   When all else fails you can run the program in the debugger to track what is happening and why. 
>> 
>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>> 
>>  First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>> 
>>  Second thing, if it is then why is it not triggering the new numerical factorization.
>> 
>> 
>>   Barry
>> 
>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>> 
>>> Thanks, Adam.
>>> 
>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>> 
>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>> 
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>> 
>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>> 
>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>> 
>>>> Mark
>>>> 
>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>> 
>>>>> Thank you.
>>>>> 
>>>>> Can I ask another question? 
>>>>> 
>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>> 
>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>> 
>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>> 
>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Hi, Barry,
>>>>>>> 
>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>> 
>>>>>>> And -pc_mg_log shows the following output
>>>>>>> 
>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>> 
>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Hi, Mark,
>>>>>>>>> 
>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>> 
>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>> 
>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
> 


From bsmith at mcs.anl.gov  Wed Apr  4 15:28:57 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Apr 2012 15:28:57 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <291C0F1C-F7C4-465F-8EC5-A1AA4B71A40C@columbia.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404132146.APY73074@batman.int.colorado.edu>
	<291C0F1C-F7C4-465F-8EC5-A1AA4B71A40C@columbia.edu>
Message-ID: <6F47F285-8D26-49A4-B0F9-92FD8EE5CB94@mcs.anl.gov>


On Apr 4, 2012, at 3:22 PM, Mark F. Adams wrote:

> 
> On Apr 4, 2012, at 3:21 PM, Yuqi Wu wrote:
> 
>> I have three SYMBOLIC factorization and three SYMBOLIC factorization done in my program.
>> 
>> I didn't use the SAME_NONZERO_PATTERN is provided in the compute Jacobian routine. If I use the flag then it will be two SYMBOLIC factorization and three SYMBOLIC factorization done in my program.
> 
> 2 SYMB and 3 (or it seems 4 to me) NUMERIC factorizations.

  I also think it should be 4 NUMERIC factorization.  To the debugger :-)

 Barry

> 
> Mark
> 
>> 
>> For -info output
>> I got none message with "Setting PC with identical preconditioner"
>> 
>> I got five message with "Setting up new PC"
>> 
>> I got none message with "Setting up PC with same nonzero pattern"
>> 
>> I got three message with "Setting up PC with different nonzero pattern"
>> 
>> 
>> For messages with "Setting up new PC", I got the following output 
>> 
>> [0] PCSetUp(): Setting up new PC
>> [0] PCSetUp_MG(): Using outer operators to define finest grid operator 
>> because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.
>> [0] MatGetSymbolicTranspose_SeqAIJ(): Getting Symbolic Transpose.
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
>> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
>> [0] MatRestoreSymbolicTranspose_SeqAIJ(): Restoring Symbolic Transpose.
>> [0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Reallocs 1; Fill ratio: given 1 needed 1.43239.
>> [0] MatPtAPSymbolic_SeqAIJ_SeqAIJ(): Use MatPtAP(A,P,MatReuse,1.43239,&C) for best performance.
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
>> [0] PCSetUp(): Setting up new PC
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] VecScatterCreate(): Special case: sequential vector general to stride
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11585 X 11585; storage space: 0 unneeded,458097 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 168
>> [0] Mat_CheckInode(): Found 8271 nodes of 11585. Limit used: 5. Using Inode routines
>> [0] PCSetUp(): Setting up new PC
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] VecScatterCreate(): Special case: sequential vector general to stride
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
>> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
>>  0 KSP Residual norm 1.014990964599e+02 
>> [0] PCSetUp(): Setting up new PC
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>> [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 5.755920981112e-13 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 5.479558824115e+02 at iteration 1
>> [0] PCSetUp(): Setting up new PC
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>      Residual norms for coarse_ solve.
>>      0 KSP Residual norm 5.698312810532e-16 
>> 
>> For the message with "Setting up PC with different nonzero pattern", I got the following outputs
>> 
>> [0] PCSetUp(): Setting up PC with different nonzero pattern
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
>> [0] PCSetUp(): Setting up PC with different nonzero pattern
>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 4186 X 4186; storage space: 0 unneeded,656174 used
>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 410
>> [0] Mat_CheckInode(): Found 1794 nodes of 4186. Limit used: 5. Using Inode routines
>>  0 KSP Residual norm 9.922628060272e-05 
>> [0] PCSetUp(): Setting up PC with different nonzero pattern
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>> 
>> Thank you so much for your help.
>> 
>> Yuqi
>> 
>> 
>>> Setting up new PC
>>> Setting up PC with same nonzero pattern\
>>> Setting up PC with different nonzero pattern\n
>> 
>> 
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>> 
>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>> 
>>> Setting PC with identical preconditioner\
>>> Setting up new PC
>>> Setting up PC with same nonzero pattern\
>>> Setting up PC with different nonzero pattern\n
>>> 
>>> How many, and exactly what messages of this form are you getting?
>>> 
>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>> 
>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>> 
>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>> 
>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>> 
>>> 
>>> Barry
>>> 
>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>> 
>>>> Thanks, Adam.
>>>> 
>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>> 
>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>> 
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>> 
>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>> 
>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>> 
>>>>> Mark
>>>>> 
>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Thank you.
>>>>>> 
>>>>>> Can I ask another question? 
>>>>>> 
>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>> 
>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>> 
>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>> 
>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Hi, Barry,
>>>>>>>> 
>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>> 
>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>> 
>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>> 
>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Mark,
>>>>>>>>>> 
>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>> 
>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>> 
>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
> 


From andrew.spott at gmail.com  Wed Apr  4 16:47:28 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 4 Apr 2012 15:47:28 -0600
Subject: [petsc-users] Most efficient way to transfer a vector to a single
	processor
Message-ID: <3ECDFD35-3EAA-4B51-80B9-14474FE7C136@gmail.com>

I'm interested in moving a vector to a single processor (so I can output it in some custom way), but I can't find a simple way to do that.

Is the easiest way to output the parallel vector to a file and then read it in to a single processor vector?

Also, an unrelated question:  If I have a vector that I'm using as the result of a matrix vector product, but it is initially empty, do I need to initialize it in any way?  or can I just create it and run with it:

? create x and A ...
VecCreate(world, &b);
MatGetVecs(A,&x,&b);

? do stuff with x and A ...

MatMult(A,x,b);

Thanks for the help,

-Andrew

From knepley at gmail.com  Wed Apr  4 16:51:15 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 4 Apr 2012 16:51:15 -0500
Subject: [petsc-users] Most efficient way to transfer a vector to a
	single processor
In-Reply-To: <3ECDFD35-3EAA-4B51-80B9-14474FE7C136@gmail.com>
References: <3ECDFD35-3EAA-4B51-80B9-14474FE7C136@gmail.com>
Message-ID: <CAMYG4GkQdLDZJidGqHS3fSMVkfpXPCr17DmGA0rRjLooG0Qb-g@mail.gmail.com>

On Wed, Apr 4, 2012 at 4:47 PM, Andrew Spott <andrew.spott at gmail.com> wrote:

> I'm interested in moving a vector to a single processor (so I can output
> it in some custom way), but I can't find a simple way to do that.
>
> Is the easiest way to output the parallel vector to a file and then read
> it in to a single processor vector?
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Vec/VecScatterCreateToZero.html


> Also, an unrelated question:  If I have a vector that I'm using as the
> result of a matrix vector product, but it is initially empty, do I need to
> initialize it in any way?  or can I just create it and run with it:
>
> ? create x and A ...
> VecCreate(world, &b);
>

This create is wrong, that Vec will be overwritten.


> MatGetVecs(A,&x,&b);
>

You can jsut use these vectors.

   Matt


> ? do stuff with x and A ...
>
> MatMult(A,x,b);
>
> Thanks for the help,
>
> -Andrew




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/8ea1870b/attachment.htm>

From dharmareddy84 at gmail.com  Wed Apr  4 18:51:59 2012
From: dharmareddy84 at gmail.com (Dharmendar Reddy)
Date: Wed, 4 Apr 2012 18:51:59 -0500
Subject: [petsc-users] SNESConvergedTest In Fortran
Message-ID: <CAOwTPdfczQs+8BEFkXK+9CMbzzqLk+9V4KfhRJTYK7WFg7RrTQ@mail.gmail.com>

Hello,
        How should i implement the function below in Fortran

    PetscErrorCode
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode>
func (SNES <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNES.html#SNES>
snes,PetscInt <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
it,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
xnorm,PetscReal
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
gnorm,
                         PetscReal
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
f,SNESConvergedReason
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESConvergedReason.html#SNESConvergedReason>
*reason,void *cctx)

        should it be a subroutine

    subroutine func (SNES
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNES.html#SNES>
snes,PetscInt <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
it,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
xnorm,PetscReal
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
gnorm,
          PetscReal
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal>
f,SNESConvergedReason
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESConvergedReason.html#SNESConvergedReason>
*reason,void *cctx, PetscErrorCode
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode>
ierr)

       or a function whose result is of type PetscErrorCode

thanks
Reddy

-- 
-----------------------------------------------------
Dharmendar Reddy Palle
Graduate Student
Microelectronics Research center,
University of Texas at Austin,
10100 Burnet Road, Bldg. 160
MER 2.608F, TX 78758-4445
e-mail: dharmareddy84 at gmail.com
Phone: +1-512-350-9082
United States of America.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/b998975c/attachment.htm>

From knepley at gmail.com  Wed Apr  4 18:53:29 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 4 Apr 2012 18:53:29 -0500
Subject: [petsc-users] SNESConvergedTest In Fortran
In-Reply-To: <CAOwTPdfczQs+8BEFkXK+9CMbzzqLk+9V4KfhRJTYK7WFg7RrTQ@mail.gmail.com>
References: <CAOwTPdfczQs+8BEFkXK+9CMbzzqLk+9V4KfhRJTYK7WFg7RrTQ@mail.gmail.com>
Message-ID: <CAMYG4G=Bp8pYb_6z_0p44Yz7vE+hstLCEvW-MGSLymZNQ2JfMw@mail.gmail.com>

On Wed, Apr 4, 2012 at 6:51 PM, Dharmendar Reddy <dharmareddy84 at gmail.com>wrote:

> Hello,
>         How should i implement the function below in Fortran
>
>     PetscErrorCode <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode> func (SNES <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNES.html#SNES> snes,PetscInt <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> it,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> xnorm,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> gnorm,
>
>                          PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> f,SNESConvergedReason <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESConvergedReason.html#SNESConvergedReason> *reason,void *cctx)
>
>         should it be a subroutine
>
>     subroutine func (SNES <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNES.html#SNES> snes,PetscInt <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> it,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> xnorm,PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> gnorm,
>
>           PetscReal <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscReal.html#PetscReal> f,SNESConvergedReason <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESConvergedReason.html#SNESConvergedReason> *reason,void *cctx, PetscErrorCode <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode> ierr)
>
>        or a function whose result is of type PetscErrorCode
>

Subroutine.

   Matt


> thanks
> Reddy
>
> --
> -----------------------------------------------------
> Dharmendar Reddy Palle
> Graduate Student
> Microelectronics Research center,
> University of Texas at Austin,
> 10100 Burnet Road, Bldg. 160
> MER 2.608F, TX 78758-4445
> e-mail: dharmareddy84 at gmail.com
> Phone: +1-512-350-9082
> United States of America.
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120404/6bbac516/attachment.htm>

From ywu at culink.Colorado.EDU  Wed Apr  4 19:45:55 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 18:45:55 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
Message-ID: <20120404184555.APY97983@batman.int.colorado.edu>

---- Original message ----
>Date: Wed, 4 Apr 2012 13:55:55 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>  Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>
>   When you run the program with -info it will print information of the form: (run on one process to make life simple)
>
>Setting PC with identical preconditioner\
>Setting up new PC
>Setting up PC with same nonzero pattern\
>Setting up PC with different nonzero pattern\n
>
>   How many, and exactly what messages of this form are you getting?
>
>    When all else fails you can run the program in the debugger to track what is happening and why. 
>  
>  Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>
>   First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>
>   Second thing, if it is then why is it not triggering the new numerical factorization.

I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 

But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?

Best

Yuqi


>
>
>    Barry
>
>On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>
>> Thanks, Adam.
>> 
>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>> 
>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>> 
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>> 
>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>> 
>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>> 
>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>> 
>>> Mark
>>> 
>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>> 
>>>> Thank you.
>>>> 
>>>> Can I ask another question? 
>>>> 
>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>> 
>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>> 
>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>> 
>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Barry,
>>>>>> 
>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>> 
>>>>>> And -pc_mg_log shows the following output
>>>>>> 
>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>> 
>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Hi, Mark,
>>>>>>>> 
>>>>>>>> Thank you so much for your suggestion.
>>>>>>>> 
>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>> 
>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>> 
>>>>> 
>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Wed Apr  4 19:51:23 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 4 Apr 2012 19:51:23 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404184555.APY97983@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
Message-ID: <49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>


  Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 

   When it gets to the lines 

if (pc->ops->setup) {
    ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
  }

don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"

> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 

   It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?

  Barry



On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:

> ---- Original message ----
>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>> 
>>  When you run the program with -info it will print information of the form: (run on one process to make life simple)
>> 
>> Setting PC with identical preconditioner\
>> Setting up new PC
>> Setting up PC with same nonzero pattern\
>> Setting up PC with different nonzero pattern\n
>> 
>>  How many, and exactly what messages of this form are you getting?
>> 
>>   When all else fails you can run the program in the debugger to track what is happening and why. 
>> 
>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>> 
>>  First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>> 
>>  Second thing, if it is then why is it not triggering the new numerical factorization.
> 
> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
> 
> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
> 
> Best
> 
> Yuqi
> 
> 
>> 
>> 
>>   Barry
>> 
>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>> 
>>> Thanks, Adam.
>>> 
>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>> 
>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>> 
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>> 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>> 
>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>> 
>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>> 
>>>> Mark
>>>> 
>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>> 
>>>>> Thank you.
>>>>> 
>>>>> Can I ask another question? 
>>>>> 
>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>> 
>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>> 
>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>> 
>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Hi, Barry,
>>>>>>> 
>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>> 
>>>>>>> And -pc_mg_log shows the following output
>>>>>>> 
>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>> 
>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Hi, Mark,
>>>>>>>>> 
>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>> 
>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>> 
>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 


From ywu at culink.Colorado.EDU  Wed Apr  4 23:55:43 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Wed,  4 Apr 2012 22:55:43 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
Message-ID: <20120404225543.APZ06409@batman.int.colorado.edu>

Thank you Barry.

I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.

In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.

I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?

  /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
  /* so use those from global PC */
  /* Is this what we always want? What if user wants to keep old one? */
  ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
  ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
  ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
  if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
    ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
    ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
  }

Thank you.

Yuqi



---- Original message ----
>Date: Wed, 4 Apr 2012 19:51:23 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>  Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>
>   When it gets to the lines 
>
>if (pc->ops->setup) {
>    ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>  }
>
>don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>
>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>
>   It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>
>  Barry
>
>
>
>On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>> 
>>>  When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>> 
>>> Setting PC with identical preconditioner\
>>> Setting up new PC
>>> Setting up PC with same nonzero pattern\
>>> Setting up PC with different nonzero pattern\n
>>> 
>>>  How many, and exactly what messages of this form are you getting?
>>> 
>>>   When all else fails you can run the program in the debugger to track what is happening and why. 
>>> 
>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>> 
>>>  First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>> 
>>>  Second thing, if it is then why is it not triggering the new numerical factorization.
>> 
>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>> 
>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>>> 
>>> 
>>>   Barry
>>> 
>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>> 
>>>> Thanks, Adam.
>>>> 
>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>> 
>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>> 
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>> 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>> 
>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>> 
>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>> 
>>>>> Mark
>>>>> 
>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Thank you.
>>>>>> 
>>>>>> Can I ask another question? 
>>>>>> 
>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>> 
>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>> 
>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>> 
>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Hi, Barry,
>>>>>>>> 
>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>> 
>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>> 
>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>> 
>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Mark,
>>>>>>>>>> 
>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>> 
>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>> 
>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>

From behzad.baghapour at gmail.com  Thu Apr  5 03:24:50 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 5 Apr 2012 12:54:50 +0430
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
	<CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
	<CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>
Message-ID: <CA+jOtDKNUXsG9ZkhAEbrM21AZFoTcE8bq97s0sRTyNOjrn1DtA@mail.gmail.com>

I already applied PostCheck routine for SNES but this function is not
called during the Newton iteration since "print to screen alarm" is not
demonstrated. Is there a flag to activate postcheck, running code with
prescribed option, etc?

Thanks a lot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/47336cab/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 06:43:16 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 11:43:16 +0000
Subject: [petsc-users] question about MatMatMultTranspose
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>

Hi,

I would like to use PETSc perform matrix-matrix multiplication C=A*B^T. I notice that MatMatMultTranspose could calculate C=A^T*B, but which is not I want.

Does anyone know how to perform C=A*B^T using PETSc (I know I could first calculate B^T and call MatMatMult with A and B^T, but it might not be efficient, is it?)? Thank you in advance.

Cheers

Gao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/c6c9387d/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr  5 06:54:26 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 05 Apr 2012 13:54:26 +0200
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <4F184AF9.2000608@gfz-potsdam.de>
References: <4F18290F.3080301@gfz-potsdam.de> <4F184AF9.2000608@gfz-potsdam.de>
Message-ID: <4F7D87F2.9050408@gfz-potsdam.de>

Hello,

The inconsistency is still there. I pulled petsc-dev today and came 
across with the same problem:

src/vec/vec/impls/seq/bvec2.c
239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
240:    *z = PetscSqrtReal(*z);

This doesn't work with Intel MKL. However in one of the mailing list 
discussions it was said there is now check for that.
This check didn't work for me apparently. Any ideas why?

Regards,
Alexander

On 19.01.2012 17:55, Alexander Grayver wrote:
> This (dirty) patch solves the problem:
>
> diff -r a3e9ca59ab58 src/vec/vec/impls/seq/bvec2.c
> --- a/src/vec/vec/impls/seq/bvec2.c     Tue Jan 17 22:04:05 2012 -0600
> +++ b/src/vec/vec/impls/seq/bvec2.c     Thu Jan 19 17:28:39 2012 +0100
> @@ -232,12 +232,13 @@
>    PetscErrorCode    ierr;
>    PetscInt          n = xin->map->n;
>    PetscBLASInt      one = 1, bn = PetscBLASIntCast(n);
> +  PetscScalar cnorm;
>
>    PetscFunctionBegin;
>    if (type == NORM_2 || type == NORM_FROBENIUS) {
>      ierr = VecGetArrayRead(xin,&xx);CHKERRQ(ierr);
> -    *z = BLASdot_(&bn,xx,&one,xx,&one);
> -    *z = PetscSqrtReal(*z);
> +    zdotc(&cnorm,&bn,xx,&one,xx,&one);
> +    *z = PetscSqrtReal(PetscAbsScalar(cnorm));
>      ierr = VecRestoreArrayRead(xin,&xx);CHKERRQ(ierr);
>      ierr = PetscLogFlops(PetscMax(2.0*n-1,0.0));CHKERRQ(ierr);
>    } else if (type == NORM_INFINITY) {
>
> The same is applied to mpi vector implementation from 
> /petsc-dev/src/vec/vec/impls/mpi/pvec2.c
> Of course it works only if one uses Intel MKL BLAS/LAPACK and double 
> complex arithmetics. Which is my case.
>
> On 19.01.2012 15:30, Alexander Grayver wrote:
>> Dear petsc team,
>>
>> I've been struggling with this problem for a long time and seeking 
>> for your advice now.
>> Let's take simple petsc program:
>>
>> #include <petscksp.h>
>> int main(int argc,char **args)
>> {
>>   Vec                b;
>>   PetscReal      norm;
>>
>>   PetscInitialize(&argc,&args,(char *)0,NULL);
>>
>>   VecCreateSeq(PETSC_COMM_SELF,100,&b);
>>   VecSet(b,2.0);
>>   VecNorm(b,NORM_1_AND_2,&norm);
>>
>>   return 0;
>> }
>>
>> This program works well if I compile petsc with non-mkl blas/lapack.
>> However, if I compile petsc with mkl blas/lapack this program crashes:
>>
>> zdotc,                   FP=7fff8f121e30
>> VecNorm_Seq,    FP=7fff8f121f90
>> VecNorm_Seq,    FP=7fff8f1220f0
>> VecNorm,            FP=7fff8f122230
>> main,                   FP=7fff8f122290
>>
>> It crashes in zdotc. You call this routine as following:
>>
>> *z = BLASdot_(&bn,xx,&one,xx,&one);
>>
>> When I look at the zdotc interface in mkl.h I see:
>> void    ZDOTC(MKL_Complex16 *pres, const MKL_INT *n, const 
>> MKL_Complex16 *x, const MKL_INT *incx, const  MKL_Complex16 *y, const 
>> MKL_INT *incy);
>>
>> I also found example here: 
>> http://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-blas-cblas-and-lapack-compilinglinking-functions-fortran-and-cc-calls/
>> There are 6 input parameters. But in "classical" BLAS implementation 
>> 5 input and 1 output.
>>
>> For example, NORM_1 works well since interface to dzasum is the same.
>>
>> Any ideas?
>>
>
>


-- 
Regards,
Alexander


From behzad.baghapour at gmail.com  Thu Apr  5 07:02:03 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 5 Apr 2012 16:32:03 +0430
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CA+jOtDKNUXsG9ZkhAEbrM21AZFoTcE8bq97s0sRTyNOjrn1DtA@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
	<CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
	<CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>
	<CA+jOtDKNUXsG9ZkhAEbrM21AZFoTcE8bq97s0sRTyNOjrn1DtA@mail.gmail.com>
Message-ID: <CA+jOtDK2LXE2uJKczwt185t-qPZzH3d5XKCCgujS=O1z_sogjw@mail.gmail.com>

I could solve the problem by shifting the postCheck below the
SNESSetFromOptions but still have another problem with passing defined
context:

All my calcualtions are saved into global FieldContext including flow-field
variables. I passed it to PostCheck. I could apply my desired change on the
context BUT the results came back to the original values (before postCheck
function) and therefore my efferts to modify the Newton procedure did not
achieved.

How should I handle it?

Thanks again.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/35d3fc14/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr  5 07:09:45 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 05:09:45 -0700
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
Message-ID: <CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>

On Thu, Apr 5, 2012 at 04:43, Gao Bin <bin.gao at uit.no> wrote:

> I would like to use PETSc perform matrix-matrix multiplication C=A*B^T. I
> notice that MatMatMultTranspose could calculate C=A^T*B, but which is not I
> want.
>
> Does anyone know how to perform C=A*B^T using PETSc (I know I could first
> calculate B^T and call MatMatMult with A and B^T, but it might not be
> efficient, is it?)? Thank you in advance.
>

Use petsc-dev. That routine is renamed (
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatTransposeMatMult.html)
and the one you want is

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/297d05f9/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Thu Apr  5 07:13:35 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 05:13:35 -0700
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <4F7D87F2.9050408@gfz-potsdam.de>
References: <4F18290F.3080301@gfz-potsdam.de> <4F184AF9.2000608@gfz-potsdam.de>
	<4F7D87F2.9050408@gfz-potsdam.de>
Message-ID: <CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>

On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <agrayver at gfz-potsdam.de>wrote:

> The inconsistency is still there. I pulled petsc-dev today and came across
> with the same problem:
>
> src/vec/vec/impls/seq/bvec2.c
> 239:    *z = PetscRealPart(BLASdot_(&bn,xx,**&one,xx,&one));
> 240:    *z = PetscSqrtReal(*z);
>
> This doesn't work with Intel MKL. However in one of the mailing list
> discussions it was said there is now check for that.
> This check didn't work for me apparently. Any ideas why?
>

As always, we need configure.log.

In include/petscblaslapack_uscore.h (and other places), you will find the
following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not
found by configure.

#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
EXTERN_C_BEGIN
extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const
PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const
PetscBLASInt *sy) {
  PetscScalar tmpz;
  zdotc_(&tmpz,n,x,sx,y,sy);
  return tmpz;
}
EXTERN_C_END
#else
#  define BLASdot_     zdotc_
#endif
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/10025d7a/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr  5 07:17:07 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 05:17:07 -0700
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CA+jOtDK2LXE2uJKczwt185t-qPZzH3d5XKCCgujS=O1z_sogjw@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
	<CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
	<CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>
	<CA+jOtDKNUXsG9ZkhAEbrM21AZFoTcE8bq97s0sRTyNOjrn1DtA@mail.gmail.com>
	<CA+jOtDK2LXE2uJKczwt185t-qPZzH3d5XKCCgujS=O1z_sogjw@mail.gmail.com>
Message-ID: <CAM9tzSmWiLJ6MJurmt24f0XP02GR0UaGrdHhhArbA0B58ZN8uA@mail.gmail.com>

On Thu, Apr 5, 2012 at 05:02, behzad baghapour
<behzad.baghapour at gmail.com>wrote:

> I could solve the problem by shifting the postCheck below the
> SNESSetFromOptions but still have another problem with passing defined
> context:
>
> All my calcualtions are saved into global FieldContext including
> flow-field variables. I passed it to PostCheck. I could apply my desired
> change on the context BUT the results came back to the original values
> (before postCheck function) and therefore my efferts to modify the Newton
> procedure did not achieved.
>
> How should I handle it?
>

You will have to explain more completely what is happening. If you want the
postcheck to modify a context, you have to make sure that it sees the same
context. PETSc NEVER does anything with the context pointers, it just
passes it through to the callbacks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/5e41a685/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 07:16:40 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 12:16:40 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>,
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>

Hi Jed,

Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:

This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.

Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.

Cheers

Gao
________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
Sent: Thursday, April 05, 2012 2:09 PM
To: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

On Thu, Apr 5, 2012 at 04:43, Gao Bin <bin.gao at uit.no<mailto:bin.gao at uit.no>> wrote:
I would like to use PETSc perform matrix-matrix multiplication C=A*B^T. I notice that MatMatMultTranspose could calculate C=A^T*B, but which is not I want.

Does anyone know how to perform C=A*B^T using PETSc (I know I could first calculate B^T and call MatMatMult with A and B^T, but it might not be efficient, is it?)? Thank you in advance.

Use petsc-dev. That routine is renamed (http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatTransposeMatMult.html) and the one you want is

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/e2c9a14d/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr  5 07:22:43 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 05 Apr 2012 14:22:43 +0200
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>
References: <4F18290F.3080301@gfz-potsdam.de>
	<4F184AF9.2000608@gfz-potsdam.de>	<4F7D87F2.9050408@gfz-potsdam.de>
	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>
Message-ID: <4F7D8E93.9000506@gfz-potsdam.de>

On 05.04.2012 14:13, Jed Brown wrote:
> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver 
> <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>
>     The inconsistency is still there. I pulled petsc-dev today and
>     came across with the same problem:
>
>     src/vec/vec/impls/seq/bvec2.c
>     239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
>     240:    *z = PetscSqrtReal(*z);
>
>     This doesn't work with Intel MKL. However in one of the mailing
>     list discussions it was said there is now check for that.
>     This check didn't work for me apparently. Any ideas why?
>
>
> As always, we need configure.log.
>
Please find it attached.

> In include/petscblaslapack_uscore.h (and other places), you will find 
> the following. So we need to know why 
> PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.

This I can see...

>
> #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
> EXTERN_C_BEGIN
> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const 
> PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const 
> PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const 
> PetscBLASInt *sy) {
>   PetscScalar tmpz;
>   zdotc_(&tmpz,n,x,sx,y,sy);
>   return tmpz;
> }
> EXTERN_C_END
> #else
> #  define BLASdot_     zdotc_
> #endif


-- 
Regards,
Alexander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/5cf3f9d0/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.zip
Type: application/x-zip-compressed
Size: 380896 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/5cf3f9d0/attachment-0001.bin>

From jedbrown at mcs.anl.gov  Thu Apr  5 07:32:34 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 05:32:34 -0700
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
Message-ID: <CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>

On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:

> Thank you for your quick reply. But as pointed out at
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html
> :
>
> This routine is currently only implemented for pairs of SeqAIJ matrices. C
> will be of type MATSEQAIJ.
>
> Therefore I can not use it for dense matrix, am I right? If so, will
> MatMatTransposeMult be extended for other types of matrix later on? Thank
> you very much.
>

This is much simpler than the sparse case. Hong, did you intend to get
around to this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/2f9dc251/attachment.htm>

From knepley at gmail.com  Thu Apr  5 07:42:52 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 5 Apr 2012 07:42:52 -0500
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <4F7D8E93.9000506@gfz-potsdam.de>
References: <4F18290F.3080301@gfz-potsdam.de> <4F184AF9.2000608@gfz-potsdam.de>
	<4F7D87F2.9050408@gfz-potsdam.de>
	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>
	<4F7D8E93.9000506@gfz-potsdam.de>
Message-ID: <CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>

On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver
<agrayver at gfz-potsdam.de>wrote:

> **
> On 05.04.2012 14:13, Jed Brown wrote:
>
> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <agrayver at gfz-potsdam.de>wrote:
>
>> The inconsistency is still there. I pulled petsc-dev today and came
>> across with the same problem:
>>
>> src/vec/vec/impls/seq/bvec2.c
>> 239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
>> 240:    *z = PetscSqrtReal(*z);
>>
>> This doesn't work with Intel MKL. However in one of the mailing list
>> discussions it was said there is now check for that.
>> This check didn't work for me apparently. Any ideas why?
>>
>
> As always, we need configure.log.
>
>  Please find it attached.
>

Please ALWAYS ALWAYS ALWAYS send configure.log to petsc-maint at mcs.anl.gov.
They are too
big for the public list.

Second, your problem is that we get a link error tying to test this so it
looks like
your BLAS have no complex implementations:

sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
-Qoption,cpp,--extended_float_type -g
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
-L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil
-L/opt/mpi/intel/openmpi-1.4.2/lib
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L//opt/intel/Compiler/11.1/072/lib/intel64
-L/usr/lib64/gcc/x86_64-suse-linux/4.3
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc
-lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L//opt/intel/Compiler/11.1/072/lib/intel64
-L/usr/lib64/gcc/x86_64-suse-linux/4.3
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L/usr/x86_64-suse-linux/lib -ldl
Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
-Qoption,cpp,--extended_float_type -g
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
-L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil
-L/opt/mpi/intel/openmpi-1.4.2/lib
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L//opt/intel/Compiler/11.1/072/lib/intel64
-L/usr/lib64/gcc/x86_64-suse-linux/4.3
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc
-lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L//opt/intel/Compiler/11.1/072/lib/intel64
-L/usr/lib64/gcc/x86_64-suse-linux/4.3
-L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
-L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
-L/usr/x86_64-suse-linux/lib -ldl
sh:
Possible ERROR while running linker:
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main':
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined
reference to `zdotc_'
 output: ret = 256
error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o:
In function `main':
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined
reference to `zdotc_'
}


    Matt


> In include/petscblaslapack_uscore.h (and other places), you will find the
> following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not
> found by configure.
>
> This I can see...
>
>
>
>  #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
> EXTERN_C_BEGIN
> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
> PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const
> PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const
> PetscBLASInt *sy) {
>   PetscScalar tmpz;
>   zdotc_(&tmpz,n,x,sx,y,sy);
>   return tmpz;
> }
> EXTERN_C_END
> #else
> #  define BLASdot_     zdotc_
> #endif
>
>
>
> --
> Regards,
> Alexander
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/1ebd68ae/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 07:42:08 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 12:42:08 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>

Hi Jed,

Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.

P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.

Cheers

Gao
________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
Sent: Thursday, April 05, 2012 2:32 PM
To: PETSc users list; Hong Zhang
Subject: Re: [petsc-users] question about MatMatMultTranspose

On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no<mailto:bin.gao at uit.no>> wrote:
Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:

This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.

Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.

This is much simpler than the sparse case. Hong, did you intend to get around to this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/e56a946b/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr  5 07:54:06 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 05 Apr 2012 14:54:06 +0200
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>
References: <4F18290F.3080301@gfz-potsdam.de>
	<4F184AF9.2000608@gfz-potsdam.de>	<4F7D87F2.9050408@gfz-potsdam.de>	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>	<4F7D8E93.9000506@gfz-potsdam.de>
	<CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>
Message-ID: <4F7D95EE.2000000@gfz-potsdam.de>

On 05.04.2012 14:42, Matthew Knepley wrote:
> On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver 
> <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>
>     On 05.04.2012 14:13, Jed Brown wrote:
>>     On Thu, Apr 5, 2012 at 04:54, Alexander Grayver
>>     <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>>
>>         The inconsistency is still there. I pulled petsc-dev today
>>         and came across with the same problem:
>>
>>         src/vec/vec/impls/seq/bvec2.c
>>         239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
>>         240:    *z = PetscSqrtReal(*z);
>>
>>         This doesn't work with Intel MKL. However in one of the
>>         mailing list discussions it was said there is now check for that.
>>         This check didn't work for me apparently. Any ideas why?
>>
>>
>>     As always, we need configure.log.
>>
>     Please find it attached.
>
>
> Please ALWAYS ALWAYS ALWAYS send configure.log to 
> petsc-maint at mcs.anl.gov <mailto:petsc-maint at mcs.anl.gov>. They are too
> big for the public list.

Sorry, next time I will.

>
> Second, your problem is that we get a link error tying to test this so 
> it looks like
> your BLAS have no complex implementations:

It does have zdotc. Old petsc-dev works with it fine (after I patched 
vec implementaton myself to get around this issue).

home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a 
| grep zdotc
cblas_zdotci_lp64.o:
0000000000000000 T cblas_zdotci_sub
cblas_zdotc_lp64.o:
0000000000000000 T cblas_zdotc_sub
                  U zdotc
_zdotci_lp64.o:
                  U mkl_blas_zdotci
0000000000000000 T zdotci
0000000000000000 T zdotci_
_zdotc_lp64.o:
                  U mkl_blas_zdotc
0000000000000000 T zdotc
0000000000000000 T zdotc_
home>

>
> sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o 
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572 
> -Qoption,cpp,--extended_float_type -g 
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl 
> -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl 
> -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L//opt/intel/Compiler/11.1/072/lib/intel64 
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s 
> -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L//opt/intel/Compiler/11.1/072/lib/intel64 
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L/usr/x86_64-suse-linux/lib -ldl
> Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o 
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572 
> -Qoption,cpp,--extended_float_type -g 
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl 
> -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl 
> -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L//opt/intel/Compiler/11.1/072/lib/intel64 
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s 
> -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L//opt/intel/Compiler/11.1/072/lib/intel64 
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3 
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib 
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf 
> -L/usr/x86_64-suse-linux/lib -ldl
> sh:
> Possible ERROR while running linker: 
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function 
> `main':
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined 
> reference to `zdotc_'
>  output: ret = 256
> error message = 
> {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function 
> `main':
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined 
> reference to `zdotc_'
> }
>
>
>     Matt
>
>     In include/petscblaslapack_uscore.h (and other places), you will
>     find the following. So we need to know why
>     PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
>
>     This I can see...
>
>
>>
>>     #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
>>     EXTERN_C_BEGIN
>>     extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
>>     PetscScalar*,const PetscBLASInt*,const PetscScalar*,const
>>     PetscBLASInt*);
>>     PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt
>>     *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar
>>     *y,const PetscBLASInt *sy) {
>>       PetscScalar tmpz;
>>       zdotc_(&tmpz,n,x,sx,y,sy);
>>       return tmpz;
>>     }
>>     EXTERN_C_END
>>     #else
>>     #  define BLASdot_     zdotc_
>>     #endif
>
>
>     -- 
>     Regards,
>     Alexander
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
Regards,
Alexander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/2a4d02de/attachment-0001.htm>

From bsmith at mcs.anl.gov  Thu Apr  5 08:02:25 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 08:02:25 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
Message-ID: <BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>


   Gao,

    PETSc is mostly designed and implemented for large sparse matrix problems. We are not really experts for large dense matrix problems. Note that PETSc Seq dense matrices are just stored using the usual column oriented single array for the matrix (like Blas 2 and 3) so you can always use MatGetArray() and make some dense computations yourself directly. 

     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.

    Barry

On Apr 5, 2012, at 7:42 AM, Gao Bin wrote:

> Hi Jed,
> 
> Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.
> 
> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.
> 
> Cheers
> 
> Gao
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
> Sent: Thursday, April 05, 2012 2:32 PM
> To: PETSc users list; Hong Zhang
> Subject: Re: [petsc-users] question about MatMatMultTranspose
> 
> On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
> Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:
> 
> This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.
> 
> Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.
> 
> This is much simpler than the sparse case. Hong, did you intend to get around to this?


From bsmith at mcs.anl.gov  Thu Apr  5 08:05:42 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 08:05:42 -0500
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <4F7D95EE.2000000@gfz-potsdam.de>
References: <4F18290F.3080301@gfz-potsdam.de>
	<4F184AF9.2000608@gfz-potsdam.de>	<4F7D87F2.9050408@gfz-potsdam.de>	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>	<4F7D8E93.9000506@gfz-potsdam.de>
	<CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>
	<4F7D95EE.2000000@gfz-potsdam.de>
Message-ID: <B0E545A0-C077-4959-835A-96273E4DB53F@mcs.anl.gov>


   Matt,

    It doesn't look like that link line is even linking against the MKL libraries?

   Barry

On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:

> On 05.04.2012 14:42, Matthew Knepley wrote:
>> On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <agrayver at gfz-potsdam.de> wrote:
>> On 05.04.2012 14:13, Jed Brown wrote:
>>> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <agrayver at gfz-potsdam.de> wrote:
>>> The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
>>> 
>>> src/vec/vec/impls/seq/bvec2.c
>>> 239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
>>> 240:    *z = PetscSqrtReal(*z);
>>> 
>>> This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that.
>>> This check didn't work for me apparently. Any ideas why?
>>> 
>>> As always, we need configure.log.
>>> 
>> Please find it attached.
>> 
>> Please ALWAYS ALWAYS ALWAYS send configure.log to petsc-maint at mcs.anl.gov. They are too
>> big for the public list.
> 
> Sorry, next time I will.
> 
>> 
>> Second, your problem is that we get a link error tying to test this so it looks like
>> your BLAS have no complex implementations:
> 
> It does have zdotc. Old petsc-dev works with it fine (after I patched vec implementaton myself to get around this issue).
> 
> home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a | grep zdotc
> cblas_zdotci_lp64.o:
> 0000000000000000 T cblas_zdotci_sub
> cblas_zdotc_lp64.o:
> 0000000000000000 T cblas_zdotc_sub
>                  U zdotc
> _zdotci_lp64.o:
>                  U mkl_blas_zdotci
> 0000000000000000 T zdotci
> 0000000000000000 T zdotci_
> _zdotc_lp64.o:
>                  U mkl_blas_zdotc
> 0000000000000000 T zdotc
> 0000000000000000 T zdotc_
> home>
> 
>> 
>> sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl 
>> Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl 
>> sh: 
>> Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main':
>> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_'
>>  output: ret = 256
>> error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main':
>> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_'
>> }
>> 
>> 
>>     Matt
>>  
>> In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
>> 
>> This I can see...
>> 
>> 
>>> 
>>> #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
>>> EXTERN_C_BEGIN
>>> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
>>> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) {
>>>   PetscScalar tmpz;
>>>   zdotc_(&tmpz,n,x,sx,y,sy);
>>>   return tmpz;
>>> }
>>> EXTERN_C_END
>>> #else
>>> #  define BLASdot_     zdotc_
>>> #endif
>> 
>> 
>>  
>> -- 
>> Regards,
>> Alexander
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> -- 
> Regards,
> Alexander
> 


From bsmith at mcs.anl.gov  Thu Apr  5 08:10:03 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 08:10:03 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120404225543.APZ06409@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
Message-ID: <D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>


   It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.


   Barry 

   If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.


On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:

> Thank you Barry.
> 
> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
> 
> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
> 
> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
> 
>  /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>  /* so use those from global PC */
>  /* Is this what we always want? What if user wants to keep old one? */
>  ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>  ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>  ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>  if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>    ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>    ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>  }
> 
> Thank you.
> 
> Yuqi
> 
> 
> 
> ---- Original message ----
>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>> 
>>  When it gets to the lines 
>> 
>> if (pc->ops->setup) {
>>   ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>> }
>> 
>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>> 
>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>> 
>>  It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>> 
>> Barry
>> 
>> 
>> 
>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>> 
>>> ---- Original message ----
>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>> 
>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>> 
>>>> Setting PC with identical preconditioner\
>>>> Setting up new PC
>>>> Setting up PC with same nonzero pattern\
>>>> Setting up PC with different nonzero pattern\n
>>>> 
>>>> How many, and exactly what messages of this form are you getting?
>>>> 
>>>>  When all else fails you can run the program in the debugger to track what is happening and why. 
>>>> 
>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>> 
>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>> 
>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>> 
>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>> 
>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>>> 
>>>> 
>>>>  Barry
>>>> 
>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>> 
>>>>> Thanks, Adam.
>>>>> 
>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>> 
>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>> 
>>>>> 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>> 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>> 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>> 
>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>> 
>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>> 
>>>>>> Mark
>>>>>> 
>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Thank you.
>>>>>>> 
>>>>>>> Can I ask another question? 
>>>>>>> 
>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>> 
>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>> 
>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>> 
>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Hi, Barry,
>>>>>>>>> 
>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>> 
>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>> 
>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>> 
>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>> 
>>>>>>>>> Best
>>>>>>>>> 
>>>>>>>>> Yuqi
>>>>>>>>> 
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>> 
>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>> 
>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>> 
>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> 


From behzad.baghapour at gmail.com  Thu Apr  5 08:34:11 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 5 Apr 2012 18:04:11 +0430
Subject: [petsc-users] Calling function from outside of SNES
In-Reply-To: <CAM9tzSmWiLJ6MJurmt24f0XP02GR0UaGrdHhhArbA0B58ZN8uA@mail.gmail.com>
References: <CA+jOtDJFiR5NvLq2Deau+DmPOUNz_DJ23xsiu6-GnzpGL5WaDg@mail.gmail.com>
	<CAM9tzSmu3crNSq_6_FxPHJa6dY2wSP=sue5VyM5J=cwajM6x1w@mail.gmail.com>
	<CA+jOtDLNZqXrWT-3dderRScOMEJNsBEuRLPV=_S-tGJUVLHRTQ@mail.gmail.com>
	<CAM9tzSnn_7mjcTjo7umMKZJwyJGeRHmaqavNT-2VOBvq8biazw@mail.gmail.com>
	<CA+jOtDKLbSDuC688S5QRXErvtL9avY1gfezjQgibCcnastFDtw@mail.gmail.com>
	<CAM9tzSkODB3C0544+RtDzsht7KMYXKfZ40sh7Y=S=gwm0gMYcw@mail.gmail.com>
	<CA+jOtDKNUXsG9ZkhAEbrM21AZFoTcE8bq97s0sRTyNOjrn1DtA@mail.gmail.com>
	<CA+jOtDK2LXE2uJKczwt185t-qPZzH3d5XKCCgujS=O1z_sogjw@mail.gmail.com>
	<CAM9tzSmWiLJ6MJurmt24f0XP02GR0UaGrdHhhArbA0B58ZN8uA@mail.gmail.com>
Message-ID: <CA+jOtDLFg9ytbB486z7JmvXmANcrdNAAhQ9Ov5iiW7XxUrdYGQ@mail.gmail.com>

Thanks about your comment. I've got it. I think it is better to apply
changes to the solution vector (x) obtained by SNES, then implement all
what is gained to the my defined context at the end of the procedure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/3663cc16/attachment.htm>

From knepley at gmail.com  Thu Apr  5 08:53:44 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 5 Apr 2012 08:53:44 -0500
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <B0E545A0-C077-4959-835A-96273E4DB53F@mcs.anl.gov>
References: <4F18290F.3080301@gfz-potsdam.de> <4F184AF9.2000608@gfz-potsdam.de>
	<4F7D87F2.9050408@gfz-potsdam.de>
	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>
	<4F7D8E93.9000506@gfz-potsdam.de>
	<CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>
	<4F7D95EE.2000000@gfz-potsdam.de>
	<B0E545A0-C077-4959-835A-96273E4DB53F@mcs.anl.gov>
Message-ID: <CAMYG4Gk9qiUqgZOyk1cgyanG2rzWSe6W-Z4L3Tj2ZKr2Wdq2FQ@mail.gmail.com>

On Thu, Apr 5, 2012 at 8:05 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Matt,
>
>    It doesn't look like that link line is even linking against the MKL
> libraries?


Dang, this is a bug with specifying --with-blas-lapack-lib, since it all
gets dumped in
self.lapackLib, then self.blasLib is empty. Pushed a fix to BuildSystem.
Testing now.

   Matt


>
>   Barry
>
> On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:
>
> > On 05.04.2012 14:42, Matthew Knepley wrote:
> >> On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <
> agrayver at gfz-potsdam.de> wrote:
> >> On 05.04.2012 14:13, Jed Brown wrote:
> >>> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <
> agrayver at gfz-potsdam.de> wrote:
> >>> The inconsistency is still there. I pulled petsc-dev today and came
> across with the same problem:
> >>>
> >>> src/vec/vec/impls/seq/bvec2.c
> >>> 239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
> >>> 240:    *z = PetscSqrtReal(*z);
> >>>
> >>> This doesn't work with Intel MKL. However in one of the mailing list
> discussions it was said there is now check for that.
> >>> This check didn't work for me apparently. Any ideas why?
> >>>
> >>> As always, we need configure.log.
> >>>
> >> Please find it attached.
> >>
> >> Please ALWAYS ALWAYS ALWAYS send configure.log to
> petsc-maint at mcs.anl.gov. They are too
> >> big for the public list.
> >
> > Sorry, next time I will.
> >
> >>
> >> Second, your problem is that we get a link error tying to test this so
> it looks like
> >> your BLAS have no complex implementations:
> >
> > It does have zdotc. Old petsc-dev works with it fine (after I patched
> vec implementaton myself to get around this issue).
> >
> > home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a
> | grep zdotc
> > cblas_zdotci_lp64.o:
> > 0000000000000000 T cblas_zdotci_sub
> > cblas_zdotc_lp64.o:
> > 0000000000000000 T cblas_zdotc_sub
> >                  U zdotc
> > _zdotci_lp64.o:
> >                  U mkl_blas_zdotci
> > 0000000000000000 T zdotci
> > 0000000000000000 T zdotci_
> > _zdotc_lp64.o:
> >                  U mkl_blas_zdotc
> > 0000000000000000 T zdotc
> > 0000000000000000 T zdotc_
> > home>
> >
> >>
> >> sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
> -Qoption,cpp,--extended_float_type -g
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
> -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil
> -L/opt/mpi/intel/openmpi-1.4.2/lib
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L//opt/intel/Compiler/11.1/072/lib/intel64
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc
> -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L//opt/intel/Compiler/11.1/072/lib/intel64
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L/usr/x86_64-suse-linux/lib -ldl
> >> Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
> -Qoption,cpp,--extended_float_type -g
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
> -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil
> -L/opt/mpi/intel/openmpi-1.4.2/lib
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L//opt/intel/Compiler/11.1/072/lib/intel64
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc
> -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L//opt/intel/Compiler/11.1/072/lib/intel64
> -L/usr/lib64/gcc/x86_64-suse-linux/4.3
> -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
> -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
> -L/usr/x86_64-suse-linux/lib -ldl
> >> sh:
> >> Possible ERROR while running linker:
> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main':
> >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined
> reference to `zdotc_'
> >>  output: ret = 256
> >> error message =
> {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function
> `main':
> >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined
> reference to `zdotc_'
> >> }
> >>
> >>
> >>     Matt
> >>
> >> In include/petscblaslapack_uscore.h (and other places), you will find
> the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was
> not found by configure.
> >>
> >> This I can see...
> >>
> >>
> >>>
> >>> #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
> >>> EXTERN_C_BEGIN
> >>> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
> PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
> >>> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const
> PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const
> PetscBLASInt *sy) {
> >>>   PetscScalar tmpz;
> >>>   zdotc_(&tmpz,n,x,sx,y,sy);
> >>>   return tmpz;
> >>> }
> >>> EXTERN_C_END
> >>> #else
> >>> #  define BLASdot_     zdotc_
> >>> #endif
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Alexander
> >>
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> >> -- Norbert Wiener
> >
> >
> > --
> > Regards,
> > Alexander
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/1e37afbe/attachment-0001.htm>

From agrayver at gfz-potsdam.de  Thu Apr  5 09:06:55 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 05 Apr 2012 16:06:55 +0200
Subject: [petsc-users] MKL BLAS interface inconsistency
In-Reply-To: <CAMYG4Gk9qiUqgZOyk1cgyanG2rzWSe6W-Z4L3Tj2ZKr2Wdq2FQ@mail.gmail.com>
References: <4F18290F.3080301@gfz-potsdam.de>
	<4F184AF9.2000608@gfz-potsdam.de>	<4F7D87F2.9050408@gfz-potsdam.de>	<CAM9tzSn1en2Dv0RdHrL6d9kSx4TbX2fYC+U6MMbDW97T3w=YrA@mail.gmail.com>	<4F7D8E93.9000506@gfz-potsdam.de>	<CAMYG4GkxDoTn34Vg+qkRUkjjY891Luw=7=6acGZ8hCu_e6Vz-A@mail.gmail.com>	<4F7D95EE.2000000@gfz-potsdam.de>	<B0E545A0-C077-4959-835A-96273E4DB53F@mcs.anl.gov>
	<CAMYG4Gk9qiUqgZOyk1cgyanG2rzWSe6W-Z4L3Tj2ZKr2Wdq2FQ@mail.gmail.com>
Message-ID: <4F7DA6FF.4010108@gfz-potsdam.de>

Now everything works! Thanks for such a quick fix.

On 05.04.2012 15:53, Matthew Knepley wrote:
> On Thu, Apr 5, 2012 at 8:05 AM, Barry Smith <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>       Matt,
>
>        It doesn't look like that link line is even linking against the
>     MKL libraries?
>
>
> Dang, this is a bug with specifying --with-blas-lapack-lib, since it 
> all gets dumped in
> self.lapackLib, then self.blasLib is empty. Pushed a fix to 
> BuildSystem. Testing now.
>
>    Matt
>
>
>       Barry
>
>     On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:
>
>     > On 05.04.2012 14:42, Matthew Knepley wrote:
>     >> On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver
>     <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>     >> On 05.04.2012 14:13, Jed Brown wrote:
>     >>> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver
>     <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>     >>> The inconsistency is still there. I pulled petsc-dev today and
>     came across with the same problem:
>     >>>
>     >>> src/vec/vec/impls/seq/bvec2.c
>     >>> 239:    *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one));
>     >>> 240:    *z = PetscSqrtReal(*z);
>     >>>
>     >>> This doesn't work with Intel MKL. However in one of the
>     mailing list discussions it was said there is now check for that.
>     >>> This check didn't work for me apparently. Any ideas why?
>     >>>
>     >>> As always, we need configure.log.
>     >>>
>     >> Please find it attached.
>     >>
>     >> Please ALWAYS ALWAYS ALWAYS send configure.log to
>     petsc-maint at mcs.anl.gov <mailto:petsc-maint at mcs.anl.gov>. They are too
>     >> big for the public list.
>     >
>     > Sorry, next time I will.
>     >
>     >>
>     >> Second, your problem is that we get a link error tying to test
>     this so it looks like
>     >> your BLAS have no complex implementations:
>     >
>     > It does have zdotc. Old petsc-dev works with it fine (after I
>     patched vec implementaton myself to get around this issue).
>     >
>     > home> nm
>     /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a |
>     grep zdotc
>     > cblas_zdotci_lp64.o:
>     > 0000000000000000 T cblas_zdotci_sub
>     > cblas_zdotc_lp64.o:
>     > 0000000000000000 T cblas_zdotc_sub
>     >                  U zdotc
>     > _zdotci_lp64.o:
>     >                  U mkl_blas_zdotci
>     > 0000000000000000 T zdotci
>     > 0000000000000000 T zdotci_
>     > _zdotc_lp64.o:
>     >                  U mkl_blas_zdotc
>     > 0000000000000000 T zdotc
>     > 0000000000000000 T zdotc_
>     > home>
>     >
>     >>
>     >> sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
>     /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
>     -Qoption,cpp,--extended_float_type -g
>     /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
>     -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal
>     -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L//opt/intel/Compiler/11.1/072/lib/intel64
>     -L/usr/lib64/gcc/x86_64-suse-linux/4.3
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s
>     -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L//opt/intel/Compiler/11.1/072/lib/intel64
>     -L/usr/lib64/gcc/x86_64-suse-linux/4.3
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L/usr/x86_64-suse-linux/lib -ldl
>     >> Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc  -o
>     /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest    -wd1572
>     -Qoption,cpp,--extended_float_type -g
>     /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o  -ldl
>     -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal
>     -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L//opt/intel/Compiler/11.1/072/lib/intel64
>     -L/usr/lib64/gcc/x86_64-suse-linux/4.3
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s
>     -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L//opt/intel/Compiler/11.1/072/lib/intel64
>     -L/usr/lib64/gcc/x86_64-suse-linux/4.3
>     -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib
>     -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf
>     -L/usr/x86_64-suse-linux/lib -ldl
>     >> sh:
>     >> Possible ERROR while running linker:
>     /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In
>     function `main':
>     >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14:
>     undefined reference to `zdotc_'
>     >>  output: ret = 256
>     >> error message =
>     {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In
>     function `main':
>     >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14:
>     undefined reference to `zdotc_'
>     >> }
>     >>
>     >>
>     >>     Matt
>     >>
>     >> In include/petscblaslapack_uscore.h (and other places), you
>     will find the following. So we need to know why
>     PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
>     >>
>     >> This I can see...
>     >>
>     >>
>     >>>
>     >>> #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT
>     >>> EXTERN_C_BEGIN
>     >>> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
>     PetscScalar*,const PetscBLASInt*,const PetscScalar*,const
>     PetscBLASInt*);
>     >>> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt
>     *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar
>     *y,const PetscBLASInt *sy) {
>     >>>   PetscScalar tmpz;
>     >>>   zdotc_(&tmpz,n,x,sx,y,sy);
>     >>>   return tmpz;
>     >>> }
>     >>> EXTERN_C_END
>     >>> #else
>     >>> #  define BLASdot_     zdotc_
>     >>> #endif
>     >>
>     >>
>     >>
>     >> --
>     >> Regards,
>     >> Alexander
>     >>
>     >>
>     >>
>     >>
>     >> --
>     >> What most experimenters take for granted before they begin
>     their experiments is infinitely more interesting than any results
>     to which their experiments lead.
>     >> -- Norbert Wiener
>     >
>     >
>     > --
>     > Regards,
>     > Alexander
>     >
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
Regards,
Alexander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/fc8edd1a/attachment.htm>

From hzhang at mcs.anl.gov  Thu Apr  5 09:30:15 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 5 Apr 2012 09:30:15 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
Message-ID: <CAGCphBs8+z8TP=kde6m3KkAXCj_KwzN2T9J5C7U6iVYVy8jTfA@mail.gmail.com>

Bin :
We do not have plan for supporting parallel (mpiaij format)
C = A*B^T
because sparse inner product is too expensive, and we have parallel C =
A^T*B.

For sequential C = A*B^T, currently we only support
C_seqaij = A_seqaij*B_seqaij^T

Do you want
C_seqdense = A_seqaij*B_seqaij^T
or
C_seqdense = A_seqaij*B_seqdense^T?

Hong

>
> Good to know it is simpler ;-) I am switching to the developed version and
> try it. Again, thank you very much.
>
> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I
> ask if they are too difficult to implement (for instance, C=A*B^T and
> C=A^T*B for MATMPIDENSE)? Thank you.
>
> Cheers
>
> Gao
>  ------------------------------
> *From:* petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov]
> on behalf of Jed Brown [jedbrown at mcs.anl.gov]
> *Sent:* Thursday, April 05, 2012 2:32 PM
> *To:* PETSc users list; Hong Zhang
>
> *Subject:* Re: [petsc-users] question about MatMatMultTranspose
>
>   On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
>
>> Thank you for your quick reply. But as pointed out at
>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html
>> :
>>
>> This routine is currently only implemented for pairs of SeqAIJ matrices.
>> C will be of type MATSEQAIJ.
>>
>> Therefore I can not use it for dense matrix, am I right? If so, will
>> MatMatTransposeMult be extended for other types of matrix later on? Thank
>> you very much.
>>
>
> This is much simpler than the sparse case. Hong, did you intend to get
> around to this?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/fe0af8aa/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr  5 09:50:53 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 07:50:53 -0700
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
Message-ID: <CAM9tzSngt1oq4viJscC1VhbW8fbvGCLykAQ5gmzURG4SXscGHQ@mail.gmail.com>

On Thu, Apr 5, 2012 at 06:02, Barry Smith <bsmith at mcs.anl.gov> wrote:

> Parallel dense we know very little about and cannot write those routines,
> sadly there are no decently supported parallel dense matrix general purpose
> libraries out there that we can use (and no Scalapack, plapack and
> elemental do not count as decent AND supported) so it is unlikely WE will
> write the MPIDENSE versions of these routines. Though if someone else
> writes them we would be happy to include them. So basically for parallel
> dense I have no suggestions.
>

Elemental is the one to use. Several of us have talked about writing the
interface, but nobody has had the time to follow through.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/d4f5fec7/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 10:09:35 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 15:09:35 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CAGCphBs8+z8TP=kde6m3KkAXCj_KwzN2T9J5C7U6iVYVy8jTfA@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>,
	<CAGCphBs8+z8TP=kde6m3KkAXCj_KwzN2T9J5C7U6iVYVy8jTfA@mail.gmail.com>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C209@MBS04.uit.no>

Hi Hong,

Thank you! From Barry's email, I have learned that PETSc aims at sparse matrix. Therefore, I may consider using MatGetArray() for further parallel calculations, or using Elemental/PLAPACK for dense matrix calculations. I notice PETSc could use PLAPACK, but if I understand correctly, that has only been implemented for LU, Cholesky decomposition, not matrix-matrix multiplications (although it looks to be simple).

As regards my question, I am sorry that I need

C_seqdense = A_seqdense*B_seqdense^T
C_seqdense = A_seqdense^T*B_seqdense

for the time being, not aij format.

Cheers

Gao
________________________________
From: Hong Zhang [hzhang at mcs.anl.gov]
Sent: Thursday, April 05, 2012 4:30 PM
To: Gao Bin
Cc: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

Bin :
We do not have plan for supporting parallel (mpiaij format)
C = A*B^T
because sparse inner product is too expensive, and we have parallel C = A^T*B.

For sequential C = A*B^T, currently we only support
C_seqaij = A_seqaij*B_seqaij^T

Do you want
C_seqdense = A_seqaij*B_seqaij^T
or
C_seqdense = A_seqaij*B_seqdense^T?

Hong

Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.

P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.

Cheers

Gao
________________________________
From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] on behalf of Jed Brown [jedbrown at mcs.anl.gov<mailto:jedbrown at mcs.anl.gov>]
Sent: Thursday, April 05, 2012 2:32 PM
To: PETSc users list; Hong Zhang

Subject: Re: [petsc-users] question about MatMatMultTranspose

On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no<mailto:bin.gao at uit.no>> wrote:
Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:

This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.

Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.

This is much simpler than the sparse case. Hong, did you intend to get around to this?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/d8312793/attachment-0001.htm>

From bin.gao at uit.no  Thu Apr  5 10:11:03 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 15:11:03 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>,
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C216@MBS04.uit.no>

Hi Barray,

Thank you for your detailed reply. I will have a look at Elemental and/or PLAPACK for parallel dense matrix operations. Thank you.

Cheers

Gao
________________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
Sent: Thursday, April 05, 2012 3:02 PM
To: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

   Gao,

    PETSc is mostly designed and implemented for large sparse matrix problems. We are not really experts for large dense matrix problems. Note that PETSc Seq dense matrices are just stored using the usual column oriented single array for the matrix (like Blas 2 and 3) so you can always use MatGetArray() and make some dense computations yourself directly.

     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.

    Barry

On Apr 5, 2012, at 7:42 AM, Gao Bin wrote:

> Hi Jed,
>
> Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.
>
> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.
>
> Cheers
>
> Gao
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
> Sent: Thursday, April 05, 2012 2:32 PM
> To: PETSc users list; Hong Zhang
> Subject: Re: [petsc-users] question about MatMatMultTranspose
>
> On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
> Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:
>
> This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.
>
> Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.
>
> This is much simpler than the sparse case. Hong, did you intend to get around to this?


From hanangul12 at yahoo.co.uk  Thu Apr  5 10:19:31 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 5 Apr 2012 16:19:31 +0100 (BST)
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
Message-ID: <1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>

Hi users and developers, 
Summer greetings from Delft. 


For the system Au = f, I want to apply a multilevel preconditioner with a KSP (say FGMRES) . The preconditioner
reads as

Prec = C +? M^-1 (I - A*C) , where


	* ?C reads coarse grid correction operator i.e. C =? P*A_2h\R? [ R? restriction, A_2h coarse operator, P interpolation] 
	* M is say some sparse matrix (resembling to A) 

What makes it multilevel is I have to approximate coarse operator "A_2h" with few KSP iterations preconditioned by the above defined
preconditioner "Prec", but off course at the level 2h and hence at every coarse level until coarsest. 


I only know, it can be implemented with PCMG. I am little afraid of DMMG. 

Explanatory suggestions to customize PCMG for what is requiredare very appreciated. 


Thanks in advance.?


regards, Abdul 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/bed5b5e9/attachment.htm>

From bsmith at mcs.anl.gov  Thu Apr  5 10:19:43 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 10:19:43 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CA+S8wr-fPV0sGzxwcVn9Vq-37k8+SRadCbStDwPb82EYcA3UdA@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
	<CA+S8wr9JUBb80B8Y+yYSh9r7afMy-TBz9GcQHRRpMa0dpgrfvA@mail.gmail.com>
	<CA+S8wr-fPV0sGzxwcVn9Vq-37k8+SRadCbStDwPb82EYcA3UdA@mail.gmail.com>
Message-ID: <C4124E55-8DD2-4D7C-B879-5A150D6D3A35@mcs.anl.gov>


On Apr 5, 2012, at 9:54 AM, Jack Poulson wrote:

> On Thu, Apr 5, 2012 at 9:20 AM, Jack Poulson <jack.poulson at gmail.com> wrote:
> On Thu, Apr 5, 2012 at 8:02 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.
>    
> 
> If you let me know what your major complaints are, I'll look into them.
> 
> Jack 
> 
> Forwarding to Jed and Matt; seriously, let me know what your complaints are. If I can, I will fix them.
> 
> Jack

   Jack,

     I apologize, I was too quick to dismiss elemental. We went to the effort to do the MPIDENSE interface for PLAPACK and that ended up being a complete waste and left a bad taste in my mouth since I consider that wasted time when we could have done something else.

    So we could try again with Elemental but its a question of having a competent person to do the interface efficiently. Perhaps you have a student or someone you know that knows elemental we could hire for a short time to do an interface and see how it goes?

    Barry



From hanangul12 at yahoo.co.uk  Thu Apr  5 10:22:36 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 5 Apr 2012 16:22:36 +0100 (BST)
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
Message-ID: <1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>

Hi users and developers, 
Summer greetings from Delft. 


For the system Au = f, I want to apply a multilevel preconditioner with a KSP (say FGMRES) . The preconditioner
reads as

Prec = C +? M^-1 (I - A*C) , where


	* ?C reads coarse grid correction operator i.e. C =? P*A_2h\R? [ R? restriction, A_2h coarse operator, P interpolation] 
	* M is say some sparse matrix (resembling to A) 

What makes it multilevel is I have to approximate coarse operator "A_2h" with few KSP iterations preconditioned by the above defined
preconditioner "Prec" , but off course at the level 2h and hence at every coarse level until coarsest. 


I only know, it can be implemented with PCMG. I am little afraid of DMMG. 

Explanatory suggestions to customize PCMG for what is required are very appreciated. 


Thanks in advance.?


regards, Abdul 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/cfe132bf/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 10:27:18 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 15:27:18 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>,
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C25B@MBS04.uit.no>

Hi, again

Sorry, I was wrong in my last email about the interface to PLAPACK in PETSc. It looks like I could do parallel dense matrix multiplication using PETSc by enabling PLAPACK interface.

Cheers

Gao
________________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
Sent: Thursday, April 05, 2012 3:02 PM
To: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

   Gao,

    PETSc is mostly designed and implemented for large sparse matrix problems. We are not really experts for large dense matrix problems. Note that PETSc Seq dense matrices are just stored using the usual column oriented single array for the matrix (like Blas 2 and 3) so you can always use MatGetArray() and make some dense computations yourself directly.

     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.

    Barry

On Apr 5, 2012, at 7:42 AM, Gao Bin wrote:

> Hi Jed,
>
> Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.
>
> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.
>
> Cheers
>
> Gao
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
> Sent: Thursday, April 05, 2012 2:32 PM
> To: PETSc users list; Hong Zhang
> Subject: Re: [petsc-users] question about MatMatMultTranspose
>
> On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
> Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:
>
> This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.
>
> Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.
>
> This is much simpler than the sparse case. Hong, did you intend to get around to this?


From bsmith at mcs.anl.gov  Thu Apr  5 10:29:47 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 10:29:47 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C25B@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>,
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
	<CCA7D73E69B27E4AB6075A96C35D44394774C25B@MBS04.uit.no>
Message-ID: <36A0EA64-15E2-4004-A7B8-953D9A4D2C5A@mcs.anl.gov>


On Apr 5, 2012, at 10:27 AM, Gao Bin wrote:

> Hi, again
> 
> Sorry, I was wrong in my last email about the interface to PLAPACK in PETSc. It looks like I could do parallel dense matrix multiplication using PETSc by enabling PLAPACK interface.

   There are problems with PLAPACK, even the matrix-matrix product has buggyness issues. I don't recommend you do this.


    Barry

> 
> Cheers
> 
> Gao
> ________________________________________
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
> Sent: Thursday, April 05, 2012 3:02 PM
> To: PETSc users list
> Subject: Re: [petsc-users] question about MatMatMultTranspose
> 
>   Gao,
> 
>    PETSc is mostly designed and implemented for large sparse matrix problems. We are not really experts for large dense matrix problems. Note that PETSc Seq dense matrices are just stored using the usual column oriented single array for the matrix (like Blas 2 and 3) so you can always use MatGetArray() and make some dense computations yourself directly.
> 
>     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.
> 
>    Barry
> 
> On Apr 5, 2012, at 7:42 AM, Gao Bin wrote:
> 
>> Hi Jed,
>> 
>> Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.
>> 
>> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.
>> 
>> Cheers
>> 
>> Gao
>> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
>> Sent: Thursday, April 05, 2012 2:32 PM
>> To: PETSc users list; Hong Zhang
>> Subject: Re: [petsc-users] question about MatMatMultTranspose
>> 
>> On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
>> Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:
>> 
>> This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.
>> 
>> Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.
>> 
>> This is much simpler than the sparse case. Hong, did you intend to get around to this?
> 


From jedbrown at mcs.anl.gov  Thu Apr  5 10:30:00 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 08:30:00 -0700
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
Message-ID: <CAM9tzS=Gz4JKwAQc+HKe38W_gNs9m-Eed6ZteS54jJh+bVTyFw@mail.gmail.com>

On Thu, Apr 5, 2012 at 08:22, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>wrote:

> For the system Au = f, I want to apply a multilevel preconditioner with a
> KSP (say FGMRES) . The preconditioner
> reads as
>
> Prec = C +  M^-1 (I - A*C) , where
>
>
>    -  C reads coarse grid correction operator i.e. C =  P*A_2h\R  [ R
>    restriction, A_2h coarse operator, P interpolation]
>    - M is say some sparse matrix (resembling to A)
>
> What makes it multilevel is I have to approximate coarse operator "A_2h"
> with few KSP iterations preconditioned by the above defined
> preconditioner "Prec" , but off course at the level 2h and hence at every
> coarse level until coarsest.
>

Hmm, if you use a constant number of iterations in C at each level, the
number of coarsest grid iterations will grow exponentially in the number of
levels. Is that really what you want?


> I only know, it can be implemented with PCMG. I am little afraid of DMMG.
>

Don't use DMMG, it is being removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/6be581ae/attachment-0001.htm>

From jack.poulson at gmail.com  Thu Apr  5 10:37:34 2012
From: jack.poulson at gmail.com (Jack Poulson)
Date: Thu, 5 Apr 2012 10:37:34 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <C4124E55-8DD2-4D7C-B879-5A150D6D3A35@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
	<CA+S8wr9JUBb80B8Y+yYSh9r7afMy-TBz9GcQHRRpMa0dpgrfvA@mail.gmail.com>
	<CA+S8wr-fPV0sGzxwcVn9Vq-37k8+SRadCbStDwPb82EYcA3UdA@mail.gmail.com>
	<C4124E55-8DD2-4D7C-B879-5A150D6D3A35@mcs.anl.gov>
Message-ID: <CA+S8wr_7J0Pj2QDYbnj4H7aNfhEHPS5OBtE9WaFXyKyU+KGwyw@mail.gmail.com>

On Thu, Apr 5, 2012 at 10:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Apr 5, 2012, at 9:54 AM, Jack Poulson wrote:
>
> > On Thu, Apr 5, 2012 at 9:20 AM, Jack Poulson <jack.poulson at gmail.com>
> wrote:
> > On Thu, Apr 5, 2012 at 8:02 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Parallel dense we know very little about and cannot write those
> routines, sadly there are no decently supported parallel dense matrix
> general purpose libraries out there that we can use (and no Scalapack,
> plapack and elemental do not count as decent AND supported) so it is
> unlikely WE will write the MPIDENSE versions of these routines. Though if
> someone else writes them we would be happy to include them. So basically
> for parallel dense I have no suggestions.
> >
> >
> > If you let me know what your major complaints are, I'll look into them.
> >
> > Jack
> >
> > Forwarding to Jed and Matt; seriously, let me know what your complaints
> are. If I can, I will fix them.
> >
> > Jack
>
>    Jack,
>
>     I apologize, I was too quick to dismiss elemental. We went to the
> effort to do the MPIDENSE interface for PLAPACK and that ended up being a
> complete waste and left a bad taste in my mouth since I consider that
> wasted time when we could have done something else.
>
>
No offense taken; it is understandable that PLAPACK did not meet PETSc's
rigorous development standards, as it was closer to a research code. With
that said, the reason I wrote Elemental is that I wanted a robust linear
algebra package that made use of PLAPACK's insights (ala exposing matrix
distributions as objects).

Given that you had a bad experience with PLAPACK, perhaps the best
assurance I can give you is that I am actively dogfooding Elemental in my
other projects (see http://bitbucket.org/poulson/clique and
http://bitbucket.org/poulson/psp). In addition, van de Geijn has a student
working on extending Elemental's matrix-matrix multiplication code to more
sophisticated distribution schemes. I _highly_ doubt that you will find a
bug in Elemental's matrix-matrix multiplication, but, if you do, I can
assure you that I will fix it within a few hours. The code is quite simple:
http://code.google.com/p/elemental/source/browse/#hg%2Finclude%2Felemental%2Fbasic%2Flevel3%2FGemm


>    So we could try again with Elemental but its a question of having a
> competent person to do the interface efficiently. Perhaps you have a
> student or someone you know that knows elemental we could hire for a short
> time to do an interface and see how it goes?
>
>
Unfortunately I am still just a lowly Ph.D. student (but hopefully that
will change in December). I would be happy to try to help, and _may_ find
the time to write the interface myself, but, until graduation, I can't
promise more than maintenance. I can talk with a few folks that might be
interested, but I would think that this should only take a few days of
work, not anything that warranted an appointment.

Jack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/7711de18/attachment.htm>

From bsmith at mcs.anl.gov  Thu Apr  5 10:38:37 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 10:38:37 -0500
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
Message-ID: <B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>


On Apr 5, 2012, at 10:22 AM, Abdul Hanan Sheikh wrote:

> Hi users and developers, 
> Summer greetings from Delft. 
> 
> 
> For the system Au = f, I want to apply a multilevel preconditioner with a KSP (say FGMRES) . The preconditioner
> reads as
> 
> Prec = C +  M^-1 (I - A*C) , where
> 
> 	?  C reads coarse grid correction operator i.e. C =  P*A_2h\R  [ R  restriction, A_2h coarse operator, P interpolation]
> 	? M is say some sparse matrix (resembling to A) 
> What makes it multilevel is I have to approximate coarse operator "A_2h" with few KSP iterations preconditioned by the above defined
> preconditioner "Prec" , but off course at the level 2h and hence at every coarse level until coarsest. 
> 
> I only know, it can be implemented with PCMG. I am little afraid of DMMG. 
> Explanatory suggestions to customize PCMG for what is required are very appreciated. 


    C +  M^-1 (I - A*C)   is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way. 

    Do NOT use DMMG it is deprecated and being removed from PETSc.

    To use PCMG you must provide the P and R with PCMGSetInterpolation and PCMGSetRestriction. You would us
 PCMGGetCoarseSolve(pc,&user.ksp_coarse);CHKERRQ(ierr);
   ierr = KSPSetOptionsPrefix(user.ksp_coarse,"coarse_");CHKERRQ(ierr);
  ierr = KSPSetFromOptions(user.ksp_coarse);CHKERRQ(ierr);
  ierr = KSPSetOperators(user.ksp_coarse,user.coarse.J,user.coarse.J,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

   to provide the coarse A_c and depending what your "smoother" M is you would possibly pass a different matrix into the second location of KSPSetOperators() for each level to represent the "M is say some sparse matrix resembling A."

    src/ksp/ksp/examples/tests/ex19.c is a simple example showing how all the values are set for two levels.

   Barry

> 
> Thanks in advance. 
> 
> 
> regards, Abdul 
> 
> 


From hzhang at mcs.anl.gov  Thu Apr  5 10:39:06 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 5 Apr 2012 10:39:06 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394774C209@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<CAGCphBs8+z8TP=kde6m3KkAXCj_KwzN2T9J5C7U6iVYVy8jTfA@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C209@MBS04.uit.no>
Message-ID: <CAGCphBtDZOfSF+3d4p=ETkvEzwRU5OJ7Z4c7Anjsn30rxHt9gQ@mail.gmail.com>

Bin :

>
> Thank you! From Barry's email, I have learned that PETSc aims at sparse
> matrix. Therefore, I may consider using MatGetArray() for further parallel
> calculations, or using Elemental/PLAPACK for dense matrix calculations. I
> notice PETSc could use PLAPACK, but if I understand correctly, that has
> only been implemented for LU, Cholesky decomposition, not matrix-matrix
> multiplications (although it looks to be simple).
>

We do have
MatMatMult_MPIDense_MPIDense() in ~petsc/src/mat/impls/dense/mpi/mpidense.c.
i.e., we tried to interface PLAPACK's dense matrix-matrix multiplication.
However, in MatMatMultSymbolic_MPIDense_MPIDense():
SETERRQ(((PetscObject)A)->comm,PETSC_ERR_LIB,"Due to apparent bugs in
PLAPACK,this is not currently supported");
Therefore, this does not work.

>
> As regards my question, I am sorry that I need
>
> C_seqdense = A_seqdense*B_seqdense^T
>  C_seqdense = A_seqdense^T*B_seqdense
>

These are purely dense computation,  Elemental/PLAPACK should be used.
Hong


>
> Gao
>   ------------------------------
> *From:* Hong Zhang [hzhang at mcs.anl.gov]
> *Sent:* Thursday, April 05, 2012 4:30 PM
> *To:* Gao Bin
> *Cc:* PETSc users list
>
> *Subject:* Re: [petsc-users] question about MatMatMultTranspose
>
>  Bin :
> We do not have plan for supporting parallel (mpiaij format)
> C = A*B^T
> because sparse inner product is too expensive, and we have parallel C =
> A^T*B.
>
>  For sequential C = A*B^T, currently we only support
> C_seqaij = A_seqaij*B_seqaij^T
>
>  Do you want
> C_seqdense = A_seqaij*B_seqaij^T
> or
> C_seqdense = A_seqaij*B_seqdense^T?
>
>  Hong
>
>>
>> Good to know it is simpler ;-) I am switching to the developed version
>> and try it. Again, thank you very much.
>>
>> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May
>> I ask if they are too difficult to implement (for instance, C=A*B^T and
>> C=A^T*B for MATMPIDENSE)? Thank you.
>>
>> Cheers
>>
>> Gao
>>  ------------------------------
>>  *From:* petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov]
>> on behalf of Jed Brown [jedbrown at mcs.anl.gov]
>>  *Sent:* Thursday, April 05, 2012 2:32 PM
>> *To:* PETSc users list; Hong Zhang
>>
>> *Subject:* Re: [petsc-users] question about MatMatMultTranspose
>>
>>    On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
>>
>>> Thank you for your quick reply. But as pointed out at
>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html
>>> :
>>>
>>> This routine is currently only implemented for pairs of SeqAIJ matrices.
>>> C will be of type MATSEQAIJ.
>>>
>>> Therefore I can not use it for dense matrix, am I right? If so, will
>>> MatMatTransposeMult be extended for other types of matrix later on? Thank
>>> you very much.
>>>
>>
>> This is much simpler than the sparse case. Hong, did you intend to get
>> around to this?
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/258f9438/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 10:40:29 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 15:40:29 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <36A0EA64-15E2-4004-A7B8-953D9A4D2C5A@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>,
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>,
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
	<CCA7D73E69B27E4AB6075A96C35D44394774C25B@MBS04.uit.no>,
	<36A0EA64-15E2-4004-A7B8-953D9A4D2C5A@mcs.anl.gov>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C278@MBS04.uit.no>

Hi Barry,

Thank you for pointing this issue. I think I will try Elemental for parallel dense matrix, otherwise I will stick on PETSc ;-)

Cheers

Gao
________________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
Sent: Thursday, April 05, 2012 5:29 PM
To: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

On Apr 5, 2012, at 10:27 AM, Gao Bin wrote:

> Hi, again
>
> Sorry, I was wrong in my last email about the interface to PLAPACK in PETSc. It looks like I could do parallel dense matrix multiplication using PETSc by enabling PLAPACK interface.

   There are problems with PLAPACK, even the matrix-matrix product has buggyness issues. I don't recommend you do this.


    Barry

>
> Cheers
>
> Gao
> ________________________________________
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
> Sent: Thursday, April 05, 2012 3:02 PM
> To: PETSc users list
> Subject: Re: [petsc-users] question about MatMatMultTranspose
>
>   Gao,
>
>    PETSc is mostly designed and implemented for large sparse matrix problems. We are not really experts for large dense matrix problems. Note that PETSc Seq dense matrices are just stored using the usual column oriented single array for the matrix (like Blas 2 and 3) so you can always use MatGetArray() and make some dense computations yourself directly.
>
>     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.
>
>    Barry
>
> On Apr 5, 2012, at 7:42 AM, Gao Bin wrote:
>
>> Hi Jed,
>>
>> Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.
>>
>> P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.
>>
>> Cheers
>>
>> Gao
>> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov]
>> Sent: Thursday, April 05, 2012 2:32 PM
>> To: PETSc users list; Hong Zhang
>> Subject: Re: [petsc-users] question about MatMatMultTranspose
>>
>> On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no> wrote:
>> Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:
>>
>> This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.
>>
>> Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.
>>
>> This is much simpler than the sparse case. Hong, did you intend to get around to this?
>


From bsmith at mcs.anl.gov  Thu Apr  5 10:41:24 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 10:41:24 -0500
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CA+S8wr_7J0Pj2QDYbnj4H7aNfhEHPS5OBtE9WaFXyKyU+KGwyw@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<BAE6BBD6-5DA0-4018-94AB-DB117BC1EB33@mcs.anl.gov>
	<CA+S8wr9JUBb80B8Y+yYSh9r7afMy-TBz9GcQHRRpMa0dpgrfvA@mail.gmail.com>
	<CA+S8wr-fPV0sGzxwcVn9Vq-37k8+SRadCbStDwPb82EYcA3UdA@mail.gmail.com>
	<C4124E55-8DD2-4D7C-B879-5A150D6D3A35@mcs.anl.gov>
	<CA+S8wr_7J0Pj2QDYbnj4H7aNfhEHPS5OBtE9WaFXyKyU+KGwyw@mail.gmail.com>
Message-ID: <18CF2711-70F3-4D2B-A6D7-0F1587B3EF0F@mcs.anl.gov>


  Jack,

    Thanks. Yes, in theory for some one who knows either elemental or PETSc well doing the interface should not be a long project. But if they don't know either it would be a long learning curve first.

    Barry

On Apr 5, 2012, at 10:37 AM, Jack Poulson wrote:

> On Thu, Apr 5, 2012 at 10:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Apr 5, 2012, at 9:54 AM, Jack Poulson wrote:
> 
> > On Thu, Apr 5, 2012 at 9:20 AM, Jack Poulson <jack.poulson at gmail.com> wrote:
> > On Thu, Apr 5, 2012 at 8:02 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Parallel dense we know very little about and cannot write those routines, sadly there are no decently supported parallel dense matrix general purpose libraries out there that we can use (and no Scalapack, plapack and elemental do not count as decent AND supported) so it is unlikely WE will write the MPIDENSE versions of these routines. Though if someone else writes them we would be happy to include them. So basically for parallel dense I have no suggestions.
> >
> >
> > If you let me know what your major complaints are, I'll look into them.
> >
> > Jack
> >
> > Forwarding to Jed and Matt; seriously, let me know what your complaints are. If I can, I will fix them.
> >
> > Jack
> 
>   Jack,
> 
>     I apologize, I was too quick to dismiss elemental. We went to the effort to do the MPIDENSE interface for PLAPACK and that ended up being a complete waste and left a bad taste in my mouth since I consider that wasted time when we could have done something else.
> 
> 
> No offense taken; it is understandable that PLAPACK did not meet PETSc's rigorous development standards, as it was closer to a research code. With that said, the reason I wrote Elemental is that I wanted a robust linear algebra package that made use of PLAPACK's insights (ala exposing matrix distributions as objects).
> 
> Given that you had a bad experience with PLAPACK, perhaps the best assurance I can give you is that I am actively dogfooding Elemental in my other projects (see http://bitbucket.org/poulson/clique and http://bitbucket.org/poulson/psp). In addition, van de Geijn has a student working on extending Elemental's matrix-matrix multiplication code to more sophisticated distribution schemes. I _highly_ doubt that you will find a bug in Elemental's matrix-matrix multiplication, but, if you do, I can assure you that I will fix it within a few hours. The code is quite simple:
> http://code.google.com/p/elemental/source/browse/#hg%2Finclude%2Felemental%2Fbasic%2Flevel3%2FGemm
>  
>    So we could try again with Elemental but its a question of having a competent person to do the interface efficiently. Perhaps you have a student or someone you know that knows elemental we could hire for a short time to do an interface and see how it goes?
> 
> 
> Unfortunately I am still just a lowly Ph.D. student (but hopefully that will change in December). I would be happy to try to help, and _may_ find the time to write the interface myself, but, until graduation, I can't promise more than maintenance. I can talk with a few folks that might be interested, but I would think that this should only take a few days of work, not anything that warranted an appointment.
>  
> Jack


From jedbrown at mcs.anl.gov  Thu Apr  5 10:56:37 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 08:56:37 -0700
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
Message-ID: <CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>

On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:

>    C +  M^-1 (I - A*C)   is exactly a multiplicative Schwarz method with
> two preconditioners. Or in another way to put it it is exactly a two level
> multigrid so you can use PCMG in a straightforward way.


It looks to me like he wants to do nested iteration with recursive coarse
levels rather than just iteration in a smoother.

*I need the accuracy at first and somehow at second coarse level. *
*After these two levels, the number of iterations at each level till
coarsest is constantly one.  *
*
*
Okay, but one iteration of a preconditioned Krylov method is still two
preconditioner applications, so the method would still be exponential in
the number of levels. If Richardson iteration is okay, you can use
-pc_mg_cycle_type w to do two iterations instead of 1. The code would
actually work fine with increasing this further, but it looks like you
would currently have to cheat a bit to get many recursive cycles.

PCMGSetCycleType(pc,(PCMGCycleType)5);

There is currently no interface for setting a different number of cycles on
each level, but you could include mgimpl.h and change the levels->cycles
manually.

Note that W cycles are already often considered to be too much work on
coarse levels to justify in a parallel setting, so it's somewhat unlikely
that running more coarse level iterations will really be better than a more
standard multigrid with stronger smoothers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/88af584a/attachment.htm>

From bin.gao at uit.no  Thu Apr  5 10:56:44 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 5 Apr 2012 15:56:44 +0000
Subject: [petsc-users] question about MatMatMultTranspose
In-Reply-To: <CAGCphBtDZOfSF+3d4p=ETkvEzwRU5OJ7Z4c7Anjsn30rxHt9gQ@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<CAM9tzS=V-3xVMaKg6LYXa6Si-fMkwFZTvD1ZxVVOxgDNPZCCrg@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C16D@MBS04.uit.no>
	<CAM9tzSnCZvTYVxJbNN+Z4Yu=3D3r5sgKPwh4rXA_qYczc1DQJQ@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C19B@MBS04.uit.no>
	<CAGCphBs8+z8TP=kde6m3KkAXCj_KwzN2T9J5C7U6iVYVy8jTfA@mail.gmail.com>
	<CCA7D73E69B27E4AB6075A96C35D44394774C209@MBS04.uit.no>,
	<CAGCphBtDZOfSF+3d4p=ETkvEzwRU5OJ7Z4c7Anjsn30rxHt9gQ@mail.gmail.com>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394774C29D@MBS04.uit.no>

Hi, Hong

Thank you. If there is no bug for using PLAPACK for matrix-matrix multiplications, I will try :-) Thanks again.

Cheers

Gao
________________________________
From: Hong Zhang [hzhang at mcs.anl.gov]
Sent: Thursday, April 05, 2012 5:39 PM
To: Gao Bin
Cc: PETSc users list
Subject: Re: [petsc-users] question about MatMatMultTranspose

Bin :

Thank you! From Barry's email, I have learned that PETSc aims at sparse matrix. Therefore, I may consider using MatGetArray() for further parallel calculations, or using Elemental/PLAPACK for dense matrix calculations. I notice PETSc could use PLAPACK, but if I understand correctly, that has only been implemented for LU, Cholesky decomposition, not matrix-matrix multiplications (although it looks to be simple).

We do have
MatMatMult_MPIDense_MPIDense() in ~petsc/src/mat/impls/dense/mpi/mpidense.c.
i.e., we tried to interface PLAPACK's dense matrix-matrix multiplication.
However, in MatMatMultSymbolic_MPIDense_MPIDense():
SETERRQ(((PetscObject)A)->comm,PETSC_ERR_LIB,"Due to apparent bugs in PLAPACK,this is not currently supported");
Therefore, this does not work.

As regards my question, I am sorry that I need

C_seqdense = A_seqdense*B_seqdense^T
C_seqdense = A_seqdense^T*B_seqdense

These are purely dense computation,  Elemental/PLAPACK should be used.
Hong



Gao
________________________________
From: Hong Zhang [hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>]
Sent: Thursday, April 05, 2012 4:30 PM
To: Gao Bin
Cc: PETSc users list

Subject: Re: [petsc-users] question about MatMatMultTranspose

Bin :
We do not have plan for supporting parallel (mpiaij format)
C = A*B^T
because sparse inner product is too expensive, and we have parallel C = A^T*B.

For sequential C = A*B^T, currently we only support
C_seqaij = A_seqaij*B_seqaij^T

Do you want
C_seqdense = A_seqaij*B_seqaij^T
or
C_seqdense = A_seqaij*B_seqdense^T?

Hong

Good to know it is simpler ;-) I am switching to the developed version and try it. Again, thank you very much.

P.S., Moreover, I notice that some functions is not for MATMPIDENSE. May I ask if they are too difficult to implement (for instance, C=A*B^T and C=A^T*B for MATMPIDENSE)? Thank you.

Cheers

Gao
________________________________
From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] on behalf of Jed Brown [jedbrown at mcs.anl.gov<mailto:jedbrown at mcs.anl.gov>]
Sent: Thursday, April 05, 2012 2:32 PM
To: PETSc users list; Hong Zhang

Subject: Re: [petsc-users] question about MatMatMultTranspose

On Thu, Apr 5, 2012 at 05:16, Gao Bin <bin.gao at uit.no<mailto:bin.gao at uit.no>> wrote:
Thank you for your quick reply. But as pointed out at http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMatTransposeMult.html:

This routine is currently only implemented for pairs of SeqAIJ matrices. C will be of type MATSEQAIJ.

Therefore I can not use it for dense matrix, am I right? If so, will MatMatTransposeMult be extended for other types of matrix later on? Thank you very much.

This is much simpler than the sparse case. Hong, did you intend to get around to this?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/83f9c67f/attachment.htm>

From ywu at culink.Colorado.EDU  Thu Apr  5 11:56:52 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Thu,  5 Apr 2012 10:56:52 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
Message-ID: <20120405105652.APZ53790@batman.int.colorado.edu>

Thank you, Barry.

I have send my code to petsc-maint at mcs.anl.gov. Thank you for your help.

Yuqi

---- Original message ----
>Date: Thu, 5 Apr 2012 08:10:03 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>   It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>
>
>   Barry 
>
>   If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>
>
>On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>
>> Thank you Barry.
>> 
>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>> 
>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>> 
>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>> 
>>  /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>  /* so use those from global PC */
>>  /* Is this what we always want? What if user wants to keep old one? */
>>  ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>  ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>  ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>  if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>    ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>    ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>  }
>> 
>> Thank you.
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>> 
>>>  When it gets to the lines 
>>> 
>>> if (pc->ops->setup) {
>>>   ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>> }
>>> 
>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>> 
>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>> 
>>>  It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>> 
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>> 
>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>> 
>>>>> Setting PC with identical preconditioner\
>>>>> Setting up new PC
>>>>> Setting up PC with same nonzero pattern\
>>>>> Setting up PC with different nonzero pattern\n
>>>>> 
>>>>> How many, and exactly what messages of this form are you getting?
>>>>> 
>>>>>  When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>> 
>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>> 
>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>> 
>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>> 
>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>> 
>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>  Barry
>>>>> 
>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Thanks, Adam.
>>>>>> 
>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>> 
>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>> 
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>> 
>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>> 
>>>>>>> Mark
>>>>>>> 
>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Thank you.
>>>>>>>> 
>>>>>>>> Can I ask another question? 
>>>>>>>> 
>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>> 
>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>> 
>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>> 
>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Barry,
>>>>>>>>>> 
>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>> 
>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>> 
>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>> 
>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>> 
>>>>>>>>>> Best
>>>>>>>>>> 
>>>>>>>>>> Yuqi
>>>>>>>>>> 
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>> 
>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>> 
>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>

From hanangul12 at yahoo.co.uk  Thu Apr  5 13:12:41 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 5 Apr 2012 19:12:41 +0100 (BST)
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
Message-ID: <1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>

Hi,, 
Thanks for more details!! 
Recursive two level multigrid? (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
No Problem with recursion and nested iterations. 
Coming to two level, 
How can I get exactly Prec = C + ?M^-1 (I - A*C) in a two level multigrid ? 
What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
to get the "Prec" ? 
what operators should be chosen in these two ksp contexts ? 


Awiaitng, 
with regards, Abdul











>________________________________
> From: Jed Brown <jedbrown at mcs.anl.gov>
>To: PETSc users list <petsc-users at mcs.anl.gov> 
>Cc: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
>Sent: Thursday, 5 April 2012, 17:56
>Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
>
>On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>? ?C + ?M^-1 (I - A*C) ? is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way.
>
>It looks to me like he wants to do nested iteration with recursive coarse levels rather than just iteration in a smoother.
>
>
>I need the accuracy at first and somehow at second coarse level.?
>After these two levels, the number of iterations at each level till coarsest is constantly one. ?
>
>
>Okay, but one iteration of a preconditioned Krylov method is still two preconditioner applications, so the method would still be exponential in the number of levels. If Richardson iteration is okay, you can use -pc_mg_cycle_type w to do two iterations instead of 1. The code would actually work fine with increasing this further, but it looks like you would currently have to cheat a bit to get many recursive cycles.
>
>
>PCMGSetCycleType(pc,(PCMGCycleType)5);
>
>
>There is currently no interface for setting a different number of cycles on each level, but you could include mgimpl.h and change the levels->cycles manually.
>
>
>Note that W cycles are already often considered to be too much work on coarse levels to justify in a parallel setting, so it's somewhat unlikely that running more coarse level iterations will really be better than a more standard multigrid with stronger smoothers.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/a5b62f3e/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr  5 13:17:49 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 11:17:49 -0700
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
Message-ID: <CAM9tzSnPdOqkpVFkCXKBoDRywc2NS-01Vz8GjxzkriVOPfS9Yw@mail.gmail.com>

On Thu, Apr 5, 2012 at 11:12, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>wrote:

> Recursive two level multigrid  (by using PCMG with my desired stuff in it,
> in PCSHELL) is okay. I will do it recursively.
> No Problem with recursion and nested iterations.
> Coming to two level,
> How can I get exactly Prec = C +  M^-1 (I - A*C) in a two level multigrid
> ?
> What should be specific choice of pre_smoother_ksp_context and
> post_smoother_ksp_context
> to get the "Prec" ?
> what operators should be chosen in these two ksp contexts ?
>

This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one
post-smoothing iteration).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/65a58f43/attachment.htm>

From hanangul12 at yahoo.co.uk  Thu Apr  5 13:24:37 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 5 Apr 2012 19:24:37 +0100 (BST)
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <CAM9tzSnPdOqkpVFkCXKBoDRywc2NS-01Vz8GjxzkriVOPfS9Yw@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
	<CAM9tzSnPdOqkpVFkCXKBoDRywc2NS-01Vz8GjxzkriVOPfS9Yw@mail.gmail.com>
Message-ID: <1333650277.79322.YahooMailNeo@web28502.mail.ukl.yahoo.com>




>________________________________
> From: Jed Brown <jedbrown at mcs.anl.gov>
>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
>Cc: PETSc users list <petsc-users at mcs.anl.gov> 
>Sent: Thursday, 5 April 2012, 20:17
>Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
>
>On Thu, Apr 5, 2012 at 11:12, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> wrote:
>
>Recursive two level multigrid? (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
>>No Problem with recursion and nested iterations. 
>>Coming to two level, 
>>How can I get exactly Prec = C + ?M^-1 (I - A*C) in a two level multigrid ? 
>>What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
>>to get the "Prec" ? 
>>what operators should be chosen in these two ksp contexts ? 
>
>This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one post-smoothing iteration).Thanks. and cleared!
>
>
>Post-smoothing context details would be appreciated!?
>To my understing, those would be:?
>1 post-smoothing iteration with?
>KSP type Richardson?
>PC type to be applied on M is PCShellForM (say there PCLU in PCShellForM, though I need to change this, so I say this) .?
>
>
>This will complete my preconditioner "Prec" ???
>
>
>Bundle of thanks,?
>Abdul?
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/bb2088df/attachment.htm>

From bsmith at mcs.anl.gov  Thu Apr  5 13:27:19 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 13:27:19 -0500
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
Message-ID: <D7BDB503-0A42-4560-B56C-FD68B5B95D67@mcs.anl.gov>


    
On Apr 5, 2012, at 1:12 PM, Abdul Hanan Sheikh wrote:

> Hi,, 
> Thanks for more details!! 
> Recursive two level multigrid  (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
> No Problem with recursion and nested iterations. 
> Coming to two level, 
> How can I get exactly Prec = C +  M^-1 (I - A*C) in a two level multigrid ? 
> What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
> to get the "Prec" ? 
> what operators should be chosen in these two ksp contexts ? 

   If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.

   I see Jed just wrote "This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one post-smoothing iteration)."  This is also correct and just a different way of expressing the same thing.


    Barry

What motivates you to use this form? It is non-standard. Normal multigrid with a pre-smooth and post-smooth (both M^-1) can be written algebraically as 

M^-1   + (I - M^-1A)C(I- AM^-1)  which also equals M^-1 + C - CAM^-1 - M^-1A C + M^-1 A C A M^-1

Note that I don't view writing the operations out this way as particularly informative (for example witing it as C +  M^-1 (I - A*C)), it conceals the fact that your algorithm first projects the error onto the coarse grid space and then projects the new error onto some fine grid space. Normal multigrid with a pre and post first projects the error onto some fine grid space, then projects the new error onto the coarse grid space then projects the newest error onto the fine grid space again.
    
> 
> 
> Awiaitng, 
> with regards, Abdul
> 
> 
> 
> 
> 
> 
> 
> 
> From: Jed Brown <jedbrown at mcs.anl.gov>
> To: PETSc users list <petsc-users at mcs.anl.gov> 
> Cc: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
> Sent: Thursday, 5 April 2012, 17:56
> Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
> On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:
>    C +  M^-1 (I - A*C)   is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way.
> 
> It looks to me like he wants to do nested iteration with recursive coarse levels rather than just iteration in a smoother.
> 
> I need the accuracy at first and somehow at second coarse level. 
> After these two levels, the number of iterations at each level till coarsest is constantly one.  
> 
> Okay, but one iteration of a preconditioned Krylov method is still two preconditioner applications, so the method would still be exponential in the number of levels. If Richardson iteration is okay, you can use -pc_mg_cycle_type w to do two iterations instead of 1. The code would actually work fine with increasing this further, but it looks like you would currently have to cheat a bit to get many recursive cycles.
> 
> PCMGSetCycleType(pc,(PCMGCycleType)5);
> 
> There is currently no interface for setting a different number of cycles on each level, but you could include mgimpl.h and change the levels->cycles manually.
> 
> Note that W cycles are already often considered to be too much work on coarse levels to justify in a parallel setting, so it's somewhat unlikely that running more coarse level iterations will really be better than a more standard multigrid with stronger smoothers.
> 
> 


From jedbrown at mcs.anl.gov  Thu Apr  5 13:27:21 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 5 Apr 2012 11:27:21 -0700
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333650277.79322.YahooMailNeo@web28502.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
	<CAM9tzSnPdOqkpVFkCXKBoDRywc2NS-01Vz8GjxzkriVOPfS9Yw@mail.gmail.com>
	<1333650277.79322.YahooMailNeo@web28502.mail.ukl.yahoo.com>
Message-ID: <CAM9tzS=FeP9Vex69sVxbKQxRO=JWTDhtzPO=nZCGcifJaRS3Aw@mail.gmail.com>

On Thu, Apr 5, 2012 at 11:24, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>wrote:

> Post-smoothing context details would be appreciated!
> To my understing, those would be:
> 1 post-smoothing iteration with
> KSP type Richardson
> PC type to be applied on M is PCShellForM (say there PCLU in PCShellForM,
> though I need to change this, so I say this) .
>
> This will complete my preconditioner "Prec" ??
>

What are you asking? You can use any preconditioner you like for the
post-smoother.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/7bc2bf5f/attachment-0001.htm>

From ywu at culink.Colorado.EDU  Thu Apr  5 14:06:22 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Thu,  5 Apr 2012 13:06:22 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
Message-ID: <20120405130622.APZ65036@batman.int.colorado.edu>

Hi, Barry,

If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);

Otherwise, I got the following error messages in PETSc
[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!

Can I use the following commands to setup the coarse operator
      
ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

Thank you.

Best

Yuqi


---- Original message ----
>Date: Thu, 5 Apr 2012 08:10:03 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>   It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>
>
>   Barry 
>
>   If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>
>
>On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>
>> Thank you Barry.
>> 
>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>> 
>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>> 
>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>> 
>>  /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>  /* so use those from global PC */
>>  /* Is this what we always want? What if user wants to keep old one? */
>>  ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>  ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>  ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>  if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>    ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>    ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>  }
>> 
>> Thank you.
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>> 
>>>  When it gets to the lines 
>>> 
>>> if (pc->ops->setup) {
>>>   ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>> }
>>> 
>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>> 
>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>> 
>>>  It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>> 
>>> Barry
>>> 
>>> 
>>> 
>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>> 
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>> 
>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>> 
>>>>> Setting PC with identical preconditioner\
>>>>> Setting up new PC
>>>>> Setting up PC with same nonzero pattern\
>>>>> Setting up PC with different nonzero pattern\n
>>>>> 
>>>>> How many, and exactly what messages of this form are you getting?
>>>>> 
>>>>>  When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>> 
>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>> 
>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>> 
>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>> 
>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>> 
>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>  Barry
>>>>> 
>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Thanks, Adam.
>>>>>> 
>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>> 
>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>> 
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>> 
>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>> 
>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>> 
>>>>>>> Mark
>>>>>>> 
>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Thank you.
>>>>>>>> 
>>>>>>>> Can I ask another question? 
>>>>>>>> 
>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>> 
>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>> 
>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>> 
>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Barry,
>>>>>>>>>> 
>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>> 
>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>> 
>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>> 
>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>> 
>>>>>>>>>> Best
>>>>>>>>>> 
>>>>>>>>>> Yuqi
>>>>>>>>>> 
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>> 
>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>> 
>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Thu Apr  5 14:10:02 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 14:10:02 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120405130622.APZ65036@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
Message-ID: <C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>


  Yes, at least that is the intention.

  Barry

On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:

> Hi, Barry,
> 
> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
> 
> Otherwise, I got the following error messages in PETSc
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
> 
> Can I use the following commands to setup the coarse operator
> 
> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
> 
> Thank you.
> 
> Best
> 
> Yuqi
> 
> 
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>>  It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>> 
>> 
>>  Barry 
>> 
>>  If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>> 
>> 
>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>> 
>>> Thank you Barry.
>>> 
>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>> 
>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>> 
>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>> 
>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>> /* so use those from global PC */
>>> /* Is this what we always want? What if user wants to keep old one? */
>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>   ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>   ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>> }
>>> 
>>> Thank you.
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>> 
>>>> When it gets to the lines 
>>>> 
>>>> if (pc->ops->setup) {
>>>>  ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>> }
>>>> 
>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>> 
>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>> 
>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>> 
>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>> 
>>>>>> Setting PC with identical preconditioner\
>>>>>> Setting up new PC
>>>>>> Setting up PC with same nonzero pattern\
>>>>>> Setting up PC with different nonzero pattern\n
>>>>>> 
>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>> 
>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>> 
>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>> 
>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>> 
>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>> 
>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>> 
>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Thanks, Adam.
>>>>>>> 
>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>> 
>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>> 
>>>>>>> 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>> 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>> 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>> 
>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>> 
>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>> 
>>>>>>>> Mark
>>>>>>>> 
>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Thank you.
>>>>>>>>> 
>>>>>>>>> Can I ask another question? 
>>>>>>>>> 
>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>> 
>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>> 
>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>> 
>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>> 
>>>>>>>>> Best
>>>>>>>>> 
>>>>>>>>> Yuqi
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>> 
>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>> 
>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>> 
>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>> 
>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>> 
>>>>>>>>>>> Best
>>>>>>>>>>> 
>>>>>>>>>>> Yuqi
>>>>>>>>>>> 
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 


From hanangul12 at yahoo.co.uk  Thu Apr  5 14:27:35 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 5 Apr 2012 20:27:35 +0100 (BST)
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <D7BDB503-0A42-4560-B56C-FD68B5B95D67@mcs.anl.gov>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
	<D7BDB503-0A42-4560-B56C-FD68B5B95D67@mcs.anl.gov>
Message-ID: <1333654055.77827.YahooMailNeo@web28512.mail.ukl.yahoo.com>

Thanks for explaining all .?

Purpose is to implement an adapted deflation preconditioner.?
Basically, it has nothing to do more with multigrid algorithm.?
To have PETSC codes, i was left out to use PCMG framework to implment this.?

? If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.

Does C_c mean recursive action ? if so, then fine!?

Setting M^-1 preconditioner in ksp_post context with what ksp_type ? ?to my understanding, Richardson ?
?Just need an endorsement this time.?

Thank you!
regards, Abdul




>________________________________
> From: Barry Smith <bsmith at mcs.anl.gov>
>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
>Cc: Jed Brown <jedbrown at mcs.anl.gov> 
>Sent: Thursday, 5 April 2012, 20:27
>Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
>
>? ? 
>On Apr 5, 2012, at 1:12 PM, Abdul Hanan Sheikh wrote:
>
>> Hi,, 
>> Thanks for more details!! 
>> Recursive two level multigrid? (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
>> No Problem with recursion and nested iterations. 
>> Coming to two level, 
>> How can I get exactly Prec = C +? M^-1 (I - A*C) in a two level multigrid ? 
>> What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
>> to get the "Prec" ? 
>> what operators should be chosen in these two ksp contexts ? 
>
>?  If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.
>
>?  I see Jed just wrote "This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one post-smoothing iteration)."? This is also correct and just a different way of expressing the same thing.
>
>
>? ? Barry
>
>What motivates you to use this form? It is non-standard. Normal multigrid with a pre-smooth and post-smooth (both M^-1) can be written algebraically as 
>
>M^-1?  + (I - M^-1A)C(I- AM^-1)? which also equals M^-1 + C - CAM^-1 - M^-1A C + M^-1 A C A M^-1
>
>Note that I don't view writing the operations out this way as particularly informative (for example witing it as C +? M^-1 (I - A*C)), it conceals the fact that your algorithm first projects the error onto the coarse grid space and then projects the new error onto some fine grid space. Normal multigrid with a pre and post first projects the error onto some fine grid space, then projects the new error onto the coarse grid space then projects the newest error onto the fine grid space again.
>? ? 
>> 
>> 
>> Awiaitng, 
>> with regards, Abdul
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> From: Jed Brown <jedbrown at mcs.anl.gov>
>> To: PETSc users list <petsc-users at mcs.anl.gov> 
>> Cc: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
>> Sent: Thursday, 5 April 2012, 17:56
>> Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
>> 
>> On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>? ? C +? M^-1 (I - A*C)?  is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way.
>> 
>> It looks to me like he wants to do nested iteration with recursive coarse levels rather than just iteration in a smoother.
>> 
>> I need the accuracy at first and somehow at second coarse level. 
>> After these two levels, the number of iterations at each level till coarsest is constantly one.? 
>> 
>> Okay, but one iteration of a preconditioned Krylov method is still two preconditioner applications, so the method would still be exponential in the number of levels. If Richardson iteration is okay, you can use -pc_mg_cycle_type w to do two iterations instead of 1. The code would actually work fine with increasing this further, but it looks like you would currently have to cheat a bit to get many recursive cycles.
>> 
>> PCMGSetCycleType(pc,(PCMGCycleType)5);
>> 
>> There is currently no interface for setting a different number of cycles on each level, but you could include mgimpl.h and change the levels->cycles manually.
>> 
>> Note that W cycles are already often considered to be too much work on coarse levels to justify in a parallel setting, so it's somewhat unlikely that running more coarse level iterations will really be better than a more standard multigrid with stronger smoothers.
>> 
>> 
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/a7ebd0ee/attachment-0001.htm>

From bsmith at mcs.anl.gov  Thu Apr  5 14:32:06 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 14:32:06 -0500
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333654055.77827.YahooMailNeo@web28512.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
	<D7BDB503-0A42-4560-B56C-FD68B5B95D67@mcs.anl.gov>
	<1333654055.77827.YahooMailNeo@web28512.mail.ukl.yahoo.com>
Message-ID: <25739E2D-B048-4DF4-B228-83ED86033DF4@mcs.anl.gov>


On Apr 5, 2012, at 2:27 PM, Abdul Hanan Sheikh wrote:

> Thanks for explaining all . 
> 
> Purpose is to implement an adapted deflation preconditioner. 
> Basically, it has nothing to do more with multigrid algorithm. 
> To have PETSC codes, i was left out to use PCMG framework to implment this. 
> 
>   If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.
> 
> Does C_c mean recursive action ? if so, then fine! 

   What do you mean by recursive action? As I outlined it will solve the system represented by A_c (by a direct solver by default but you can change that).

> 
> Setting M^-1 preconditioner in ksp_post context with what ksp_type ?  to my understanding, Richardson ?

   Yes, or if you only apply once then preonly.

>  Just need an endorsement this time. 
> 
> Thank you!
> regards, Abdul
> 
> 
> 
> From: Barry Smith <bsmith at mcs.anl.gov>
> To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
> Cc: Jed Brown <jedbrown at mcs.anl.gov> 
> Sent: Thursday, 5 April 2012, 20:27
> Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
> 
>     
> On Apr 5, 2012, at 1:12 PM, Abdul Hanan Sheikh wrote:
> 
> > Hi,, 
> > Thanks for more details!! 
> > Recursive two level multigrid  (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
> > No Problem with recursion and nested iterations. 
> > Coming to two level, 
> > How can I get exactly Prec = C +  M^-1 (I - A*C) in a two level multigrid ? 
> > What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
> > to get the "Prec" ? 
> > what operators should be chosen in these two ksp contexts ? 
> 
>   If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.
> 
>   I see Jed just wrote "This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one post-smoothing iteration)."  This is also correct and just a different way of expressing the same thing.
> 
> 
>     Barry
> 
> What motivates you to use this form? It is non-standard. Normal multigrid with a pre-smooth and post-smooth (both M^-1) can be written algebraically as 
> 
> M^-1  + (I - M^-1A)C(I- AM^-1)  which also equals M^-1 + C - CAM^-1 - M^-1A C + M^-1 A C A M^-1
> 
> Note that I don't view writing the operations out this way as particularly informative (for example witing it as C +  M^-1 (I - A*C)), it conceals the fact that your algorithm first projects the error onto the coarse grid space and then projects the new error onto some fine grid space. Normal multigrid with a pre and post first projects the error onto some fine grid space, then projects the new error onto the coarse grid space then projects the newest error onto the fine grid space again.
>     
> > 
> > 
> > Awiaitng, 
> > with regards, Abdul
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > From: Jed Brown <jedbrown at mcs.anl.gov>
> > To: PETSc users list <petsc-users at mcs.anl.gov> 
> > Cc: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
> > Sent: Thursday, 5 April 2012, 17:56
> > Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> > 
> > On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >    C +  M^-1 (I - A*C)  is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way.
> > 
> > It looks to me like he wants to do nested iteration with recursive coarse levels rather than just iteration in a smoother.
> > 
> > I need the accuracy at first and somehow at second coarse level. 
> > After these two levels, the number of iterations at each level till coarsest is constantly one.  
> > 
> > Okay, but one iteration of a preconditioned Krylov method is still two preconditioner applications, so the method would still be exponential in the number of levels. If Richardson iteration is okay, you can use -pc_mg_cycle_type w to do two iterations instead of 1. The code would actually work fine with increasing this further, but it looks like you would currently have to cheat a bit to get many recursive cycles.
> > 
> > PCMGSetCycleType(pc,(PCMGCycleType)5);
> > 
> > There is currently no interface for setting a different number of cycles on each level, but you could include mgimpl.h and change the levels->cycles manually.
> > 
> > Note that W cycles are already often considered to be too much work on coarse levels to justify in a parallel setting, so it's somewhat unlikely that running more coarse level iterations will really be better than a more standard multigrid with stronger smoothers.
> > 
> > 
> 
> 
> 


From bsmith at mcs.anl.gov  Thu Apr  5 15:00:43 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 15:00:43 -0500
Subject: [petsc-users] Implementation of Multilevel preconditioner
In-Reply-To: <1333654718.19981.YahooMailNeo@web28503.mail.ukl.yahoo.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394774C13C@MBS04.uit.no>
	<1333639171.97359.YahooMailNeo@web28514.mail.ukl.yahoo.com>
	<1333639356.68392.YahooMailNeo@web28503.mail.ukl.yahoo.com>
	<B99495E1-3FF4-4C87-BA94-1A69D2AE53BF@mcs.anl.gov>
	<CAM9tzSkpZxJTQuuVMgiTrX63t8G-ft1aO2B3jp0LVrS+ZxPxHw@mail.gmail.com>
	<1333649561.71078.YahooMailNeo@web28516.mail.ukl.yahoo.com>
	<D7BDB503-0A42-4560-B56C-FD68B5B95D67@mcs.anl.gov>
	<1333654055.77827.YahooMailNeo@web28512.mail.ukl.yahoo.com>
	<25739E2D-B048-4DF4-B228-83ED86033DF4@mcs.anl.gov>
	<1333654718.19981.YahooMailNeo@web28503.mail.ukl.yahoo.com>
Message-ID: <C9FD3363-607A-4CAE-A978-D21554BC70AE@mcs.anl.gov>


  preonly is the same as Richardson with exactly one application of the preconditioner so you can use either one.

   Barry

On Apr 5, 2012, at 2:38 PM, Abdul Hanan Sheikh wrote:

> 
> From: Barry Smith <bsmith at mcs.anl.gov>
> To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
> Cc: PETSc users list <petsc-users at mcs.anl.gov>; Jed Brown <jedbrown at mcs.anl.gov> 
> Sent: Thursday, 5 April 2012, 21:32
> Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> 
> 
> On Apr 5, 2012, at 2:27 PM, Abdul Hanan Sheikh wrote:
> 
> > Thanks for explaining all . 
> > 
> > Purpose is to implement an adapted deflation preconditioner. 
> > Basically, it has nothing to do more with multigrid algorithm. 
> > To have PETSC codes, i was left out to use PCMG framework to implment this. 
> > 
> >  If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1 apply shell to post-smooth KSP.
> > 
> > Does C_c mean recursive action ? if so, then fine! 
> 
>   What do you mean by recursive action? As I outlined it will solve the system represented by A_c (by a direct solver by default but you can change that).
> Got it!  it was A_c !  ok. 
> > 
> > Setting M^-1 preconditioner in ksp_post context with what ksp_type ?  to my understanding, Richardson ?
> 
>   Yes, or if you only apply once then preonly.
> With Richardson ksp type, will not it apply once ?
> pre only will not lead to my required preconditioner, i guess. I need one Richardson with M^-1 preconditioenr. 
> 
> >  Just need an endorsement this time. 
> > 
> > Thank you!
> > regards, Abdul
> > 
> > 
> > 
> > From: Barry Smith <bsmith at mcs.anl.gov>
> > To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
> > Cc: Jed Brown <jedbrown at mcs.anl.gov> 
> > Sent: Thursday, 5 April 2012, 20:27
> > Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> > 
> > 
> >    
> > On Apr 5, 2012, at 1:12 PM, Abdul Hanan Sheikh wrote:
> > 
> > > Hi,, 
> > > Thanks for more details!! 
> > > Recursive two level multigrid  (by using PCMG with my desired stuff in it, in PCSHELL) is okay. I will do it recursively. 
> > > No Problem with recursion and nested iterations. 
> > > Coming to two level, 
> > > How can I get exactly Prec = C +  M^-1 (I - A*C) in a two level multigrid ? 
> > > What should be specific choice of pre_smoother_ksp_context and post_smoother_ksp_context 
> > > to get the "Prec" ? 
> > > what operators should be chosen in these two ksp contexts ? 
> > 
> >  If you truly want this form exactly; then it is 0 iterations of a pre-smooth and the KSP you set for the coarse mesh is C_c while you set the M^-1  apply shell to post-smooth KSP.
> > 
> >  I see Jed just wrote "This is just -pc_mg_type kaskade (zero pre-smoothing iterations, one post-smoothing iteration)."  This is also correct and just a different way of expressing the same thing.
> > 
> > 
> >    Barry
> > 
> > What motivates you to use this form? It is non-standard. Normal multigrid with a pre-smooth and post-smooth (both M^-1) can be written algebraically as 
> > 
> > M^-1  + (I - M^-1A)C(I- AM^-1)  which also equals M^-1 + C - CAM^-1 - M^-1A C + M^-1 A C A M^-1
> > 
> > Note that I don't view writing the operations out this way as particularly informative (for example witing it as C +  M^-1 (I - A*C)), it conceals the fact that your algorithm first projects the error onto the coarse grid space and then projects the new error onto some fine grid space. Normal multigrid with a pre and post first projects the error onto some fine grid space, then projects the new error onto the coarse grid space then projects the newest error onto the fine grid space again.
> >    
> > > 
> > > 
> > > Awiaitng, 
> > > with regards, Abdul
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > From: Jed Brown <jedbrown at mcs.anl.gov>
> > > To: PETSc users list <petsc-users at mcs.anl.gov> 
> > > Cc: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> 
> > > Sent: Thursday, 5 April 2012, 17:56
> > > Subject: Re: [petsc-users] Implementation of Multilevel preconditioner
> > > 
> > > On Thu, Apr 5, 2012 at 08:38, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >    C +  M^-1 (I - A*C)  is exactly a multiplicative Schwarz method with two preconditioners. Or in another way to put it it is exactly a two level multigrid so you can use PCMG in a straightforward way.
> > > 
> > > It looks to me like he wants to do nested iteration with recursive coarse levels rather than just iteration in a smoother.
> > > 
> > > I need the accuracy at first and somehow at second coarse level. 
> > > After these two levels, the number of iterations at each level till coarsest is constantly one.  
> > > 
> > > Okay, but one iteration of a preconditioned Krylov method is still two preconditioner applications, so the method would still be exponential in the number of levels. If Richardson iteration is okay, you can use -pc_mg_cycle_type w to do two iterations instead of 1. The code would actually work fine with increasing this further, but it looks like you would currently have to cheat a bit to get many recursive cycles.
> > > 
> > > PCMGSetCycleType(pc,(PCMGCycleType)5);
> > > 
> > > There is currently no interface for setting a different number of cycles on each level, but you could include mgimpl.h and change the levels->cycles manually.
> > > 
> > > Note that W cycles are already often considered to be too much work on coarse levels to justify in a parallel setting, so it's somewhat unlikely that running more coarse level iterations will really be better than a more standard multigrid with stronger smoothers.
> > > 
> > > 
> > 
> > 
> > 
> 
> 
> 


From jiangwen84 at gmail.com  Thu Apr  5 19:40:16 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Thu, 5 Apr 2012 20:40:16 -0400
Subject: [petsc-users] one process is lost but others are still running
Message-ID: <CAMJxm+BecUvt4Prp1eMhNYBawFOzotjTuyHzLkiAz1PKjt=bPA@mail.gmail.com>

Hi?

I reported this several days ago and I found my code just hanged inside
Super LU Dist solve. For the test purpose, I let my code keep on solving a
same linear system many times. My code will still hang at solving step but
not at the same stage every time. My code was distributed on 4 nodes and
each node had 4 processes(totally 16 processes). Before it gets stuck, one
process will disappear, which means that I can no longer see it by the top
command. The Other 15 processes are still running. I think those processes
might not know that one has been lost and just keep on waiting for it. It
looks like the cluster system kills that process without giving me any
error information. I am pretty sure that the memory is quite big enough for
my calculation (each core has 6GB), so I cannot figure out what causes. I
have very little knowledge about the cluster system and could you give me
any hints on this issue. Is this the problem with PETSc, Super LU or the
cluster? Thanks.

Regards,

Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/d7dae697/attachment.htm>

From hzhang at mcs.anl.gov  Thu Apr  5 20:01:36 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 5 Apr 2012 20:01:36 -0500
Subject: [petsc-users] one process is lost but others are still running
In-Reply-To: <CAMJxm+BecUvt4Prp1eMhNYBawFOzotjTuyHzLkiAz1PKjt=bPA@mail.gmail.com>
References: <CAMJxm+BecUvt4Prp1eMhNYBawFOzotjTuyHzLkiAz1PKjt=bPA@mail.gmail.com>
Message-ID: <CAGCphBt3FZJ+mzd-Gbz2aiDE5Z6YvNHEXde3d9HPHJtg-W+G_A@mail.gmail.com>

Wen:
Do you use superlu_dist as parallel direct solver?
Suggest also install mumps.
(need F90 compiler, configure petsc with
'--download-blacs --download-scalapack --download-mumps').
When superlu_dist fails, switch to mumps
(use runtime option '-pc_factor_mat_solver_package mumps').
If both solvers fail, something might be wrong with your model or code.

Hong

>
>
> I reported this several days ago and I found my code just hanged inside
> Super LU Dist solve. For the test purpose, I let my code keep on solving a
> same linear system many times. My code will still hang at solving step but
> not at the same stage every time. My code was distributed on 4 nodes and
> each node had 4 processes(totally 16 processes). Before it gets stuck, one
> process will disappear, which means that I can no longer see it by the top
> command. The Other 15 processes are still running. I think those processes
> might not know that one has been lost and just keep on waiting for it. It
> looks like the cluster system kills that process without giving me any
> error information. I am pretty sure that the memory is quite big enough for
> my calculation (each core has 6GB), so I cannot figure out what causes. I
> have very little knowledge about the cluster system and could you give me
> any hints on this issue. Is this the problem with PETSc, Super LU or the
> cluster? Thanks.
>
> Regards,
>
> Wen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/568a8082/attachment.htm>

From ywu at culink.Colorado.EDU  Thu Apr  5 21:31:43 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Thu,  5 Apr 2012 20:31:43 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
Message-ID: <20120405203143.APZ95110@batman.int.colorado.edu>

Hi, Barry,

If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()

ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?

Best

Yuqi



---- Original message ----
>Date: Thu, 5 Apr 2012 14:10:02 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>  Yes, at least that is the intention.
>
>  Barry
>
>On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>
>> Hi, Barry,
>> 
>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>> 
>> Otherwise, I got the following error messages in PETSc
>> [0]PETSC ERROR: No support for this operation for this object type!
>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>> 
>> Can I use the following commands to setup the coarse operator
>> 
>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>> 
>> Thank you.
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> ---- Original message ----
>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>>  It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>> 
>>> 
>>>  Barry 
>>> 
>>>  If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>> 
>>> 
>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>> 
>>>> Thank you Barry.
>>>> 
>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>> 
>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>> 
>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>> 
>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>> /* so use those from global PC */
>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>   ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>   ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>> }
>>>> 
>>>> Thank you.
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>> 
>>>>> When it gets to the lines 
>>>>> 
>>>>> if (pc->ops->setup) {
>>>>>  ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>> }
>>>>> 
>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>> 
>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>> 
>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> 
>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>> 
>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>> 
>>>>>>> Setting PC with identical preconditioner\
>>>>>>> Setting up new PC
>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>> 
>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>> 
>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>> 
>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>> 
>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>> 
>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>> 
>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>> 
>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Thanks, Adam.
>>>>>>>> 
>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>> 
>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>> 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>> 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>> 
>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>> 
>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>> 
>>>>>>>>> Mark
>>>>>>>>> 
>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Thank you.
>>>>>>>>>> 
>>>>>>>>>> Can I ask another question? 
>>>>>>>>>> 
>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>> 
>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>> 
>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>> 
>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>> 
>>>>>>>>>> Best
>>>>>>>>>> 
>>>>>>>>>> Yuqi
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>> 
>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>> 
>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>> 
>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>> 
>>>>>>>>>>>> Best
>>>>>>>>>>>> 
>>>>>>>>>>>> Yuqi
>>>>>>>>>>>> 
>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Thu Apr  5 21:35:58 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 21:35:58 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120405203143.APZ95110@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
	<20120405203143.APZ95110@batman.int.colorado.edu>
Message-ID: <B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>


On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:

> Hi, Barry,
> 
> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
> 
> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

  Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.

   The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
  Barry


> 
> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
> 
> Best
> 
> Yuqi
> 
> 
> 
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> Yes, at least that is the intention.
>> 
>> Barry
>> 
>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>> 
>>> Hi, Barry,
>>> 
>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>> 
>>> Otherwise, I got the following error messages in PETSc
>>> [0]PETSC ERROR: No support for this operation for this object type!
>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>> 
>>> Can I use the following commands to setup the coarse operator
>>> 
>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>> 
>>> Thank you.
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>> 
>>>> 
>>>> Barry 
>>>> 
>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>> 
>>>> 
>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>> 
>>>>> Thank you Barry.
>>>>> 
>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>> 
>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>> 
>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>> 
>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>> /* so use those from global PC */
>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>  ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>  ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>> }
>>>>> 
>>>>> Thank you.
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>>> 
>>>>>> When it gets to the lines 
>>>>>> 
>>>>>> if (pc->ops->setup) {
>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>> }
>>>>>> 
>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>> 
>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>>> 
>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>> 
>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>> 
>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>> Setting up new PC
>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>> 
>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>> 
>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>>> 
>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>> 
>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>>> 
>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>> 
>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>>> 
>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Thanks, Adam.
>>>>>>>>> 
>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>> 
>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>> 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>> 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>> 
>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>> 
>>>>>>>>> Best
>>>>>>>>> 
>>>>>>>>> Yuqi
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>> 
>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>> 
>>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>> 
>>>>>>>>>> Mark
>>>>>>>>>> 
>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thank you.
>>>>>>>>>>> 
>>>>>>>>>>> Can I ask another question? 
>>>>>>>>>>> 
>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>>> 
>>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>>> 
>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>> 
>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>> 
>>>>>>>>>>> Best
>>>>>>>>>>> 
>>>>>>>>>>> Yuqi
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>> 
>>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 


From jiangwen84 at gmail.com  Thu Apr  5 22:09:12 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Thu, 5 Apr 2012 23:09:12 -0400
Subject: [petsc-users] one process is lost but others are still running
	(Hong Zhang)
Message-ID: <CAMJxm+A7+zMcJFFN3_zCTPcnxLuysq1M2wVbYcRbESh655nwgA@mail.gmail.com>

Hi Hong,

Thanks for your reply.

Yes, I am using superlu_dist as parallel direct solver. And I will ask our
cluster administrator to install mumps and give it a shot.

But I still have a question. For the test purpose, I have already made my
code as simple as just repeat solving a same well conditioning matrix. And
for most cases, it can do the solving, maybe 2 times, 3 times or even more
until it loses one process and get stuck. Basically what I did not quite
understand is why it did not give out any error information and abort the
job even one of their processes is gone. Is it just the problem with
cluster or might it also be the problem with the libraries. Thanks.

Regards,

Wen


Wen:
> Do you use superlu_dist as parallel direct solver?
> Suggest also install mumps.
> (need F90 compiler, configure petsc with
> '--download-blacs --download-scalapack --download-mumps').
> When superlu_dist fails, switch to mumps
> (use runtime option '-pc_factor_mat_solver_package mumps').
> If both solvers fail, something might be wrong with your model or code.
>
> Hong
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120405/da7879e8/attachment.htm>

From ywu at culink.Colorado.EDU  Thu Apr  5 22:09:14 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Thu,  5 Apr 2012 21:09:14 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
	<20120405203143.APZ95110@batman.int.colorado.edu>
	<B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>
Message-ID: <20120405210914.APZ95993@batman.int.colorado.edu>

Thank you, Barry.

Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works. And I found out some interesting things, it also triggers the LU factorization for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2 SNES iteration.

Can I also ask another question? Because MatPtAP is not supported for baij matrices. I want to setup a routine to calculate coarse operator if the fine grid matrices is of type baij. Is there any approach so that I can keep the symbolic pattern and only redo the numeric part in each SNES, similar to MatPtAPSymbolic and MatPtAPNumeric?

Best

Yuqi

 

---- Original message ----
>Date: Thu, 5 Apr 2012 21:35:58 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>
>> Hi, Barry,
>> 
>> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
>> 
>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>
>  Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>
>   The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
>  Barry
>
>
>> 
>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> Yes, at least that is the intention.
>>> 
>>> Barry
>>> 
>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Barry,
>>>> 
>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>> 
>>>> Otherwise, I got the following error messages in PETSc
>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>> 
>>>> Can I use the following commands to setup the coarse operator
>>>> 
>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>> 
>>>> Thank you.
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>>> 
>>>>> 
>>>>> Barry 
>>>>> 
>>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>>> 
>>>>> 
>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Thank you Barry.
>>>>>> 
>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>> 
>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>>> 
>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>>> 
>>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>>> /* so use those from global PC */
>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>>  ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>>  ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>> }
>>>>>> 
>>>>>> Thank you.
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>>>> 
>>>>>>> When it gets to the lines 
>>>>>>> 
>>>>>>> if (pc->ops->setup) {
>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>> }
>>>>>>> 
>>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>>> 
>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>>>> 
>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>>> 
>>>>>>> Barry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>>> 
>>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>>> 
>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>> Setting up new PC
>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>> 
>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>> 
>>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>>>> 
>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>>> 
>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>>>> 
>>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>>> 
>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>>>> 
>>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> Thanks, Adam.
>>>>>>>>>> 
>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>> 
>>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>>> 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>> 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>> 
>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>> 
>>>>>>>>>> Best
>>>>>>>>>> 
>>>>>>>>>> Yuqi
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>> 
>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>> 
>>>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>>> 
>>>>>>>>>>> Mark
>>>>>>>>>>> 
>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>> 
>>>>>>>>>>>> Can I ask another question? 
>>>>>>>>>>>> 
>>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>>>> 
>>>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>>>> 
>>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>>> 
>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>> 
>>>>>>>>>>>> Best
>>>>>>>>>>>> 
>>>>>>>>>>>> Yuqi
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Thu Apr  5 22:13:36 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 5 Apr 2012 22:13:36 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120405210914.APZ95993@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
	<20120405203143.APZ95110@batman.int.colorado.edu>
	<B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>
	<20120405210914.APZ95993@batman.int.colorado.edu>
Message-ID: <495E393A-57DF-4434-959D-86F5183E6B32@mcs.anl.gov>


On Apr 5, 2012, at 10:09 PM, Yuqi Wu wrote:

> Thank you, Barry.
> 
> Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works. And I found out some interesting things, it also triggers the LU factorization for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2 SNES iteration.
> 
> Can I also ask another question? Because MatPtAP is not supported for baij matrices. I want to setup a routine to calculate coarse operator if the fine grid matrices is of type baij. Is there any approach so that I can keep the symbolic pattern and only redo the numeric part in each SNES, similar to MatPtAPSymbolic and MatPtAPNumeric?

   You call the KSPOperators() with the coarse matrix in each Jacobian computation and just use the same coarse matrrix, just put new values into the nonzero locations. that is you do not need to create a new coarse matrix with matcreate each Jacobian just use the same as before. Call MatZeroEntries() before reusing.

   Barry

> 
> Best
> 
> Yuqi
> 
> 
> 
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 21:35:58 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>> 
>>> Hi, Barry,
>>> 
>>> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
>>> 
>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>> 
>> Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>> 
>>  The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
>> Barry
>> 
>> 
>>> 
>>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> Yes, at least that is the intention.
>>>> 
>>>> Barry
>>>> 
>>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Barry,
>>>>> 
>>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>>> 
>>>>> Otherwise, I got the following error messages in PETSc
>>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>>> 
>>>>> Can I use the following commands to setup the coarse operator
>>>>> 
>>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>>> 
>>>>> Thank you.
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>>>> 
>>>>>> 
>>>>>> Barry 
>>>>>> 
>>>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>>>> 
>>>>>> 
>>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Thank you Barry.
>>>>>>> 
>>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>>> 
>>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>>>> 
>>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>>>> 
>>>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>>>> /* so use those from global PC */
>>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>>> ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>>> }
>>>>>>> 
>>>>>>> Thank you.
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>>>>> 
>>>>>>>> When it gets to the lines 
>>>>>>>> 
>>>>>>>> if (pc->ops->setup) {
>>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>>> }
>>>>>>>> 
>>>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>>>> 
>>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>>>>> 
>>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>>>> 
>>>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>>>> 
>>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>>> Setting up new PC
>>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>>> 
>>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>>> 
>>>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>>>>> 
>>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>>>> 
>>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>>>>> 
>>>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>>>> 
>>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>>>>> 
>>>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>>>> 
>>>>>>>>> Best
>>>>>>>>> 
>>>>>>>>> Yuqi
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thanks, Adam.
>>>>>>>>>>> 
>>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>> 
>>>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>>>> 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>> 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>> 
>>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>> 
>>>>>>>>>>> Best
>>>>>>>>>>> 
>>>>>>>>>>> Yuqi
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>> 
>>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>>> 
>>>>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>>>> 
>>>>>>>>>>>> Mark
>>>>>>>>>>>> 
>>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Can I ask another question? 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 


From balay at mcs.anl.gov  Thu Apr  5 22:15:56 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 5 Apr 2012 22:15:56 -0500 (CDT)
Subject: [petsc-users] one process is lost but others are still running
 (Hong Zhang)
In-Reply-To: <CAMJxm+A7+zMcJFFN3_zCTPcnxLuysq1M2wVbYcRbESh655nwgA@mail.gmail.com>
References: <CAMJxm+A7+zMcJFFN3_zCTPcnxLuysq1M2wVbYcRbESh655nwgA@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1204052212580.30382@asterix>

No point in trying to debug old code. If you wish to debug this issue
- try upgrading to petsc-3.2 [or better yet petsc-dev] - and see if
the problem persists.

petsc-dev uses superlu_dist_3.0

Satish

On Thu, 5 Apr 2012, Wen Jiang wrote:

> Hi Hong,
> 
> Thanks for your reply.
> 
> Yes, I am using superlu_dist as parallel direct solver. And I will ask our
> cluster administrator to install mumps and give it a shot.
> 
> But I still have a question. For the test purpose, I have already made my
> code as simple as just repeat solving a same well conditioning matrix. And
> for most cases, it can do the solving, maybe 2 times, 3 times or even more
> until it loses one process and get stuck. Basically what I did not quite
> understand is why it did not give out any error information and abort the
> job even one of their processes is gone. Is it just the problem with
> cluster or might it also be the problem with the libraries. Thanks.
> 
> Regards,
> 
> Wen
> 
> 
> Wen:
> > Do you use superlu_dist as parallel direct solver?
> > Suggest also install mumps.
> > (need F90 compiler, configure petsc with
> > '--download-blacs --download-scalapack --download-mumps').
> > When superlu_dist fails, switch to mumps
> > (use runtime option '-pc_factor_mat_solver_package mumps').
> > If both solvers fail, something might be wrong with your model or code.
> >
> > Hong
> >
> >
> 


From mmnasr at gmail.com  Fri Apr  6 17:56:43 2012
From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani)
Date: Fri, 6 Apr 2012 15:56:43 -0700
Subject: [petsc-users] Coarsening vectors
Message-ID: <CAE_bc5bYtxva3RxPDFOo-Gk1JOmh0zqXuB-AgDNeZtW8Lk1N8g@mail.gmail.com>

Hi,

I just noticed the command DMCoarsen().
I was wondering if there is a simple command which copies the values of the
(global) vector created via the fine DM into the coarse DM?

Thanks,
Mohamad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120406/e94e69c3/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr  6 17:59:07 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 6 Apr 2012 17:59:07 -0500
Subject: [petsc-users] Coarsening vectors
In-Reply-To: <CAE_bc5bYtxva3RxPDFOo-Gk1JOmh0zqXuB-AgDNeZtW8Lk1N8g@mail.gmail.com>
References: <CAE_bc5bYtxva3RxPDFOo-Gk1JOmh0zqXuB-AgDNeZtW8Lk1N8g@mail.gmail.com>
Message-ID: <CAM9tzSmBOWnG1s6NeCPApf5YAtHLevU0mywokncQw=eLm6bZLQ@mail.gmail.com>

On Fri, Apr 6, 2012 at 17:56, Mohamad M. Nasr-Azadani <mmnasr at gmail.com>wrote:

> I just noticed the command DMCoarsen().
> I was wondering if there is a simple command which copies the values of
> the (global) vector created via the fine DM into the coarse DM?
>

with petsc-dev or petsc-3.2 respectively,

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateInterpolation.html
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMGetInterpolation.html

then

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatRestrict.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120406/cd8f846b/attachment.htm>

From mmnasr at gmail.com  Fri Apr  6 18:09:10 2012
From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani)
Date: Fri, 6 Apr 2012 16:09:10 -0700
Subject: [petsc-users] Coarsening vectors
In-Reply-To: <CAM9tzSmBOWnG1s6NeCPApf5YAtHLevU0mywokncQw=eLm6bZLQ@mail.gmail.com>
References: <CAE_bc5bYtxva3RxPDFOo-Gk1JOmh0zqXuB-AgDNeZtW8Lk1N8g@mail.gmail.com>
	<CAM9tzSmBOWnG1s6NeCPApf5YAtHLevU0mywokncQw=eLm6bZLQ@mail.gmail.com>
Message-ID: <CAE_bc5YEC9n6Zrdg+R6WMEroD7vBcuKit+Sq=j-CHrCw-Mxzvg@mail.gmail.com>

Thanks Jed. Makes sense.
For simple copying values, is there anyway around it so we don't create the
matrix?

Thanks,
Mohamad

On Fri, Apr 6, 2012 at 3:59 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Fri, Apr 6, 2012 at 17:56, Mohamad M. Nasr-Azadani <mmnasr at gmail.com>wrote:
>
>> I just noticed the command DMCoarsen().
>> I was wondering if there is a simple command which copies the values of
>> the (global) vector created via the fine DM into the coarse DM?
>>
>
> with petsc-dev or petsc-3.2 respectively,
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateInterpolation.html
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMGetInterpolation.html
>
> then
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatRestrict.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120406/90553388/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr  6 18:21:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 6 Apr 2012 18:21:26 -0500
Subject: [petsc-users] Coarsening vectors
In-Reply-To: <CAE_bc5YEC9n6Zrdg+R6WMEroD7vBcuKit+Sq=j-CHrCw-Mxzvg@mail.gmail.com>
References: <CAE_bc5bYtxva3RxPDFOo-Gk1JOmh0zqXuB-AgDNeZtW8Lk1N8g@mail.gmail.com>
	<CAM9tzSmBOWnG1s6NeCPApf5YAtHLevU0mywokncQw=eLm6bZLQ@mail.gmail.com>
	<CAE_bc5YEC9n6Zrdg+R6WMEroD7vBcuKit+Sq=j-CHrCw-Mxzvg@mail.gmail.com>
Message-ID: <CAM9tzSkBRqjP-eWr=v0w_av9aCbMreXM0nrtMO1+HhhMjUdQKQ@mail.gmail.com>

On Fri, Apr 6, 2012 at 18:09, Mohamad M. Nasr-Azadani <mmnasr at gmail.com>wrote:

> Thanks Jed. Makes sense.
> For simple copying values, is there anyway around it so we don't create
> the matrix?
>

No, but some implementations could define the matrix without storing
entries, so it wouldn't imply extra storage. (Whether this is practical or
not depends on the discretization.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120406/132e17bf/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Sat Apr  7 13:56:17 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Sat, 7 Apr 2012 21:56:17 +0300
Subject: [petsc-users] Binary I/O
In-Reply-To: <56B1417B-B315-45D5-A9CD-69483EB84C25@mcs.anl.gov>
References: <CAE_bc5Y5x2D7TMRE7Agd8rFZhB=dpgJi1tNRQkvxUKJ6zdOVQQ@mail.gmail.com>
	<3D695C1B-7D06-4CE7-A775-0A7FF52F2C4B@mcs.anl.gov>
	<CAE_bc5a41ZSZ3vUMGRYq-y85OJvx8u5Z+fX7eRJ-Yo3HxSMvfg@mail.gmail.com>
	<CAE_bc5YBKa2Y_PMXZb1LcaVQ48bjgggc0P5Zo4MW=yoMnok36g@mail.gmail.com>
	<56B1417B-B315-45D5-A9CD-69483EB84C25@mcs.anl.gov>
Message-ID: <CADVoUZSS0K5HWN2NjOeuYQjF5+wAFiwzyQgf9eOfJcKZ0_4b_A@mail.gmail.com>

Just to clarify this a little more as it came up recently in some of Amal's
thesis work using PyClaw.  On a BlueGene/P the I/O nodes are available at
some (usually fixed) density throughout the network.  On the BlueGene/P
system at KAUST, the majority of the system is available at the I/O node
density of 128 compute nodes to 1 I/O node.  If you use a non-MPIIO binary
viewer, you are going to be limited by the network connection of a single
I/O node to the parallel file system, which is in this case 10 Gb/s.  The
I/O nodes on BG/P are relatively low-powered ppc450 nodes, so it can be
rather difficult for them to drive a full 10 Gb/s (1.25 GB/s), and they are
limited as well by the collective network between the compute nodes and
each I/O node (I believe this is 0.75 GB/s on our system).

The throughput performance of our parallel file system is limited by the
file system controllers (each controller is backed by 8 servers), which
have theoretical limits of 3GB/s each.  We have a relatively low-powered
system with 4 controllers, but that brings our theoretical I/O to 12 GB/s.
 When we do not enable MPIIO, we're limited to at most 1.25 GB/s, because
we are only pushing on one of the I/O nodes, and we can theoretically
achieve a maximum of 10% of the throughput capability of the file system.
 Switching on MPIIO eventually makes the I/O nodes no longer the
theoretical bottleneck at 16 I/O nodes = 2048 compute nodes, or 2 racks of
BG/P.

A

On Thu, Oct 13, 2011 at 5:05 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Oct 12, 2011, at 7:40 PM, Mohamad M. Nasr-Azadani wrote:
>
> > Hi again,
> >
> > On the similar topic, how hard would it be to write a function similar
> to the PETSc's VecView() associated with Binary writer to do exactly the
> same thing, i.e. write a parallel vector into one single file but when
> writing all the processors are performing the task simultaneously?
>
>    Impossible. In the end in a standard filesystem the physical hard disk
> is connected to a single CPU and everything that gets written to that hard
> disk has to go through that one CPU; there is no way physically for a bunch
> of CPUs to write together onto a single physical disk.
>
>   Now in high end parallel file systems each file may be spread over
> several hard disks (this is sometimes called stripping) (say 8). In that
> case there is some parallelism in writing the data since eight different
> parts of the vector can be sent through 8 different CPUs to 8 disks. But
> note that in general the number of these disks that file is spread over is
> small, like 8, it is never 10,000. When you have a parallel file system and
> use the option -viewer_binary_mpiio then the PETSc VecView() uses MPI IO to
> do the writing and you do get this level of parallelism so you may get
> slightly better performance than not using MPI IO.
>
>   If you are seeing long wait times in VecView() with the binary file it
> is likely it is because the file server is connected via some pretty slow
> network to the actual compute nodes on parallel machine and nothing to do
> with the details of VecView(). You need to make sure you are writing
> directly to a disk that is on a compute node of the parallel machine, not
> over some network using NFS (Network File System), this can make a huge
> difference in time.
>
>   Barry
>
> >
> > Best,
> > Mohamad
> >
> >
> > On Wed, Oct 12, 2011 at 4:17 PM, Mohamad M. Nasr-Azadani <
> mmnasr at gmail.com> wrote:
> > Thanks Barry. That makes perfect sense.
> >
> > Best,
> > Mohamad
> >
> >
> > On Wed, Oct 12, 2011 at 3:50 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > On Oct 12, 2011, at 5:42 PM, Mohamad M. Nasr-Azadani wrote:
> >
> > > Hi everyone,
> > >
> > > I think I know the answer to my question, but I was double checking.
> > > When using
> > > PetscViewerBinaryOpen();
> > >
> > > It is mentioned that
> > > "For writing files it only opens the file on processor 0 in the
> communicator."
> > >
> > > Does that mean when writing a parallel vector to file using VecView(),
> all the data from other processors is first sent to processor zero and then
> dumped into the file?
> >
> >   No all the data is not sent to process zero before writing. That is
> process 0 does not need enough memory to store all the data before writing.
> >
> >    Instead the processes take turns sending data to process 0 who
> immediately writes it out out to disk.
> >
> > > If so, that would be a very slow processor for big datasets and large
> number of processor?
> >
> >   For less than a few thousand processes this is completely fine and
> nothing else would be much faster
> >
> > > Any suggestions to speed that process up?
> >
> >   We have the various MPI IO options that uses MPI IO to have several
> processes writing to disks at the same time that is useful for very large
> numbers of processes.
> >
> >   Barry
> >
> > >
> > > Best,
> > > Mohamad
> > >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/4283ef56/attachment.htm>

From zhenglun.wei at gmail.com  Sat Apr  7 15:41:31 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Sat, 7 Apr 2012 15:41:31 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
	stack below"?
Message-ID: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>

Dear All,
    I hope you're having a nice weekend.
    I got a problem with /ksp/ksp/example/tutorial. I was trying to execute
this example with petsc-dev in a share  memory dell workstation with 8
cores. The parameter for DMDACreate2d is 500, 500, which means it generates
a 1997*1997 2d grid. It is fine to run it with single process with either
"./ex29" or "mpiexec -np 1 ./exe29". However, it came across a error when I
tempted to run it with more than 1 process. The error message is like this,
when I was trying to execute it with 4 processes. I was thinking about if
the 'stack size' is the problem, so I enlarge it to 1000000000 with
'ulimit' command. However, the same error message keeps showing up.

[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[1]PETSCERROR:
or try
http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[2]PETSCERROR:
or try
http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[2]PETSC ERROR: likely location of problem given in stack below
[2]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] VecView_VTK line 231
src/ksp/ksp/examples/tutorials/ex29.c
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Signal received!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Development HG revision:
85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011
-0700
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by
zlwei Sat Apr  7 16:33:21 2012
[1]PETSC ERROR: Libraries linked from
/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[1]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[1]PETSC ERROR: Configure options --download-f-blas-lapack=1
--download-mpich=1 --with-cc=gcc --with-fc=gfortran
PETSC_ARCH=arch-linux2-c-debug
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
[cli_1]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[2]PETSC ERROR:       INSTEAD the line number of the start of the function
[2]PETSC ERROR:       is given.
[2]PETSC ERROR: [2] VecView_VTK line 231
src/ksp/ksp/examples/tutorials/ex29.c
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Signal received!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Development HG revision:
85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011
-0700
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by
zlwei Sat Apr  7 16:33:21 2012
[2]PETSC ERROR: Libraries linked from
/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[2]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[2]PETSC ERROR: Configure options --download-f-blas-lapack=1
--download-mpich=1 --with-cc=gcc --with-fc=gfortran
PETSC_ARCH=arch-linux2-c-debug
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
[cli_2]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[3]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[3]PETSCERROR:
or try
http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[3]PETSC ERROR: likely location of problem given in stack below
[3]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[3]PETSC ERROR:       INSTEAD the line number of the start of the function
[3]PETSC ERROR:       is given.
[3]PETSC ERROR: [3] VecView_VTK line 231
src/ksp/ksp/examples/tutorials/ex29.c
[3]PETSC ERROR: --------------------- Error Message
------------------------------------
[3]PETSC ERROR: Signal received!
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: Petsc Development HG revision:
85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011
-0700
[3]PETSC ERROR: See docs/changes/index.html for recent updates.
[3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[3]PETSC ERROR: See docs/index.html for manual pages.
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by
zlwei Sat Apr  7 16:33:21 2012
[3]PETSC ERROR: Libraries linked from
/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[3]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[3]PETSC ERROR: Configure options --download-f-blas-lapack=1
--download-mpich=1 --with-cc=gcc --with-fc=gfortran
PETSC_ARCH=arch-linux2-c-debug
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
[cli_3]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR:
or try
http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecView_VTK line 231
src/ksp/ksp/examples/tutorials/ex29.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011
-0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by
zlwei Sat Apr  7 16:33:21 2012
[0]PETSC ERROR: Libraries linked from
/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[0]PETSC ERROR: Configure options --download-f-blas-lapack=1
--download-mpich=1 --with-cc=gcc --with-fc=gfortran
PETSC_ARCH=arch-linux2-c-debug
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

thanks in advance,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/187230e8/attachment.htm>

From bsmith at mcs.anl.gov  Sat Apr  7 15:46:53 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 7 Apr 2012 15:46:53 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
	stack below"?
In-Reply-To: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
Message-ID: <0604BED6-D36F-4B8B-A1B6-435AC904736E@mcs.anl.gov>


  It is crashing in 2] VecView_VTK line 231 src/ksp/ksp/examples/tutorials/ex29.c

   This is an ASCII viewer and ASCII viewers cannot be used for large size problems.

    Barry


On Apr 7, 2012, at 3:41 PM, Alan Wei wrote:

> Dear All,
>     I hope you're having a nice weekend.
>     I got a problem with /ksp/ksp/example/tutorial. I was trying to execute this example with petsc-dev in a share  memory dell workstation with 8 cores. The parameter for DMDACreate2d is 500, 500, which means it generates a 1997*1997 2d grid. It is fine to run it with single process with either "./ex29" or "mpiexec -np 1 ./exe29". However, it came across a error when I tempted to run it with more than 1 process. The error message is like this, when I was trying to execute it with 4 processes. I was thinking about if the 'stack size' is the problem, so I enlarge it to 1000000000 with 'ulimit' command. However, the same error message keeps showing up.
> 
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] VecView_VTK line 231 src/ksp/ksp/examples/tutorials/ex29.c
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Signal received!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Development HG revision: 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011 -0700
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Sat Apr  7 16:33:21 2012
> [1]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
> [1]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
> [1]PETSC ERROR: Configure options --download-f-blas-lapack=1 --download-mpich=1 --with-cc=gcc --with-fc=gfortran PETSC_ARCH=arch-linux2-c-debug
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> [cli_1]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [2]PETSC ERROR:       INSTEAD the line number of the start of the function
> [2]PETSC ERROR:       is given.
> [2]PETSC ERROR: [2] VecView_VTK line 231 src/ksp/ksp/examples/tutorials/ex29.c
> [2]PETSC ERROR: --------------------- Error Message ------------------------------------
> [2]PETSC ERROR: Signal received!
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Development HG revision: 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011 -0700
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Sat Apr  7 16:33:21 2012
> [2]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
> [2]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
> [2]PETSC ERROR: Configure options --download-f-blas-lapack=1 --download-mpich=1 --with-cc=gcc --with-fc=gfortran PETSC_ARCH=arch-linux2-c-debug
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
> [cli_2]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [3]PETSC ERROR: likely location of problem given in stack below
> [3]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [3]PETSC ERROR:       INSTEAD the line number of the start of the function
> [3]PETSC ERROR:       is given.
> [3]PETSC ERROR: [3] VecView_VTK line 231 src/ksp/ksp/examples/tutorials/ex29.c
> [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> [3]PETSC ERROR: Signal received!
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Development HG revision: 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011 -0700
> [3]PETSC ERROR: See docs/changes/index.html for recent updates.
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Sat Apr  7 16:33:21 2012
> [3]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
> [3]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
> [3]PETSC ERROR: Configure options --download-f-blas-lapack=1 --download-mpich=1 --with-cc=gcc --with-fc=gfortran PETSC_ARCH=arch-linux2-c-debug
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
> [cli_3]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] VecView_VTK line 231 src/ksp/ksp/examples/tutorials/ex29.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision: 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011 -0700
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Sat Apr  7 16:33:21 2012
> [0]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
> [0]PETSC ERROR: Configure options --download-f-blas-lapack=1 --download-mpich=1 --with-cc=gcc --with-fc=gfortran PETSC_ARCH=arch-linux2-c-debug
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [cli_0]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> thanks in advance,
> Alan


From jedbrown at mcs.anl.gov  Sat Apr  7 15:49:35 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 7 Apr 2012 15:49:35 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
Message-ID: <CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>

On Sat, Apr 7, 2012 at 15:41, Alan Wei <zhenglun.wei at gmail.com> wrote:

>     I got a problem with /ksp/ksp/example/tutorial. I was trying to
> execute this example with petsc-dev in a share  memory dell workstation
> with 8 cores. The parameter for DMDACreate2d is 500, 500, which means it
> generates a 1997*1997 2d grid. It is fine to run it with single process
> with either "./ex29" or "mpiexec -np 1 ./exe29". However, it came across a
> error when I tempted to run it with more than 1 process. The error message
> is like this, when I was trying to execute it with 4 processes. I was
> thinking about if the 'stack size' is the problem, so I enlarge it to
> 1000000000 with 'ulimit' command. However, the same error message keeps
> showing up.


This custom viewer (in the example) is broken in parallel and has always
been broken in parallel. I don't know who wrote it, but it should be
removed since there is a binary VTK PetscViewer now.

Matt, this has come up a few times before and I thought you defended that
viewer. Is there anything it does that VecView with PETSCVIEWERVTK does not?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/1c37b86b/attachment.htm>

From zhenglun.wei at gmail.com  Sat Apr  7 16:34:35 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Sat, 7 Apr 2012 16:34:35 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
Message-ID: <CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>

Thanks,
Alan

On Sat, Apr 7, 2012 at 3:49 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Sat, Apr 7, 2012 at 15:41, Alan Wei <zhenglun.wei at gmail.com> wrote:
>
>>     I got a problem with /ksp/ksp/example/tutorial. I was trying to
>> execute this example with petsc-dev in a share  memory dell workstation
>> with 8 cores. The parameter for DMDACreate2d is 500, 500, which means it
>> generates a 1997*1997 2d grid. It is fine to run it with single process
>> with either "./ex29" or "mpiexec -np 1 ./exe29". However, it came across a
>> error when I tempted to run it with more than 1 process. The error message
>> is like this, when I was trying to execute it with 4 processes. I was
>> thinking about if the 'stack size' is the problem, so I enlarge it to
>> 1000000000 with 'ulimit' command. However, the same error message keeps
>> showing up.
>
>
> This custom viewer (in the example) is broken in parallel and has always
> been broken in parallel. I don't know who wrote it, but it should be
> removed since there is a binary VTK PetscViewer now.
>
> Matt, this has come up a few times before and I thought you defended that
> viewer. Is there anything it does that VecView with PETSCVIEWERVTK does not?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/fcc5f644/attachment.htm>

From zhenglun.wei at gmail.com  Sat Apr  7 16:36:44 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Sat, 7 Apr 2012 16:36:44 -0500
Subject: [petsc-users] Makefile in distributed-memory machine
Message-ID: <CAD7ekpC9ovBULzbe_1hPyNoeZJTA6ZC5HiPA2iifoyguJLJMjg@mail.gmail.com>

Dear All,
    I have another questions on /src/ksp/ksp/example/tutorial/ex29.c. I
just install PETSc-dev in a distributed-memory. Everything is fine and it
finished 'make test'.
    However, when I was trying to 'make' ex29.c, it comes this error
message:
makefile:18: /conf/variables: No such file or directory
makefile:19: /conf/rules: No such file or directory
makefile:1145: /conf/test: No such file or directory
make: *** No rule to make target `/conf/test'.  Stop.

   what should I do for this, Thanks, in advance,

Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/ddec3358/attachment.htm>

From jedbrown at mcs.anl.gov  Sat Apr  7 16:39:12 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 7 Apr 2012 16:39:12 -0500
Subject: [petsc-users] Makefile in distributed-memory machine
In-Reply-To: <CAD7ekpC9ovBULzbe_1hPyNoeZJTA6ZC5HiPA2iifoyguJLJMjg@mail.gmail.com>
References: <CAD7ekpC9ovBULzbe_1hPyNoeZJTA6ZC5HiPA2iifoyguJLJMjg@mail.gmail.com>
Message-ID: <CAM9tzS=czAZZ87PDfQXLks0S8p80GidQwmkrdhQT=NHtfJ0sTg@mail.gmail.com>

On Sat, Apr 7, 2012 at 16:36, Alan Wei <zhenglun.wei at gmail.com> wrote:

>     I have another questions on /src/ksp/ksp/example/tutorial/ex29.c. I
> just install PETSc-dev in a distributed-memory. Everything is fine and it
> finished 'make test'.
>     However, when I was trying to 'make' ex29.c, it comes this error
> message:
> makefile:18: /conf/variables: No such file or directory
> makefile:19: /conf/rules: No such file or directory
> makefile:1145: /conf/test: No such file or directory
> make: *** No rule to make target `/conf/test'.  Stop.
>
>    what should I do for this, Thanks, in advance,
>

Set the PETSC_DIR environment variable.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/a1a4c2ff/attachment.htm>

From sean at mcs.anl.gov  Sat Apr  7 16:39:15 2012
From: sean at mcs.anl.gov (Sean Farley)
Date: Sat, 7 Apr 2012 16:39:15 -0500
Subject: [petsc-users] Makefile in distributed-memory machine
In-Reply-To: <CAD7ekpC9ovBULzbe_1hPyNoeZJTA6ZC5HiPA2iifoyguJLJMjg@mail.gmail.com>
References: <CAD7ekpC9ovBULzbe_1hPyNoeZJTA6ZC5HiPA2iifoyguJLJMjg@mail.gmail.com>
Message-ID: <CANH+MCjK4q-AJZad+7BhchKOX=kNoX2Vm=dsQQRYjr5igqCFeA@mail.gmail.com>

>
>     I have another questions on /src/ksp/ksp/example/tutorial/ex29.c. I
> just install PETSc-dev in a distributed-memory. Everything is fine and it
> finished 'make test'.
>     However, when I was trying to 'make' ex29.c, it comes this error
> message:
> makefile:18: /conf/variables: No such file or directory
> makefile:19: /conf/rules: No such file or directory
> makefile:1145: /conf/test: No such file or directory
> make: *** No rule to make target `/conf/test'.  Stop.
>
>    what should I do for this, Thanks, in advance,
>

Make sure to set your environment variables PETSC_DIR and PETSC_ARCH, or
send them on the command line every time you run make:

$ PETSC_DIR=/path/to/petsc-dev PETSC_ARCH=arch-debug make ex29
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/fe11091b/attachment.htm>

From zhenglun.wei at gmail.com  Sat Apr  7 17:10:15 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Sat, 7 Apr 2012 17:10:15 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
Message-ID: <CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>

A little problem comes up. I commented out VTK part, the code still has the
same error message popping up.

Alan

On Sat, Apr 7, 2012 at 4:34 PM, Alan Wei <zhenglun.wei at gmail.com> wrote:

> Thanks,
> Alan
>
>
> On Sat, Apr 7, 2012 at 3:49 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Sat, Apr 7, 2012 at 15:41, Alan Wei <zhenglun.wei at gmail.com> wrote:
>>
>>>     I got a problem with /ksp/ksp/example/tutorial. I was trying to
>>> execute this example with petsc-dev in a share  memory dell workstation
>>> with 8 cores. The parameter for DMDACreate2d is 500, 500, which means it
>>> generates a 1997*1997 2d grid. It is fine to run it with single process
>>> with either "./ex29" or "mpiexec -np 1 ./exe29". However, it came across a
>>> error when I tempted to run it with more than 1 process. The error message
>>> is like this, when I was trying to execute it with 4 processes. I was
>>> thinking about if the 'stack size' is the problem, so I enlarge it to
>>> 1000000000 with 'ulimit' command. However, the same error message keeps
>>> showing up.
>>
>>
>> This custom viewer (in the example) is broken in parallel and has always
>> been broken in parallel. I don't know who wrote it, but it should be
>> removed since there is a binary VTK PetscViewer now.
>>
>> Matt, this has come up a few times before and I thought you defended that
>> viewer. Is there anything it does that VecView with PETSCVIEWERVTK does not?
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/fb4cfaa0/attachment.htm>

From jedbrown at mcs.anl.gov  Sat Apr  7 17:12:30 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 7 Apr 2012 17:12:30 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
	<CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
Message-ID: <CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>

On Sat, Apr 7, 2012 at 17:10, Alan Wei <zhenglun.wei at gmail.com> wrote:

> A little problem comes up. I commented out VTK part, the code still has
> the same error message popping up.


Well it can't be the same error message if that code isn't being executed.
Send the whole error message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/7afa1826/attachment-0001.htm>

From zhenglun.wei at gmail.com  Sat Apr  7 17:18:10 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Sat, 7 Apr 2012 17:18:10 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
	<CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
	<CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>
Message-ID: <CAD7ekpAP_qthVjjJNxyRr6Fb2BPc5r8jLWnAdbmUZ7oLFrk5Bw@mail.gmail.com>

Thanks Dr. Brown,
    It works. I made a wrong comment command so that the code did not go
through the compilation. So it gives the same error message since I ran the
old version of execution file.
    BTW, this ex29.c is a solver for 2D poisson equation. I wonder if there
is any 3D Poisson Equation solver in PETSc code?

thanks,
Alan

On Sat, Apr 7, 2012 at 5:12 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Sat, Apr 7, 2012 at 17:10, Alan Wei <zhenglun.wei at gmail.com> wrote:
>
>> A little problem comes up. I commented out VTK part, the code still has
>> the same error message popping up.
>
>
> Well it can't be the same error message if that code isn't being executed.
> Send the whole error message.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/7117d423/attachment.htm>

From jedbrown at mcs.anl.gov  Sat Apr  7 17:28:13 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 7 Apr 2012 17:28:13 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAD7ekpAP_qthVjjJNxyRr6Fb2BPc5r8jLWnAdbmUZ7oLFrk5Bw@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
	<CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
	<CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>
	<CAD7ekpAP_qthVjjJNxyRr6Fb2BPc5r8jLWnAdbmUZ7oLFrk5Bw@mail.gmail.com>
Message-ID: <CAM9tzSkBvVt2YVwRDTMv0JPWkB8mJGmdTAgv-DX+cVB2-E9L4A@mail.gmail.com>

On Sat, Apr 7, 2012 at 17:18, Alan Wei <zhenglun.wei at gmail.com> wrote:

> Thanks Dr. Brown,
>     It works. I made a wrong comment command so that the code did not go
> through the compilation. So it gives the same error message since I ran the
> old version of execution file.
>     BTW, this ex29.c is a solver for 2D poisson equation. I wonder if
> there is any 3D Poisson Equation solver in PETSc code?
>

http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex45.c.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120407/a7ec5342/attachment.htm>

From ywu at culink.Colorado.EDU  Sun Apr  8 18:59:14 2012
From: ywu at culink.Colorado.EDU (Yuqi Wu)
Date: Sun,  8 Apr 2012 17:59:14 -0600 (MDT)
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <495E393A-57DF-4434-959D-86F5183E6B32@mcs.anl.gov>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
	<20120405203143.APZ95110@batman.int.colorado.edu>
	<B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>
	<20120405210914.APZ95993@batman.int.colorado.edu> <495E393A-57DF-4
	434-959D-86F5183E6B32@mcs.anl.gov>
Message-ID: <20120408175914.AQB41382@batman.int.colorado.edu>

Thank you, Barry.

If I have two matrix A and B of type MPIBAIJ, how can I get the product of these two matrix C= AB? When I called the routine MatMatMult(), I got the following error

MatMatMult not supported for B of type mpibaij!



---- Original message ----
>Date: Thu, 5 Apr 2012 22:13:36 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG  
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 5, 2012, at 10:09 PM, Yuqi Wu wrote:
>
>> Thank you, Barry.
>> 
>> Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works. And I found out some interesting things, it also triggers the LU factorization for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2 SNES iteration.
>> 
>> Can I also ask another question? Because MatPtAP is not supported for baij matrices. I want to setup a routine to calculate coarse operator if the fine grid matrices is of type baij. Is there any approach so that I can keep the symbolic pattern and only redo the numeric part in each SNES, similar to MatPtAPSymbolic and MatPtAPNumeric?
>
>   You call the KSPOperators() with the coarse matrix in each Jacobian computation and just use the same coarse matrrix, just put new values into the nonzero locations. that is you do not need to create a new coarse matrix with matcreate each Jacobian just use the same as before. Call MatZeroEntries() before reusing.
>
>   Barry
>
>> 
>> Best
>> 
>> Yuqi
>> 
>> 
>> 
>> ---- Original message ----
>>> Date: Thu, 5 Apr 2012 21:35:58 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG  
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>> 
>>> 
>>> On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>>> 
>>>> Hi, Barry,
>>>> 
>>>> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
>>>> 
>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>> 
>>> Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>>> 
>>>  The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
>>> Barry
>>> 
>>> 
>>>> 
>>>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
>>>> 
>>>> Best
>>>> 
>>>> Yuqi
>>>> 
>>>> 
>>>> 
>>>> ---- Original message ----
>>>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>> 
>>>>> 
>>>>> Yes, at least that is the intention.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>>>> 
>>>>>> Hi, Barry,
>>>>>> 
>>>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>>>> 
>>>>>> Otherwise, I got the following error messages in PETSc
>>>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>>>> 
>>>>>> Can I use the following commands to setup the coarse operator
>>>>>> 
>>>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>>>> 
>>>>>> Thank you.
>>>>>> 
>>>>>> Best
>>>>>> 
>>>>>> Yuqi
>>>>>> 
>>>>>> 
>>>>>> ---- Original message ----
>>>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>> 
>>>>>>> 
>>>>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>>>>> 
>>>>>>> 
>>>>>>> Barry 
>>>>>>> 
>>>>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>>>>> 
>>>>>>> 
>>>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>>>> 
>>>>>>>> Thank you Barry.
>>>>>>>> 
>>>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>>>> 
>>>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>>>>> 
>>>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>>>>> 
>>>>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>>>>> /* so use those from global PC */
>>>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>>>> ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>>>> }
>>>>>>>> 
>>>>>>>> Thank you.
>>>>>>>> 
>>>>>>>> Yuqi
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>>>>>> 
>>>>>>>>> When it gets to the lines 
>>>>>>>>> 
>>>>>>>>> if (pc->ops->setup) {
>>>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>>>>> 
>>>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>>>>>> 
>>>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>>>>> 
>>>>>>>>> Barry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>>>> 
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>>>>> 
>>>>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>>>>> 
>>>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>>>> Setting up new PC
>>>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>>>> 
>>>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>>>> 
>>>>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>>>>>> 
>>>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>>>>> 
>>>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>>>>>> 
>>>>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>>>>> 
>>>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>>>>>> 
>>>>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>>>>> 
>>>>>>>>>> Best
>>>>>>>>>> 
>>>>>>>>>> Yuqi
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Barry
>>>>>>>>>>> 
>>>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Thanks, Adam.
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>>> 
>>>>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>>>>> 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>>> 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>>> 
>>>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best
>>>>>>>>>>>> 
>>>>>>>>>>>> Yuqi
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Mark
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Can I ask another question? 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>

From bsmith at mcs.anl.gov  Sun Apr  8 19:40:38 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 8 Apr 2012 19:40:38 -0500
Subject: [petsc-users] Questions about PCMG
In-Reply-To: <20120408175914.AQB41382@batman.int.colorado.edu>
References: <20120403132536.APX82674@batman.int.colorado.edu>
	<4B325F90-AD6F-4B17-92F4-65BEC908CB1D@columbia.edu>
	<20120403175628.APY01823@batman.int.colorado.edu>
	<99774891-1374-46C6-A661-3C32593EA25B@mcs.anl.gov>
	<20120403185732.APY04392@batman.int.colorado.edu>
	<215C87D8-2BD1-44C9-BC50-152A4840C6D0@mcs.anl.gov>
	<20120404115320.APY65194@batman.int.colorado.edu>
	<63828EB3-F4C3-4497-A099-5D1A95150442@columbia.edu>
	<20120404123452.APY68748@batman.int.colorado.edu>
	<64946653-C92E-4DC5-9C4C-27ED2345C374@mcs.anl.gov>
	<20120404184555.APY97983@batman.int.colorado.edu>
	<49C3BDC3-21EF-45E1-B167-82FDA37910A3@mcs.anl.gov>
	<20120404225543.APZ06409@batman.int.colorado.edu>
	<D18978C6-5FCF-4A85-B1A3-07F3025FDB81@mcs.anl.gov>
	<20120405130622.APZ65036@batman.int.colorado.edu>
	<C704F351-BCD4-4DA8-BE45-7DB17F053F82@mcs.anl.gov>
	<20120405203143.APZ95110@batman.int.colorado.edu>
	<B33A9DE8-CCCE-45F7-BFE4-E8BDE9506726@mcs.anl.gov>
	<20120405210914.APZ95993@batman.int.colorado.edu> <495E393A-57DF-4 
	434-959D-86F5183E6B32@mcs.anl.gov>
	<20120408175914.AQB41382@batman.int.colorado.edu>
Message-ID: <3B7C5F4B-ABCA-40A3-B800-300EA024EFA2@mcs.anl.gov>


  Unfortunately we only have those functions for AIJ matrices, so you can either use MatConvert to get AIJ matrices or create AIJ matrices at the start.

  Barry

On Apr 8, 2012, at 6:59 PM, Yuqi Wu wrote:

> Thank you, Barry.
> 
> If I have two matrix A and B of type MPIBAIJ, how can I get the product of these two matrix C= AB? When I called the routine MatMatMult(), I got the following error
> 
> MatMatMult not supported for B of type mpibaij!
> 
> 
> 
> ---- Original message ----
>> Date: Thu, 5 Apr 2012 22:13:36 -0500
>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>> Subject: Re: [petsc-users] Questions about PCMG  
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> 
>> 
>> On Apr 5, 2012, at 10:09 PM, Yuqi Wu wrote:
>> 
>>> Thank you, Barry.
>>> 
>>> Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works. And I found out some interesting things, it also triggers the LU factorization for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2 SNES iteration.
>>> 
>>> Can I also ask another question? Because MatPtAP is not supported for baij matrices. I want to setup a routine to calculate coarse operator if the fine grid matrices is of type baij. Is there any approach so that I can keep the symbolic pattern and only redo the numeric part in each SNES, similar to MatPtAPSymbolic and MatPtAPNumeric?
>> 
>>  You call the KSPOperators() with the coarse matrix in each Jacobian computation and just use the same coarse matrrix, just put new values into the nonzero locations. that is you do not need to create a new coarse matrix with matcreate each Jacobian just use the same as before. Call MatZeroEntries() before reusing.
>> 
>>  Barry
>> 
>>> 
>>> Best
>>> 
>>> Yuqi
>>> 
>>> 
>>> 
>>> ---- Original message ----
>>>> Date: Thu, 5 Apr 2012 21:35:58 -0500
>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>> 
>>>> 
>>>> On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>>>> 
>>>>> Hi, Barry,
>>>>> 
>>>>> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
>>>>> 
>>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>> 
>>>> Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>>>> 
>>>> The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
>>>> Barry
>>>> 
>>>> 
>>>>> 
>>>>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Yuqi
>>>>> 
>>>>> 
>>>>> 
>>>>> ---- Original message ----
>>>>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>> 
>>>>>> 
>>>>>> Yes, at least that is the intention.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>>>>> 
>>>>>>> Hi, Barry,
>>>>>>> 
>>>>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>>>>> 
>>>>>>> Otherwise, I got the following error messages in PETSc
>>>>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>>>>> 
>>>>>>> Can I use the following commands to setup the coarse operator
>>>>>>> 
>>>>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>>>>> 
>>>>>>> Thank you.
>>>>>>> 
>>>>>>> Best
>>>>>>> 
>>>>>>> Yuqi
>>>>>>> 
>>>>>>> 
>>>>>>> ---- Original message ----
>>>>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Barry 
>>>>>>>> 
>>>>>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>>>>> 
>>>>>>>>> Thank you Barry.
>>>>>>>>> 
>>>>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>>>>> 
>>>>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>>>>>> 
>>>>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>>>>>> 
>>>>>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>>>>>> /* so use those from global PC */
>>>>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n  because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>>>>> ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> Thank you.
>>>>>>>>> 
>>>>>>>>> Yuqi
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ---- Original message ----
>>>>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early. 
>>>>>>>>>> 
>>>>>>>>>> When it gets to the lines 
>>>>>>>>>> 
>>>>>>>>>> if (pc->ops->setup) {
>>>>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>>>>>> 
>>>>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. 
>>>>>>>>>> 
>>>>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>>>>>> 
>>>>>>>>>> Barry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>>>>> 
>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>>>>>> 
>>>>>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>>>>>> 
>>>>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>>>>> Setting up new PC
>>>>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>>>>> 
>>>>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>>>>> 
>>>>>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why. 
>>>>>>>>>>>> 
>>>>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>>>>>> 
>>>>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration? 
>>>>>>>>>>>> 
>>>>>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>>>>>> 
>>>>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration. 
>>>>>>>>>>> 
>>>>>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>>>>>> 
>>>>>>>>>>> Best
>>>>>>>>>>> 
>>>>>>>>>>> Yuqi
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Barry
>>>>>>>>>>>> 
>>>>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks, Adam.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use 
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here.  It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Are you using Galerkin coarse grids?  Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Mark
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Can I ask another question? 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times. 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> MatLUFactorSym         3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01  1  0  0  0  2   1  0  0  0  2     0
>>>>>>>>>>>>>>> MatLUFactorNum         3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97  0  0  0  32 97  0  0  0   280
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1.  9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> MGSetup Level 0        2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01  1  0  3 11  2   1  0  3 11  2     0
>>>>>>>>>>>>>>>>> MGSmooth Level 0       9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28  3 34  35 71 28  3 35   415
>>>>>>>>>>>>>>>>> MGSetup Level 1        2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01  0  0  1  3  1   0  0  1  3  1     0
>>>>>>>>>>>>>>>>> MGSmooth Level 1      18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15  3 19  17 28 15  3 19   339
>>>>>>>>>>>>>>>>> MGResid Level 1        9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00  0  0  2  0  0   0  0  2  0  0    70
>>>>>>>>>>>>>>>>> MGInterp Level 1      18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01  0  0  4  0  1   0  0  4  0  1    37
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG  
>>>>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp. 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 


From zhenglun.wei at gmail.com  Mon Apr  9 16:13:46 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Mon, 09 Apr 2012 16:13:46 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <CAM9tzSkfgnp9yZwPjRVdfDTbn9=SN7-e52YbbZRQF3oDqga83g@mail.gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
	<CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
	<CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>
	<CAD7ekpAP_qthVjjJNxyRr6Fb2BPc5r8jLWnAdbmUZ7oLFrk5Bw@mail.gmail.com>
	<CAM9tzSkBvVt2YVwRDTMv0JPWkB8mJGmdTAgv-DX+cVB2-E9L4A@mail.gmail.com>
	<CAM9tzSkfgnp9yZwPjRVdfDTbn9=SN7-e52YbbZRQF3oDqga83g@mail.gmail.com>
Message-ID: <4F83510A.7010008@gmail.com>

Thank you, Dr. Brown, for pointing me out this example. However, I met a 
problem when I was trying to modify it.
     I added a 'OutputFunction' subroutine to access the solution data 
by using DMDAGetVecArray. I tried it with single process, it keeps 
giving me this error:
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision: 
85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 
2011 -0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex45 on a arch-linu named l2118a-linux.soecs.ku.edu by 
zlwei Mon Apr  9 17:11:02 2012
[0]PETSC ERROR: Libraries linked from 
/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[0]PETSC ERROR: Configure options --download-f-blas-lapack=1 
--download-mpich=1 --with-cc=gcc --with-fc=gfortran 
PETSC_ARCH=arch-linux2-c-debug
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: DMDAVecGetArray() line 51 in src/dm/impls/da/dagetarray.c

      My code is attached.

thanks in advance,
Alan

On 4/7/2012 5:30 PM, Jed Brown wrote:
> On Sat, Apr 7, 2012 at 17:28, Jed Brown <jedbrown at mcs.anl.gov 
> <mailto:jedbrown at mcs.anl.gov>> wrote:
>
>     On Sat, Apr 7, 2012 at 17:18, Alan Wei <zhenglun.wei at gmail.com
>     <mailto:zhenglun.wei at gmail.com>> wrote:
>
>         Thanks Dr. Brown,
>             It works. I made a wrong comment command so that the code
>         did not go through the compilation. So it gives the same error
>         message since I ran the old version of execution file.
>             BTW, this ex29.c is a solver for 2D poisson equation. I
>         wonder if there is any 3D Poisson Equation solver in PETSc code?
>
>
>     http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex45.c.html
>
>
> I also see that your petsc-dev is from several months ago, you may 
> want to update because the interface has been simplified slightly (the 
> old thing still works, but this is somewhat simpler).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/ba444170/attachment.htm>
-------------- next part --------------

/*
Laplacian in 3D. Modeled by the partial differential equation

   - Laplacian u = 1,0 < x,y,z < 1,

with boundary conditions

   u = 1 for x = 0, x = 1, y = 0, y = 1, z = 0, z = 1.

   This uses multigrid to solve the linear system

   The same as ex22.c except it does not use DMMG, it uses its replacement.
   See src/snes/examples/tutorials/ex50.c

   Can also be run with -pc_type exotic -ksp_type fgmres

*/

static char help[] = "Solves 3D Laplacian using multigrid.\n\n";

#include <petscksp.h>
#include <petscdmda.h>

extern PetscErrorCode ComputeMatrix(DM,Vec,Mat,Mat,MatStructure*);
extern PetscErrorCode ComputeRHS(DM,Vec,Vec);
extern PetscErrorCode ComputeInitialGuess(DM,Vec);
extern PetscErrorCode OutputFunction(DM, Vec);
#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  PetscErrorCode ierr;
  KSP            ksp;
  PetscReal      norm;
  DM             da;
  Vec            x,b,r;
  Mat            A;

  PetscInitialize(&argc,&argv,(char *)0,help);

  ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
  ierr = DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,-7,-7,-7,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,0,0,0,&da);CHKERRQ(ierr);  
  ierr = DMSetInitialGuess(da,ComputeInitialGuess);CHKERRQ(ierr);
  ierr = DMSetFunction(da,ComputeRHS);CHKERRQ(ierr);
  ierr = DMSetJacobian(da,ComputeMatrix);CHKERRQ(ierr);
  ierr = KSPSetDM(ksp,da);CHKERRQ(ierr);
  /*  ierr = KSPSetDMActive(ksp,PETSC_FALSE);CHKERRQ(ierr);*/
  ierr = DMDestroy(&da);CHKERRQ(ierr);

  ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
  ierr = KSPSetUp(ksp);CHKERRQ(ierr);
  ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);
  ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr);
  ierr = KSPGetRhs(ksp,&b);CHKERRQ(ierr);
  ierr = VecDuplicate(b,&r);CHKERRQ(ierr);
  ierr = KSPGetOperators(ksp,&A,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);

  ierr = MatMult(A,x,r);CHKERRQ(ierr);
  ierr = VecAXPY(r,-1.0,b);CHKERRQ(ierr);
  ierr = VecNorm(r,NORM_2,&norm);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD,"Residual norm %G\n",norm);CHKERRQ(ierr); 
  ierr = OutputFunction(da,x);

  ierr = VecDestroy(&r);CHKERRQ(ierr);
  ierr = KSPDestroy(&ksp);CHKERRQ(ierr);
  ierr = PetscFinalize();

  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "ComputeRHS"
PetscErrorCode ComputeRHS(DM dm,Vec x,Vec b)
{
  PetscErrorCode ierr;
  PetscInt       mx,my,mz;
  PetscScalar    h;
  PetscScalar    ***xx,***bb;
  PetscInt       i,j,k,xm,ym,zm,xs,ys,zs;
  PetscScalar    Hx,Hy,Hz,HxHydHz,HyHzdHx,HxHzdHy;

  PetscFunctionBegin;
  ierr = DMDAGetInfo(dm,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);
  h    = 1.0/((mx-1)*(my-1)*(mz-1));
  ierr = VecSet(b,h);CHKERRQ(ierr);

  if (x) {
    ierr = DMDAVecGetArray(dm,x,&xx);CHKERRQ(ierr);
    ierr = DMDAVecGetArray(dm,b,&bb);CHKERRQ(ierr);

  ierr = DMDAGetInfo(dm,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);  
  Hx = 1.0 / (PetscReal)(mx-1); Hy = 1.0 / (PetscReal)(my-1); Hz = 1.0 / (PetscReal)(mz-1);
  HxHydHz = Hx*Hy/Hz; HxHzdHy = Hx*Hz/Hy; HyHzdHx = Hy*Hz/Hx;
  ierr = DMDAGetCorners(dm,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr);
  
  for (k=zs; k<zs+zm; k++){
    for (j=ys; j<ys+ym; j++){
      for(i=xs; i<xs+xm; i++){
	if (!(i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1)){
          bb[k][j][i] -= +HxHydHz*xx[k-1][j][i] + HxHzdHy*xx[k][j-1][i] + HyHzdHx*xx[k][j][i-1] + HyHzdHx*xx[k][j][i+1] + HxHzdHy*xx[k][j+1][i] + HxHydHz*xx[k+1][j][i];
        }
        bb[k][j][i] += 2.0*(HxHydHz + HxHzdHy + HyHzdHx)*xx[k][j][i];
      }
    }
  }
    ierr = DMDAVecRestoreArray(dm,x,&xx);CHKERRQ(ierr);
    ierr = DMDAVecRestoreArray(dm,b,&bb);CHKERRQ(ierr);
  }
  PetscFunctionReturn(0);
}
    
#undef __FUNCT__
#define __FUNCT__ "ComputeInitialGuess"
PetscErrorCode ComputeInitialGuess(DM dm,Vec b)
{
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr = VecSet(b,0);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

#undef __FUNCT__
#define __FUNCT__ "ComputeMatrix"
PetscErrorCode ComputeMatrix(DM dm,Vec x,Mat jac,Mat B,MatStructure *stflg)
{
  DM             da = dm;
  PetscErrorCode ierr;
  PetscInt       i,j,k,mx,my,mz,xm,ym,zm,xs,ys,zs;
  PetscScalar    v[7],Hx,Hy,Hz,HxHydHz,HyHzdHx,HxHzdHy;
  MatStencil     row,col[7];

  ierr = DMDAGetInfo(da,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);  
  Hx = 1.0 / (PetscReal)(mx-1); Hy = 1.0 / (PetscReal)(my-1); Hz = 1.0 / (PetscReal)(mz-1);
  HxHydHz = Hx*Hy/Hz; HxHzdHy = Hx*Hz/Hy; HyHzdHx = Hy*Hz/Hx;
  ierr = DMDAGetCorners(da,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr);
  
  for (k=zs; k<zs+zm; k++){
    for (j=ys; j<ys+ym; j++){
      for(i=xs; i<xs+xm; i++){
        row.i = i; row.j = j; row.k = k;
	if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1){
          v[0] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);
	  ierr = MatSetValuesStencil(B,1,&row,1,&row,v,INSERT_VALUES);CHKERRQ(ierr);
	} else {
	  v[0] = -HxHydHz;col[0].i = i; col[0].j = j; col[0].k = k-1;
	  v[1] = -HxHzdHy;col[1].i = i; col[1].j = j-1; col[1].k = k;
	  v[2] = -HyHzdHx;col[2].i = i-1; col[2].j = j; col[2].k = k;
	  v[3] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);col[3].i = row.i; col[3].j = row.j; col[3].k = row.k;
	  v[4] = -HyHzdHx;col[4].i = i+1; col[4].j = j; col[4].k = k;
	  v[5] = -HxHzdHy;col[5].i = i; col[5].j = j+1; col[5].k = k;
	  v[6] = -HxHydHz;col[6].i = i; col[6].j = j; col[6].k = k+1;
	  ierr = MatSetValuesStencil(B,1,&row,7,col,v,INSERT_VALUES);CHKERRQ(ierr);
        }
      }
    }
  }
  ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  *stflg = SAME_NONZERO_PATTERN;
  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "OutputFunction"
PetscErrorCode OutputFunction(DM da, Vec x) {
  PetscErrorCode ierr;
  PetscScalar ***array;

  PetscFunctionBegin;
  
  ierr = DMDAVecGetArray(da, x, &array);

  PetscFunctionReturn(0);
}

From mirzadeh at gmail.com  Mon Apr  9 17:06:10 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 15:06:10 -0700
Subject: [petsc-users] Global numbering obtained via
 ISPartitioningToNumbering is inconsistent with the partitioning indexset
Message-ID: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>

Hi,

I am using  'ISPartitioningToNumbering' to generate new global numbering
from a partitioning indexset and I'm baffled at the following situation.
I'm debugging my code on a simple grid consisting of 81 grid points
partitioned among two processes. When I look into the partitioning indexset
(i.e. looking at the indecies via ISView) I can see that 40 points have
been assigned to proc 0 and 41 to processor 1. Isn't it true that when 81
points are distributed among two processors 41 should go to proc 0 and 40
to proc 1?

I have based my whole code on the assumption (verified before through
mailing list i guess) that natural ordering in PETSc leads to a
distribution of points such that all processors get the same number of
points ( [0, n/p) on proc 0, [n/p, 2n/p) on proc 1, ... ) unless n%p != 0,
in which case the first k (with k = n%p) processors receive 1 point extra.
Am I wrong to assume this?

Thanks,
Mohammad

PS: Is it relevant that the partitioning indexset is obtained via
ParMetis?partitiong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/1e5928c7/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr  9 17:10:51 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Apr 2012 17:10:51 -0500
Subject: [petsc-users] Global numbering obtained via
	ISPartitioningToNumbering is inconsistent with the
	partitioning indexset
In-Reply-To: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>
References: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>
Message-ID: <A64BD599-4872-4A7F-AF34-94FDCD2BDF7C@mcs.anl.gov>


On Apr 9, 2012, at 5:06 PM, Mohammad Mirzadeh wrote:

> Hi,
> 
> I am using  'ISPartitioningToNumbering' to generate new global numbering from a partitioning indexset and I'm baffled at the following situation. I'm debugging my code on a simple grid consisting of 81 grid points partitioned among two processes. When I look into the partitioning indexset (i.e. looking at the indecies via ISView) I can see that 40 points have been assigned to proc 0 and 41 to processor 1. Isn't it true that when 81 points are distributed among two processors 41 should go to proc 0 and 40 to proc 1? 
> 
> I have based my whole code on the assumption (verified before through mailing list i guess) that natural ordering in PETSc leads to a distribution of points such that all processors get the same number of points ( [0, n/p) on proc 0, [n/p, 2n/p) on proc 1, ... ) unless n%p != 0, in which case the first k (with k = n%p) processors receive 1 point extra. Am I wrong to assume this?
> 
> Thanks,
> Mohammad
> 

> PS: Is it relevant that the partitioning indexset is obtained via ParMetis?partitiong

   Yes, ParMetis provides no guarantee about how many points would get assigned to each process.

   Barry



From mirzadeh at gmail.com  Mon Apr  9 17:25:17 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 15:25:17 -0700
Subject: [petsc-users] Global numbering obtained via
 ISPartitioningToNumbering is inconsistent with the partitioning indexset
In-Reply-To: <A64BD599-4872-4A7F-AF34-94FDCD2BDF7C@mcs.anl.gov>
References: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>
	<A64BD599-4872-4A7F-AF34-94FDCD2BDF7C@mcs.anl.gov>
Message-ID: <CAKsPKwkEXZF0QsLJ9mcnThqAbdhoDjHACV3A2vMWB3GoY_TyUQ@mail.gmail.com>

Aaah! Thanks Barry. Just to make sure though, is my assumption on the
natural ordering of PETSc correct?

Thanks

On Mon, Apr 9, 2012 at 3:10 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Apr 9, 2012, at 5:06 PM, Mohammad Mirzadeh wrote:
>
> > Hi,
> >
> > I am using  'ISPartitioningToNumbering' to generate new global numbering
> from a partitioning indexset and I'm baffled at the following situation.
> I'm debugging my code on a simple grid consisting of 81 grid points
> partitioned among two processes. When I look into the partitioning indexset
> (i.e. looking at the indecies via ISView) I can see that 40 points have
> been assigned to proc 0 and 41 to processor 1. Isn't it true that when 81
> points are distributed among two processors 41 should go to proc 0 and 40
> to proc 1?
> >
> > I have based my whole code on the assumption (verified before through
> mailing list i guess) that natural ordering in PETSc leads to a
> distribution of points such that all processors get the same number of
> points ( [0, n/p) on proc 0, [n/p, 2n/p) on proc 1, ... ) unless n%p != 0,
> in which case the first k (with k = n%p) processors receive 1 point extra.
> Am I wrong to assume this?
> >
> > Thanks,
> > Mohammad
> >
>
> > PS: Is it relevant that the partitioning indexset is obtained via
> ParMetis?partitiong
>
>    Yes, ParMetis provides no guarantee about how many points would get
> assigned to each process.
>
>   Barry
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/a0fecbf8/attachment.htm>

From zhenglun.wei at gmail.com  Mon Apr  9 17:37:45 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Mon, 09 Apr 2012 17:37:45 -0500
Subject: [petsc-users] How to access solution in
	/src/ksp/ksp/example/tutorial/ex45.c
Message-ID: <4F8364B9.1010601@gmail.com>

Dear All,
     I hope you're having a nice day.
     I'm working on ex45. It does not have any output subroutine inside. 
Therefore, I was trying to write out mine.
     I was trying to use DMDAVecGetArray; but I found that the DM object 
has been destroyed at very beginning of the main code. Therefore, I 
swithed to VecGetArray. However, it gives me warning like:
ex45.c:185: warning: passing argument 2 of 'VecGetArray' from 
incompatible pointer type
     Then, it shows a memory violation error during execution, which is 
attached as 'out'. I checked 'VecGetArray' examples in PETSc manual, it 
always shows how to use it to extract one-dimensional array. I wonder if 
it can do 3-dimensional?

     Also, there are other two minor questions:
1) in ex45.c, 'ComputeRHS', there is a if(x) {} statement, I write probe 
statements inside this if statement to see if the code ever go through 
it; the result shows it does not. I don't know why. ;(
2) probe statements are added in 'ComputeRHS' for both ex45.c and 
ex29.c. The size parameter used for DMDACreate3d in ex45 is -7, -7, -7; 
while for DMDACreate2d in ex29 is -7,-7. The probe statements show that 
the real 2d domain generated in ex29 is 25*25, which, as I understand, 
is because of multigrid. However, with multigrid scheme, the ex45 create 
a 7*7*7 3d domain. Any suggestions on this?

     I attached both ex45 and ex29 with this E-mail.

thanks in advance,
Alan

-------------- next part --------------

/*
Laplacian in 3D. Modeled by the partial differential equation

   - Laplacian u = 1,0 < x,y,z < 1,

with boundary conditions

   u = 1 for x = 0, x = 1, y = 0, y = 1, z = 0, z = 1.

   This uses multigrid to solve the linear system

   The same as ex22.c except it does not use DMMG, it uses its replacement.
   See src/snes/examples/tutorials/ex50.c

   Can also be run with -pc_type exotic -ksp_type fgmres

*/

static char help[] = "Solves 3D Laplacian using multigrid.\n\n";

#include <petscksp.h>
#include <petscdmda.h>
PetscInt i,j,k,mx,my,mz,xm,ym,zm,xs,ys,zs;

extern PetscErrorCode ComputeMatrix(DM,Vec,Mat,Mat,MatStructure*);
extern PetscErrorCode ComputeRHS(DM,Vec,Vec);
extern PetscErrorCode ComputeInitialGuess(DM, Vec);
extern PetscErrorCode OutputFunction(Vec);
#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  PetscErrorCode ierr;
  KSP            ksp;
  PetscReal      norm;
  DM             da;
  Vec            x,b,r;
  Mat            A;

  PetscInitialize(&argc,&argv,(char *)0,help);

  ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
  ierr = DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,-7,-7,-7,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,0,0,0,&da);CHKERRQ(ierr);  
  ierr = DMSetInitialGuess(da,ComputeInitialGuess);CHKERRQ(ierr);
  ierr = DMSetFunction(da,ComputeRHS);CHKERRQ(ierr);
  ierr = DMSetJacobian(da,ComputeMatrix);CHKERRQ(ierr);
  ierr = KSPSetDM(ksp,da);CHKERRQ(ierr);
  /*  ierr = KSPSetDMActive(ksp,PETSC_FALSE);CHKERRQ(ierr);*/
  ierr = DMDestroy(&da);CHKERRQ(ierr);

  ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
  ierr = KSPSetUp(ksp);CHKERRQ(ierr);
  ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);
  ierr = KSPGetSolution(ksp,&x);CHKERRQ(ierr);
  ierr = KSPGetRhs(ksp,&b);CHKERRQ(ierr);
  ierr = VecDuplicate(b,&r);CHKERRQ(ierr);
  ierr = KSPGetOperators(ksp,&A,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr);

  ierr = MatMult(A,x,r);CHKERRQ(ierr);
  ierr = VecAXPY(r,-1.0,b);CHKERRQ(ierr);
  ierr = VecNorm(r,NORM_2,&norm);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD,"Residual norm %G\n",norm);CHKERRQ(ierr); 
  ierr = OutputFunction(x);

  ierr = VecDestroy(&r);CHKERRQ(ierr);
  ierr = KSPDestroy(&ksp);CHKERRQ(ierr);
  ierr = PetscFinalize();

  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "ComputeRHS"
PetscErrorCode ComputeRHS(DM dm,Vec x,Vec b)
{
  PetscErrorCode ierr;
  PetscInt       mx,my,mz;
  PetscScalar    h;
  PetscScalar    ***xx,***bb;
  PetscScalar    Hx,Hy,Hz,HxHydHz,HyHzdHx,HxHzdHy;

  PetscFunctionBegin;
  ierr = DMDAGetInfo(dm,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);
  printf("mx = %d, my = %d, mz = %d\n", mx, my, mz);
  printf("xs = %d, ys = %d, zs = %d\n", xs, ys, zs);
  printf("xm = %d, ym = %d, zm = %d\n", xm, ym, zm);

  h    = 1.0/((mx-1)*(my-1)*(mz-1));
  ierr = VecSet(b,h);CHKERRQ(ierr);

  if (x) {
    ierr = DMDAVecGetArray(dm,x,&xx);CHKERRQ(ierr);
    ierr = DMDAVecGetArray(dm,b,&bb);CHKERRQ(ierr);

  ierr = DMDAGetInfo(dm,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr); 
  printf("mx = %d, my = %d, mz = %d\n", mx, my, mz);
  printf("xs = %d, ys = %d, zs = %d\n", xs, ys, zs);
  printf("xm = %d, ym = %d, zm = %d\n", xm, ym, zm);
 
  Hx = 1.0 / (PetscReal)(mx-1); Hy = 1.0 / (PetscReal)(my-1); Hz = 1.0 / (PetscReal)(mz-1);
  HxHydHz = Hx*Hy/Hz; HxHzdHy = Hx*Hz/Hy; HyHzdHx = Hy*Hz/Hx;
  ierr = DMDAGetCorners(dm,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr);
  
  for (k=zs; k<zs+zm; k++){
    for (j=ys; j<ys+ym; j++){
      for(i=xs; i<xs+xm; i++){
	if (!(i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1)){
          bb[k][j][i] -= +HxHydHz*xx[k-1][j][i] + HxHzdHy*xx[k][j-1][i] + HyHzdHx*xx[k][j][i-1] + HyHzdHx*xx[k][j][i+1] + HxHzdHy*xx[k][j+1][i] + HxHydHz*xx[k+1][j][i];
        }
        bb[k][j][i] += 2.0*(HxHydHz + HxHzdHy + HyHzdHx)*xx[k][j][i];
      }
    }
  }
    ierr = DMDAVecRestoreArray(dm,x,&xx);CHKERRQ(ierr);
    ierr = DMDAVecRestoreArray(dm,b,&bb);CHKERRQ(ierr);
  }
  PetscFunctionReturn(0);
}
    
#undef __FUNCT__
#define __FUNCT__ "ComputeInitialGuess"
PetscErrorCode ComputeInitialGuess(DM dm,Vec b)
{
  PetscErrorCode ierr;

  PetscFunctionBegin;
  ierr = VecSet(b,0);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

#undef __FUNCT__
#define __FUNCT__ "ComputeMatrix"
PetscErrorCode ComputeMatrix(DM dm,Vec x,Mat jac,Mat B,MatStructure *stflg)
{
  DM             da = dm;
  PetscErrorCode ierr;
  PetscScalar    v[7],Hx,Hy,Hz,HxHydHz,HyHzdHx,HxHzdHy;
  MatStencil     row,col[7];

  ierr = DMDAGetInfo(da,0,&mx,&my,&mz,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);  
  Hx = 1.0 / (PetscReal)(mx-1); Hy = 1.0 / (PetscReal)(my-1); Hz = 1.0 / (PetscReal)(mz-1);
  HxHydHz = Hx*Hy/Hz; HxHzdHy = Hx*Hz/Hy; HyHzdHx = Hy*Hz/Hx;
  ierr = DMDAGetCorners(da,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr);
  
  for (k=zs; k<zs+zm; k++){
    for (j=ys; j<ys+ym; j++){
      for(i=xs; i<xs+xm; i++){
        row.i = i; row.j = j; row.k = k;
	if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1){
          v[0] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);
	  ierr = MatSetValuesStencil(B,1,&row,1,&row,v,INSERT_VALUES);CHKERRQ(ierr);
	} else {
	  v[0] = -HxHydHz;col[0].i = i; col[0].j = j; col[0].k = k-1;
	  v[1] = -HxHzdHy;col[1].i = i; col[1].j = j-1; col[1].k = k;
	  v[2] = -HyHzdHx;col[2].i = i-1; col[2].j = j; col[2].k = k;
	  v[3] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);col[3].i = row.i; col[3].j = row.j; col[3].k = row.k;
	  v[4] = -HyHzdHx;col[4].i = i+1; col[4].j = j; col[4].k = k;
	  v[5] = -HxHzdHy;col[5].i = i; col[5].j = j+1; col[5].k = k;
	  v[6] = -HxHydHz;col[6].i = i; col[6].j = j; col[6].k = k+1;
	  ierr = MatSetValuesStencil(B,1,&row,7,col,v,INSERT_VALUES);CHKERRQ(ierr);
        }
      }
    }
  }
  ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  *stflg = SAME_NONZERO_PATTERN;
  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "OutputFunction"
PetscErrorCode OutputFunction(Vec x) 
{
  PetscErrorCode ierr;
  PetscScalar ***SolArray;
  int rank, countt, countn;
  char fname[20];
  FILE *fgrid, *fpplt3d;

  PetscFunctionBegin;

  if(x) { 
    ierr = VecGetArray(x, &SolArray);CHKERRQ(ierr);
  }

  printf("mx = %d, my = %d, mz = %d\n", mx, my, mz);
  printf("xs = %d, ys = %d, zs = %d\n", xs, ys, zs);
  printf("xm = %d, ym = %d, zm = %d\n", xm, ym, zm);

  MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
  sprintf(fname, "%s%d%s", "Rank", rank,"plt3d.q");
  fpplt3d = fopen(fname, "w");

  fprintf(fpplt3d, "%d  %d  %d\n", mx, my, mz);
  fprintf(fpplt3d, "%f  %f  %f  %f  %f  \n", 1.0, 1.0, 1.0, 1.0, 1.0);

  for(countn = 1; countn <= 4; countn++) {
    for(k = zs; k < zs+zm; k++) {
      for(j = ys; j < ys+ym; j++) {
	for(i = xs; i < xs+xm; i++) {
	  if(countn == 1) fprintf(fpplt3d, "%f  ", 1.0);
	  if(countn == 2) fprintf(fpplt3d, "%f  ", SolArray[k][j][i]);
	  if(countn == 3) fprintf(fpplt3d, "%f  ", SolArray[k][j][i]);
	  if(countn == 4) fprintf(fpplt3d, "%f  ", SolArray[k][j][i]);
	  if(countn == 5) fprintf(fpplt3d, "%f  ", SolArray[k][j][i]);
	  if(!(countt%6)) fprintf(fpplt3d, "\n");
	  countt++;
	}
      }
    }
  }
  fprintf(fpplt3d, "\n");
  fclose(fpplt3d);


  PetscFunctionReturn(0);
}
-------------- next part --------------
mx = 7, my = 7, mz = 7
xs = 0, ys = 0, zs = 0
xm = 0, ym = 0, zm = 0
Residual norm 7.06789e-07
mx = 7, my = 7, mz = 7
xs = 0, ys = 0, zs = 0
xm = 7, ym = 7, zm = 7
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] OutputFunction line 182 src/ksp/ksp/examples/tutorials/ex45.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision: 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011 -0700
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./ex45 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Mon Apr  9 18:19:47 2012
[0]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
[0]PETSC ERROR: Configure options --download-f-blas-lapack=1 --download-mpich=1 --with-cc=gcc --with-fc=gfortran PETSC_ARCH=arch-linux2-c-debug
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
-------------- next part --------------
/*T
   Concepts: KSP^solving a system of linear equations
   Concepts: KSP^Laplacian, 2d
   Processors: n
T*/

/*
Added at the request of Marc Garbey.

Inhomogeneous Laplacian in 2D. Modeled by the partial differential equation

   -div \rho grad u = f,  0 < x,y < 1,

with forcing function

   f = e^{-x^2/\nu} e^{-y^2/\nu}

with Dirichlet boundary conditions

   u = f(x,y) for x = 0, x = 1, y = 0, y = 1

or pure Neumman boundary conditions

This uses multigrid to solve the linear system
*/

static char help[] = "Solves 2D inhomogeneous Laplacian using multigrid.\n\n";

#include <petscdmda.h>
#include <petscksp.h>
#include <petscpcmg.h>
#include <petscdmmg.h>

extern PetscErrorCode ComputeMatrix(DMMG,Mat,Mat);
extern PetscErrorCode ComputeRHS(DMMG,Vec);
extern PetscErrorCode VecView_VTK(Vec, const char [], const char []);

typedef enum {DIRICHLET, NEUMANN} BCType;

typedef struct {
  PetscScalar   rho;
  PetscScalar   nu;
  BCType        bcType;
} UserContext;

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  DMMG           *dmmg;
  DM             da;
  UserContext    user;
  PetscReal      norm;
  const char     *bcTypes[2] = {"dirichlet","neumann"};
  PetscErrorCode ierr;
  PetscInt       l,bc;

  PetscInitialize(&argc,&argv,(char *)0,help);

  ierr = DMMGCreate(PETSC_COMM_WORLD,3,PETSC_NULL,&dmmg);CHKERRQ(ierr);
  ierr = DMDACreate2d(PETSC_COMM_WORLD, DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,-7,-7,PETSC_DECIDE,PETSC_DECIDE,1,1,0,0,&da);CHKERRQ(ierr);  
  ierr = DMMGSetDM(dmmg,(DM)da);CHKERRQ(ierr);
  ierr = DMDestroy(&da);CHKERRQ(ierr);
  for (l = 0; l < DMMGGetLevels(dmmg); l++) {
    ierr = DMMGSetUser(dmmg,l,&user);CHKERRQ(ierr);
  }

  ierr = PetscOptionsBegin(PETSC_COMM_WORLD, "", "Options for the inhomogeneous Poisson equation", "DMMG");
    user.rho    = 1.0;
    ierr        = PetscOptionsScalar("-rho", "The conductivity", "ex29.c", user.rho, &user.rho, PETSC_NULL);CHKERRQ(ierr);
    user.nu     = 0.1;
    ierr        = PetscOptionsScalar("-nu", "The width of the Gaussian source", "ex29.c", user.nu, &user.nu, PETSC_NULL);CHKERRQ(ierr);
    bc          = (PetscInt)DIRICHLET;
    ierr        = PetscOptionsEList("-bc_type","Type of boundary condition","ex29.c",bcTypes,2,bcTypes[0],&bc,PETSC_NULL);CHKERRQ(ierr);
    user.bcType = (BCType)bc;
  ierr = PetscOptionsEnd();

  ierr = DMMGSetKSP(dmmg,ComputeRHS,ComputeMatrix);CHKERRQ(ierr);
  if (user.bcType == NEUMANN) {
    ierr = DMMGSetNullSpace(dmmg,PETSC_TRUE,0,PETSC_NULL);CHKERRQ(ierr);
  }

  ierr = DMMGSolve(dmmg);CHKERRQ(ierr);

  ierr = MatMult(DMMGGetJ(dmmg),DMMGGetx(dmmg),DMMGGetr(dmmg));CHKERRQ(ierr);
  ierr = VecAXPY(DMMGGetr(dmmg),-1.0,DMMGGetRHS(dmmg));CHKERRQ(ierr);
  ierr = VecNorm(DMMGGetr(dmmg),NORM_2,&norm);CHKERRQ(ierr);
  /* ierr = PetscPrintf(PETSC_COMM_WORLD,"Residual norm %G\n",norm);CHKERRQ(ierr); */
  ierr = VecAssemblyBegin(DMMGGetx(dmmg));CHKERRQ(ierr);
  ierr = VecAssemblyEnd(DMMGGetx(dmmg));CHKERRQ(ierr);
//  ierr = VecView_VTK(DMMGGetRHS(dmmg), "rhs.vtk", bcTypes[user.bcType]);CHKERRQ(ierr);
//  ierr = VecView_VTK(DMMGGetx(dmmg), "solution.vtk", bcTypes[user.bcType]);CHKERRQ(ierr);

  ierr = DMMGDestroy(dmmg);CHKERRQ(ierr);
  ierr = PetscFinalize();

  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "ComputeRHS"
PetscErrorCode ComputeRHS(DMMG dmmg, Vec b)
{
  DM             da = dmmg->dm;
  UserContext    *user = (UserContext *) dmmg->user;
  PetscErrorCode ierr;
  PetscInt       i,j,mx,my,xm,ym,xs,ys;
  PetscScalar    Hx,Hy;
  PetscScalar    **array;

  PetscFunctionBegin;
  ierr = DMDAGetInfo(da, 0, &mx, &my, 0,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);
  Hx   = 1.0 / (PetscReal)(mx-1);
  Hy   = 1.0 / (PetscReal)(my-1);
  ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0);CHKERRQ(ierr);
  printf("%d  %d  %d  %d  %d  %d  \n", mx, my, xs, ys, xm, ym);
  ierr = DMDAVecGetArray(da, b, &array);CHKERRQ(ierr);
  for (j=ys; j<ys+ym; j++){
    for(i=xs; i<xs+xm; i++){
      array[j][i] = PetscExpScalar(-((PetscReal)i*Hx)*((PetscReal)i*Hx)/user->nu)*PetscExpScalar(-((PetscReal)j*Hy)*((PetscReal)j*Hy)/user->nu)*Hx*Hy;
    }
  }
  ierr = DMDAVecRestoreArray(da, b, &array);CHKERRQ(ierr);
  ierr = VecAssemblyBegin(b);CHKERRQ(ierr);
  ierr = VecAssemblyEnd(b);CHKERRQ(ierr);

  /* force right hand side to be consistent for singular matrix */
  /* note this is really a hack, normally the model would provide you with a consistent right handside */
  if (user->bcType == NEUMANN) {
    MatNullSpace nullspace;

    ierr = KSPGetNullSpace(dmmg->ksp,&nullspace);CHKERRQ(ierr);
    ierr = MatNullSpaceRemove(nullspace,b,PETSC_NULL);CHKERRQ(ierr);
  }
  PetscFunctionReturn(0);
}

    
#undef __FUNCT__
#define __FUNCT__ "ComputeRho"
PetscErrorCode ComputeRho(PetscInt i, PetscInt j, PetscInt mx, PetscInt my, PetscScalar centerRho, PetscScalar *rho)
{
  PetscFunctionBegin;
  if ((i > mx/3.0) && (i < 2.0*mx/3.0) && (j > my/3.0) && (j < 2.0*my/3.0)) {
    *rho = centerRho;
  } else {
    *rho = 1.0;
  }
  PetscFunctionReturn(0);
}

#undef __FUNCT__
#define __FUNCT__ "ComputeMatrix"
PetscErrorCode ComputeMatrix(DMMG dmmg, Mat J,Mat jac)
{
  DM             da =  dmmg->dm;
  UserContext    *user = (UserContext *) dmmg->user;
  PetscScalar    centerRho = user->rho;
  PetscErrorCode ierr;
  PetscInt       i,j,mx,my,xm,ym,xs,ys,num;
  PetscScalar    v[5],Hx,Hy,HydHx,HxdHy,rho;
  MatStencil     row, col[5];

  PetscFunctionBegin;
  ierr = DMDAGetInfo(da,0,&mx,&my,0,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);  
  Hx    = 1.0 / (PetscReal)(mx-1);
  Hy    = 1.0 / (PetscReal)(my-1);
  HxdHy = Hx/Hy;
  HydHx = Hy/Hx;
  ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0);CHKERRQ(ierr);
  for (j=ys; j<ys+ym; j++){
    for(i=xs; i<xs+xm; i++){
      row.i = i; row.j = j;
      ierr = ComputeRho(i, j, mx, my, centerRho, &rho);CHKERRQ(ierr);
      if (i==0 || j==0 || i==mx-1 || j==my-1) {
        if (user->bcType == DIRICHLET) {
           v[0] = 2.0*rho*(HxdHy + HydHx);
          ierr = MatSetValuesStencil(jac,1,&row,1,&row,v,INSERT_VALUES);CHKERRQ(ierr);
        } else if (user->bcType == NEUMANN) {
          num = 0;
          if (j!=0) {
            v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j-1;
            num++;
          }
          if (i!=0) {
            v[num] = -rho*HydHx;              col[num].i = i-1; col[num].j = j;
            num++;
          }
          if (i!=mx-1) {
            v[num] = -rho*HydHx;              col[num].i = i+1; col[num].j = j;
            num++;
          }
          if (j!=my-1) {
            v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j+1;
            num++;
          }
          v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;   col[num].j = j;
          num++;
          ierr = MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
        }
      } else {
        v[0] = -rho*HxdHy;              col[0].i = i;   col[0].j = j-1;
        v[1] = -rho*HydHx;              col[1].i = i-1; col[1].j = j;
        v[2] = 2.0*rho*(HxdHy + HydHx); col[2].i = i;   col[2].j = j;
        v[3] = -rho*HydHx;              col[3].i = i+1; col[3].j = j;
        v[4] = -rho*HxdHy;              col[4].i = i;   col[4].j = j+1;
        ierr = MatSetValuesStencil(jac,1,&row,5,col,v,INSERT_VALUES);CHKERRQ(ierr);
      }
    }
  }
  ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}
/*
#undef __FUNCT__
#define __FUNCT__ "VecView_VTK"
PetscErrorCode VecView_VTK(Vec x, const char filename[], const char bcName[])
{
  MPI_Comm           comm;
  DM                 da;
  Vec                coords;
  PetscViewer        viewer;
  PetscScalar       *array, *values;
  PetscInt           n, N, maxn, mx, my, dof;
  PetscInt           i, p;
  MPI_Status         status;
  PetscMPIInt        rank, size, tag, nn;
  PetscErrorCode     ierr;
  PetscBool          flg;

  PetscFunctionBegin;
  ierr = PetscObjectGetComm((PetscObject) x, &comm);CHKERRQ(ierr);
  ierr = PetscViewerASCIIOpen(comm, filename, &viewer);CHKERRQ(ierr);

  ierr = VecGetSize(x, &N);CHKERRQ(ierr);
  ierr = VecGetLocalSize(x, &n);CHKERRQ(ierr);
  ierr = PetscObjectQuery((PetscObject) x, "DM", (PetscObject *) &da);CHKERRQ(ierr);
  if (!da) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_ARG_WRONG,"Vector not generated from a DMDA");
  ierr = PetscTypeCompare((PetscObject)da,DMDA,&flg);CHKERRQ(ierr);
  if (!flg) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_ARG_WRONG,"Vector not generated from a DMDA");

  ierr = DMDAGetInfo(da, 0, &mx, &my, 0,0,0,0, &dof,0,0,0,0,0);CHKERRQ(ierr);

  ierr = PetscViewerASCIIPrintf(viewer, "# vtk DataFile Version 2.0\n");CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Inhomogeneous Poisson Equation with %s boundary conditions\n", bcName);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "ASCII\n");CHKERRQ(ierr);*/
  /* get coordinates of nodes */
/*  ierr = DMDAGetCoordinates(da, &coords);CHKERRQ(ierr);
  if (!coords) {
    ierr = DMDASetUniformCoordinates(da, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0);CHKERRQ(ierr);
    ierr = DMDAGetCoordinates(da, &coords);CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "DATASET RECTILINEAR_GRID\n");CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "DIMENSIONS %d %d %d\n", mx, my, 1);CHKERRQ(ierr);
  ierr = VecGetArray(coords, &array);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "X_COORDINATES %d double\n", mx);CHKERRQ(ierr);
  for(i = 0; i < mx; i++) {
    ierr = PetscViewerASCIIPrintf(viewer, "%G ", PetscRealPart(array[i*2]));CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "\n");CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Y_COORDINATES %d double\n", my);CHKERRQ(ierr);
  for(i = 0; i < my; i++) {
    ierr = PetscViewerASCIIPrintf(viewer, "%G ", PetscRealPart(array[i*mx*2+1]));CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "\n");CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Z_COORDINATES %d double\n", 1);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "%G\n", 0.0);CHKERRQ(ierr);
  ierr = VecRestoreArray(coords, &array);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "POINT_DATA %d\n", N);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "SCALARS scalars double %d\n", dof);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "LOOKUP_TABLE default\n");CHKERRQ(ierr);
  ierr = VecGetArray(x, &array);CHKERRQ(ierr);
  /* Determine maximum message to arrive */
/*  ierr = MPI_Comm_rank(comm, &rank);CHKERRQ(ierr);
  ierr = MPI_Comm_size(comm, &size);CHKERRQ(ierr);
  ierr = MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm);CHKERRQ(ierr);
  tag  = ((PetscObject) viewer)->tag;
  if (!rank) {
    ierr = PetscMalloc((maxn+1) * sizeof(PetscScalar), &values);CHKERRQ(ierr);
    for(i = 0; i < n; i++) {
      ierr = PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i]));CHKERRQ(ierr);
    }
    for(p = 1; p < size; p++) {
      ierr = MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, tag, comm, &status);CHKERRQ(ierr);
      ierr = MPI_Get_count(&status, MPIU_SCALAR, &nn);CHKERRQ(ierr);        
      for(i = 0; i < nn; i++) {
        ierr = PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i]));CHKERRQ(ierr);
      }
    }
    ierr = PetscFree(values);CHKERRQ(ierr);
  } else {
    ierr = MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm);CHKERRQ(ierr);
  }
  ierr = VecRestoreArray(x, &array);CHKERRQ(ierr);
  ierr = PetscViewerFlush(viewer);CHKERRQ(ierr);
  ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}*/

From jedbrown at mcs.anl.gov  Mon Apr  9 19:09:52 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 19:09:52 -0500
Subject: [petsc-users] Global numbering obtained via
 ISPartitioningToNumbering is inconsistent with the partitioning indexset
In-Reply-To: <CAKsPKwkEXZF0QsLJ9mcnThqAbdhoDjHACV3A2vMWB3GoY_TyUQ@mail.gmail.com>
References: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>
	<A64BD599-4872-4A7F-AF34-94FDCD2BDF7C@mcs.anl.gov>
	<CAKsPKwkEXZF0QsLJ9mcnThqAbdhoDjHACV3A2vMWB3GoY_TyUQ@mail.gmail.com>
Message-ID: <CAM9tzSmjjLJhFXerqR0z9AjNrjEPF9CPiH76ZSepqAvA_BQsNg@mail.gmail.com>

On Mon, Apr 9, 2012 at 17:25, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Just to make sure though, is my assumption on the natural ordering of
> PETSc correct?


Yes, but please don't write code that depends on that. It's plenty easy to
query the local size or the sizes/starts of other processes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/0ad28fbe/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  9 19:16:50 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 19:16:50 -0500
Subject: [petsc-users] Problem with "likely location of problem given in
 stack below"?
In-Reply-To: <4F83510A.7010008@gmail.com>
References: <CAD7ekpBOhVFiBO3eG+y7k6_gAPcX4CY-RHhq6QWyJz9icYFuKw@mail.gmail.com>
	<CAM9tzSmQyKqWAwMUi5OgYhDBEd_v8gPqv6edZDpTnemGJ+0y-w@mail.gmail.com>
	<CAD7ekpBrbqBtF8ASroqAeUuNsLHA_otanJzVYmwhB8z_weUmgg@mail.gmail.com>
	<CAD7ekpAZ0Af2aAbgYSGm9-J_PqGq88LwpAYX3xeWRyQ2nZYa0A@mail.gmail.com>
	<CAM9tzS=pnEtFOfcz6Xd_A+6bT15-obTRazDDnzbKy8cZeOuufA@mail.gmail.com>
	<CAD7ekpAP_qthVjjJNxyRr6Fb2BPc5r8jLWnAdbmUZ7oLFrk5Bw@mail.gmail.com>
	<CAM9tzSkBvVt2YVwRDTMv0JPWkB8mJGmdTAgv-DX+cVB2-E9L4A@mail.gmail.com>
	<CAM9tzSkfgnp9yZwPjRVdfDTbn9=SN7-e52YbbZRQF3oDqga83g@mail.gmail.com>
	<4F83510A.7010008@gmail.com>
Message-ID: <CAM9tzS=djUdEGRxSGHtU6kWEAQMLhKT06UwGVKHa43V-oyCTaA@mail.gmail.com>

2012/4/9 Zhenglun (Alan) Wei <zhenglun.wei at gmail.com>

>  Thank you, Dr. Brown, for pointing me out this example. However, I met a
> problem when I was trying to modify it.
>     I added a 'OutputFunction' subroutine to access the solution data by
> using DMDAGetVecArray. I tried it with single process, it keeps giving me
> this error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Null argument, when expecting valid pointer!
> [0]PETSC ERROR: Null Object: Parameter # 1!
>
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 85e6055943e0711fccdec2d08caeba48971d3d55  HG Date: Fri Sep 23 14:46:14 2011
> -0700
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex45 on a arch-linu named l2118a-linux.soecs.ku.edu by
> zlwei Mon Apr  9 17:11:02 2012
>
> [0]PETSC ERROR: Libraries linked from
> /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Fri Sep 23 17:13:32 2011
> [0]PETSC ERROR: Configure options --download-f-blas-lapack=1
> --download-mpich=1 --with-cc=gcc --with-fc=gfortran
> PETSC_ARCH=arch-linux2-c-debug
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: DMDAVecGetArray() line 51 in src/dm/impls/da/dagetarray.c
>

I suggest using the CHKERRQ macros so that the rest of the stack would be
shown here. In any case, the problem is that you gave up your reference to
the DM earlier. You can either move the DMDestroy() to the end of the
function or get access to the KSP's reference using

--- ex45.c.old  2012-04-09 19:13:21.438775057 -0500
+++ ex45.c      2012-04-09 19:13:42.661909463 -0500
@@ -60,6 +60,7 @@
   ierr = VecAXPY(r,-1.0,b);CHKERRQ(ierr);
   ierr = VecNorm(r,NORM_2,&norm);CHKERRQ(ierr);
   ierr = PetscPrintf(PETSC_COMM_WORLD,"Residual norm
%G\n",norm);CHKERRQ(ierr);
+  ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
   ierr = OutputFunction(da,x);

   ierr = VecDestroy(&r);CHKERRQ(ierr);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/8c29cb40/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  9 19:24:23 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 19:24:23 -0500
Subject: [petsc-users] How to access solution in
	/src/ksp/ksp/example/tutorial/ex45.c
In-Reply-To: <4F8364B9.1010601@gmail.com>
References: <4F8364B9.1010601@gmail.com>
Message-ID: <CAM9tzSkfApqrka97mg8Oh1yYNUc=+jFCxjutwY+vPggN5gv6Fg@mail.gmail.com>

2012/4/9 Zhenglun (Alan) Wei <zhenglun.wei at gmail.com>

> Dear All,
>    I hope you're having a nice day.
>    I'm working on ex45. It does not have any output subroutine inside.
> Therefore, I was trying to write out mine.
>    I was trying to use DMDAVecGetArray; but I found that the DM object has
> been destroyed at very beginning of the main code. Therefore, I swithed to
> VecGetArray. However, it gives me warning like:
> ex45.c:185: warning: passing argument 2 of 'VecGetArray' from incompatible
> pointer type
>    Then, it shows a memory violation error during execution, which is
> attached as 'out'. I checked 'VecGetArray' examples in PETSc manual, it
> always shows how to use it to extract one-dimensional array. I wonder if it
> can do 3-dimensional?
>

Use DMDAVecGetArray(), see my reply to your other message.


>
>    Also, there are other two minor questions:
> 1) in ex45.c, 'ComputeRHS', there is a if(x) {} statement, I write probe
> statements inside this if statement to see if the code ever go through it;
> the result shows it does not. I don't know why. ;(
>

Use a debugger and examine the stack to find out why that argument is 0.
Learning to use a debugger is absolutely a good time investment. In cases
like this, it's the difference between conclusively answering the question
in seconds and having to actually think.

2) probe statements are added in 'ComputeRHS' for both ex45.c and ex29.c.
> The size parameter used for DMDACreate3d in ex45 is -7, -7, -7; while for
> DMDACreate2d in ex29 is -7,-7. The probe statements show that the real 2d
> domain generated in ex29 is 25*25, which, as I understand, is because of
> multigrid. However, with multigrid scheme, the ex45 create a 7*7*7 3d
> domain. Any suggestions on this?
>

Please do not follow this version of ex29.c because it uses the deprecated
DMMG. The example has been updated in petsc-dev. The refinement was coming
from automatic refinement via DMMG.

http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex29.c.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/0deddca4/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 20:06:20 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 18:06:20 -0700
Subject: [petsc-users] Global numbering obtained via
 ISPartitioningToNumbering is inconsistent with the partitioning indexset
In-Reply-To: <CAM9tzSmjjLJhFXerqR0z9AjNrjEPF9CPiH76ZSepqAvA_BQsNg@mail.gmail.com>
References: <CAKsPKwnX+=wqgwMJRUs=XFS8oPKbr1zrvuREHDELmcc4muF=fg@mail.gmail.com>
	<A64BD599-4872-4A7F-AF34-94FDCD2BDF7C@mcs.anl.gov>
	<CAKsPKwkEXZF0QsLJ9mcnThqAbdhoDjHACV3A2vMWB3GoY_TyUQ@mail.gmail.com>
	<CAM9tzSmjjLJhFXerqR0z9AjNrjEPF9CPiH76ZSepqAvA_BQsNg@mail.gmail.com>
Message-ID: <CAKsPKwkmWbwd_bCM6UWJGHx_kDvo8Qd-mNwdmm8+YSnqquHdrQ@mail.gmail.com>

Thanks Jed for the advice. Actually I looked at the code and I think this
was the only place I had made this implicit assumption (but the most
important place). As a matter of fact, I make my vectors layout according
to the AO that I get from partitioning. I think my main mistake was that I
had assumed the partitioning uses the same ordering as the PETSc ordering.

The code seems to run correctly now :)





On Mon, Apr 9, 2012 at 5:09 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Mon, Apr 9, 2012 at 17:25, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Just to make sure though, is my assumption on the natural ordering of
>> PETSc correct?
>
>
> Yes, but please don't write code that depends on that. It's plenty easy to
> query the local size or the sizes/starts of other processes.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/4d31b0fa/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 20:10:58 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 18:10:58 -0700
Subject: [petsc-users] Strange error(?) message
Message-ID: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>

Hi,

I have this strange behavior that sometimes when I run the code on certain
number of processors and for a certain problem size, I get a message like:

"key # not found!"

where # is just some number (like 9087 but changes every time I run it). Is
there something wrong with my MPI? is this even an MPI message or something
else? Sorry my question is this broad, but I have no clue where this
message is popping from! PETSc does not produce any error message when this
happens and the code is valgrind clean.

I'd appreciate any input.
Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/a8513677/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 20:34:23 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 18:34:23 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
Message-ID: <CAKsPKw=0DYdtaGe2nOYaP9n6-RR4L+3UE-J_8-oQtOGLtbh6PQ@mail.gmail.com>

um, as a matter of fact, it now remains the same number as long as I use
the same number of processors!

On Mon, Apr 9, 2012 at 6:10 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi,
>
> I have this strange behavior that sometimes when I run the code on certain
> number of processors and for a certain problem size, I get a message like:
>
> "key # not found!"
>
> where # is just some number (like 9087 but changes every time I run it).
> Is there something wrong with my MPI? is this even an MPI message or
> something else? Sorry my question is this broad, but I have no clue where
> this message is popping from! PETSc does not produce any error message when
> this happens and the code is valgrind clean.
>
> I'd appreciate any input.
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/17f29dec/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Mon Apr  9 21:29:13 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 21:29:13 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
Message-ID: <CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>

Always send the whole error message. If this is all you see, then something
is misbehaving and you'll have to catch the message in a debugger.

On Mon, Apr 9, 2012 at 20:10, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Hi,
>
> I have this strange behavior that sometimes when I run the code on certain
> number of processors and for a certain problem size, I get a message like:
>
> "key # not found!"
>
> where # is just some number (like 9087 but changes every time I run it).
> Is there something wrong with my MPI? is this even an MPI message or
> something else? Sorry my question is this broad, but I have no clue where
> this message is popping from! PETSc does not produce any error message when
> this happens and the code is valgrind clean.
>
> I'd appreciate any input.
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/d9f0e187/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 21:56:07 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 19:56:07 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
Message-ID: <CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>

:( that's all I see. Do you mean running with -on_error_attach_debugger?

On Mon, Apr 9, 2012 at 7:29 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Always send the whole error message. If this is all you see, then
> something is misbehaving and you'll have to catch the message in a debugger.
>
>
> On Mon, Apr 9, 2012 at 20:10, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Hi,
>>
>> I have this strange behavior that sometimes when I run the code on
>> certain number of processors and for a certain problem size, I get a
>> message like:
>>
>> "key # not found!"
>>
>> where # is just some number (like 9087 but changes every time I run it).
>> Is there something wrong with my MPI? is this even an MPI message or
>> something else? Sorry my question is this broad, but I have no clue where
>> this message is popping from! PETSc does not produce any error message when
>> this happens and the code is valgrind clean.
>>
>> I'd appreciate any input.
>> Thanks
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/29abaee5/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  9 21:57:46 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 21:57:46 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
Message-ID: <CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>

On Mon, Apr 9, 2012 at 21:56, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> :( that's all I see. Do you mean running with -on_error_attach_debugger?


Sure, or directly in a debugger. Does execution make it to main? Something
is either not catching errors or is improperly calling exit() or similar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/6fef7909/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 22:19:13 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 20:19:13 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
Message-ID: <CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>

-on_error_attach_debugger does not start the debugger. I traced it upto the
following call:  MatPartitioningApply(). It runs fine up until that
function call and then gives the error.

Also, to make things even look more weird, I got the following once I was
running it:

Key 10176 not found!
INTERNAL ERROR: Invalid error class (66) encountered while returning from
PMPI_Waitall.  Please file a bug report.

Do you happen to know what that could mean? The funny thing is, it only
happened once!

On Mon, Apr 9, 2012 at 7:57 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Mon, Apr 9, 2012 at 21:56, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> :( that's all I see. Do you mean running with -on_error_attach_debugger?
>
>
> Sure, or directly in a debugger. Does execution make it to main? Something
> is either not catching errors or is improperly calling exit() or similar.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/1d6d0f61/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr  9 22:25:17 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 9 Apr 2012 22:25:17 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
Message-ID: <CCA1C1A3-1A73-4DCF-B0B8-F239FCFFF3C3@mcs.anl.gov>


  This is part of the excellent error handing in ParMetis (sarcasm intended), you would have found it rather quickly in the debugger. (ParMetis code follows)

idx_t BSearch(idx_t n, idx_t *array, idx_t key)
{
  idx_t a=0, b=n, c;

  while (b-a > 8) {
    c = (a+b)>>1;
    if (array[c] > key)
      b = c;
    else
      a = c;
  }

  for (c=a; c<b; c++) {
    if (array[c] == key)
      return c;
  }

  errexit("Key %"PRIDX" not found!\n", key);

  return 0;
}

  So either it is a bug in ParMetis or ParMetis is being passed a "ill-formed" graph and not detecting the "ill-formed" graph cleanly.

   What to do? Well we can't spend out lives debugging other peoples libraries so ..... I am not sure what to do next? Validate that Parmetis is getting reasonable input?

   Barry


On Apr 9, 2012, at 10:19 PM, Mohammad Mirzadeh wrote:

> -on_error_attach_debugger does not start the debugger. I traced it upto the following call:  MatPartitioningApply(). It runs fine up until that function call and then gives the error.
> 
> Also, to make things even look more weird, I got the following once I was running it:
> 
> Key 10176 not found!
> INTERNAL ERROR: Invalid error class (66) encountered while returning from
> PMPI_Waitall.  Please file a bug report.
> 
> Do you happen to know what that could mean? The funny thing is, it only happened once! 
> 
> On Mon, Apr 9, 2012 at 7:57 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Mon, Apr 9, 2012 at 21:56, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:
> :( that's all I see. Do you mean running with -on_error_attach_debugger? 
> 
> Sure, or directly in a debugger. Does execution make it to main? Something is either not catching errors or is improperly calling exit() or similar.
> 


From sean at mcs.anl.gov  Mon Apr  9 22:27:02 2012
From: sean at mcs.anl.gov (Sean Farley)
Date: Mon, 9 Apr 2012 22:27:02 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
Message-ID: <CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>

>
> -on_error_attach_debugger does not start the debugger. I traced it upto
> the following call:  MatPartitioningApply(). It runs fine up until that
> function call and then gives the error.
>
> Also, to make things even look more weird, I got the following once I was
> running it:
>
> Key 10176 not found!
> INTERNAL ERROR: Invalid error class (66) encountered while returning from
> PMPI_Waitall.  Please file a bug report.
>
> Do you happen to know what that could mean? The funny thing is, it only
> happened once!
>

This is a parmetis error from libparmetis/util.c:78 and not with petsc
(well, at least, not directly an error from petsc). How did you build
parmetis? Which parmetis version and which petsc version are you running?
You'll probably have to send configure.log to petsc-maint at mcs.anl.gov.

By the way, the relevant function that is having the error is:

/*************************************************************************
* This function does a binary search on an array for a key and returns
* the index
**************************************************************************/
idx_t BSearch(idx_t n, idx_t *array, idx_t key)

... which doesn't help much unless you can step into the debugger and set a
breakpoint here to give a traceback.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/b046d752/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 22:33:45 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 20:33:45 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CCA1C1A3-1A73-4DCF-B0B8-F239FCFFF3C3@mcs.anl.gov>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CCA1C1A3-1A73-4DCF-B0B8-F239FCFFF3C3@mcs.anl.gov>
Message-ID: <CAKsPKw=SSQZ0odA6g-B1bmXO2J1=CtPthDnep8XLGEdwSeMqwg@mail.gmail.com>

Barry,

Thanks a lot; I should definitely learn to use the debugger.

I'm not sure what makes the graph "ill-formed" in ParMetis terminology.
I'll keep looking into their manual to see if I can find anything. What
makes this even harder is the code runs fine for certain grids and then
this just happens as I refine the grid. I'll look into the function where I
set up the adjacency matrix to see if I'm doing something wrong ...

I'll keep you posted.



On Mon, Apr 9, 2012 at 8:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  This is part of the excellent error handing in ParMetis (sarcasm
> intended), you would have found it rather quickly in the debugger.
> (ParMetis code follows)
>
> idx_t BSearch(idx_t n, idx_t *array, idx_t key)
> {
>  idx_t a=0, b=n, c;
>
>  while (b-a > 8) {
>    c = (a+b)>>1;
>    if (array[c] > key)
>      b = c;
>    else
>      a = c;
>  }
>
>  for (c=a; c<b; c++) {
>    if (array[c] == key)
>      return c;
>  }
>
>  errexit("Key %"PRIDX" not found!\n", key);
>
>  return 0;
> }
>
>  So either it is a bug in ParMetis or ParMetis is being passed a
> "ill-formed" graph and not detecting the "ill-formed" graph cleanly.
>
>   What to do? Well we can't spend out lives debugging other peoples
> libraries so ..... I am not sure what to do next? Validate that Parmetis is
> getting reasonable input?
>
>   Barry
>
>
> On Apr 9, 2012, at 10:19 PM, Mohammad Mirzadeh wrote:
>
> > -on_error_attach_debugger does not start the debugger. I traced it upto
> the following call:  MatPartitioningApply(). It runs fine up until that
> function call and then gives the error.
> >
> > Also, to make things even look more weird, I got the following once I
> was running it:
> >
> > Key 10176 not found!
> > INTERNAL ERROR: Invalid error class (66) encountered while returning from
> > PMPI_Waitall.  Please file a bug report.
> >
> > Do you happen to know what that could mean? The funny thing is, it only
> happened once!
> >
> > On Mon, Apr 9, 2012 at 7:57 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > On Mon, Apr 9, 2012 at 21:56, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
> > :( that's all I see. Do you mean running with -on_error_attach_debugger?
> >
> > Sure, or directly in a debugger. Does execution make it to main?
> Something is either not catching errors or is improperly calling exit() or
> similar.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/d78d2e25/attachment-0001.htm>

From mirzadeh at gmail.com  Mon Apr  9 22:37:26 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 20:37:26 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
Message-ID: <CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>

Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since this
was not supported with 3.2-p6, and previous versions had bugs, I built
parmetis myself and used --with-parmetis-include and --with-parmetis-lib
flags to build petsc.

Should I switch to petsc-dev?

On Mon, Apr 9, 2012 at 8:27 PM, Sean Farley <sean at mcs.anl.gov> wrote:

> -on_error_attach_debugger does not start the debugger. I traced it upto
>> the following call:  MatPartitioningApply(). It runs fine up until that
>> function call and then gives the error.
>>
>> Also, to make things even look more weird, I got the following once I was
>> running it:
>>
>> Key 10176 not found!
>> INTERNAL ERROR: Invalid error class (66) encountered while returning from
>> PMPI_Waitall.  Please file a bug report.
>>
>> Do you happen to know what that could mean? The funny thing is, it only
>> happened once!
>>
>
> This is a parmetis error from libparmetis/util.c:78 and not with petsc
> (well, at least, not directly an error from petsc). How did you build
> parmetis? Which parmetis version and which petsc version are you running?
> You'll probably have to send configure.log to petsc-maint at mcs.anl.gov.
>
> By the way, the relevant function that is having the error is:
>
> /*************************************************************************
> * This function does a binary search on an array for a key and returns
> * the index
> **************************************************************************/
> idx_t BSearch(idx_t n, idx_t *array, idx_t key)
>
> ... which doesn't help much unless you can step into the debugger and set
> a breakpoint here to give a traceback.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/1886d7af/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr  9 22:42:50 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 9 Apr 2012 22:42:50 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
Message-ID: <CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>

On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since this
> was not supported with 3.2-p6, and previous versions had bugs, I built
> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
> flags to build petsc.
>
> Should I switch to petsc-dev?
>

Yes, and use --download-metis --download-parmetis because the version
upstream has some bugs for which the patches have not been applied.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/9885a1d2/attachment.htm>

From mirzadeh at gmail.com  Mon Apr  9 22:46:53 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 9 Apr 2012 20:46:53 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
Message-ID: <CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>

ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.

Thanks everyone.

On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since this
>> was not supported with 3.2-p6, and previous versions had bugs, I built
>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>> flags to build petsc.
>>
>> Should I switch to petsc-dev?
>>
>
> Yes, and use --download-metis --download-parmetis because the version
> upstream has some bugs for which the patches have not been applied.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120409/e6015ba5/attachment.htm>

From mirzadeh at gmail.com  Tue Apr 10 02:08:12 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 10 Apr 2012 00:08:12 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
Message-ID: <CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>

Just built petsc-dev and it did not help. I'm going to look into the code
to see if my graph is "ill-formed" in some sense. Just hope the problem is
from my side not a real bug in ParMetis!



On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>
> Thanks everyone.
>
>
> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since
>>> this was not supported with 3.2-p6, and previous versions had bugs, I built
>>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>>> flags to build petsc.
>>>
>>> Should I switch to petsc-dev?
>>>
>>
>> Yes, and use --download-metis --download-parmetis because the version
>> upstream has some bugs for which the patches have not been applied.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120410/99e9ebfd/attachment.htm>

From u.tabak at tudelft.nl  Tue Apr 10 04:27:42 2012
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Tue, 10 Apr 2012 11:27:42 +0200
Subject: [petsc-users] petsc binary matrix file for dense matrices
Message-ID: <4F83FD0E.2090704@tudelft.nl>

Dear all,

For a simple interface to SLEPc, I need to write my sparse or dense 
matrices in PETSc binary format, at first, I need to solve reduced dense 
generalized eigenvalue problems which are in dense matrix format.

I first had a look at the MATLAB routines to read and write the matrices 
as examples and MatLoad function manual. I got the idea however I have a 
question regarding the write operation for dense matrices, in MATLAB 
code, it is given as

     [m,n] = size(A);
     write(fd,[1211214,m*n],'int32');
     write(fd,A,'double');

I guess the dense matrix is written as a 1d array, is this right? and 
there is no need for row and column information for the matrix. I am 
guessing that MATLAB writes the matrix, A, in column-major orientation, 
is not this a problem for petsc dense matrices? Most probably this info 
is somewhere in the documentation, but could not find. Any pointers are 
appreciated.

Best,
Umut

From jedbrown at mcs.anl.gov  Tue Apr 10 06:55:37 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 10 Apr 2012 06:55:37 -0500
Subject: [petsc-users] petsc binary matrix file for dense matrices
In-Reply-To: <4F83FD0E.2090704@tudelft.nl>
References: <4F83FD0E.2090704@tudelft.nl>
Message-ID: <CAM9tzSkxUzygZRJgqvqM1ANK9+gTnojEmkLD-yftFFTadkGt1Q@mail.gmail.com>

On Tue, Apr 10, 2012 at 04:27, Umut Tabak <u.tabak at tudelft.nl> wrote:

> I first had a look at the MATLAB routines to read and write the matrices
> as examples and MatLoad function manual. I got the idea however I have a
> question regarding the write operation for dense matrices, in MATLAB code,
> it is given as
>
>    [m,n] = size(A);
>    write(fd,[1211214,m*n],'int32'**);
>    write(fd,A,'double');
>

I suggest just using

PetscBinaryWrite('filename',sparse(A))

The scripts are in $PETSC_DIR/bin/matlab.

PETSc binary format does not contain information for storing dense matrices
as dense. Note that you can still read the "sparse" matrix on disk as a
dense matrix. Also, if you compress the binary (e.g. gzip), it will be
essentially the same size (because the column indices are so regular). For
example, writing a 1000x1000 random matrix (the entries essentially don't
compress for a random matrix), I have

-rw-r--r-- 1 jed users  8000008 Apr 10 06:52 densematrix
-rw-r--r-- 1 jed users 12004016 Apr 10 06:52 sparsematrix
-rw-r--r-- 1 jed users 7539706 Apr 10 06:52 densematrix.gz
-rw-r--r-- 1 jed users 7568387 Apr 10 06:52 sparsematrix.gz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120410/c2bda468/attachment.htm>

From u.tabak at tudelft.nl  Tue Apr 10 07:34:56 2012
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Tue, 10 Apr 2012 14:34:56 +0200
Subject: [petsc-users] petsc binary matrix file for dense matrices
In-Reply-To: <CAM9tzSkxUzygZRJgqvqM1ANK9+gTnojEmkLD-yftFFTadkGt1Q@mail.gmail.com>
References: <4F83FD0E.2090704@tudelft.nl>
	<CAM9tzSkxUzygZRJgqvqM1ANK9+gTnojEmkLD-yftFFTadkGt1Q@mail.gmail.com>
Message-ID: <4F8428F0.9020608@tudelft.nl>

On 04/10/2012 01:55 PM, Jed Brown wrote:
> On Tue, Apr 10, 2012 at 04:27, Umut Tabak <u.tabak at tudelft.nl 
> <mailto:u.tabak at tudelft.nl>> wrote:
>
>     I first had a look at the MATLAB routines to read and write the
>     matrices as examples and MatLoad function manual. I got the idea
>     however I have a question regarding the write operation for dense
>     matrices, in MATLAB code, it is given as
>
>        [m,n] = size(A);
>        write(fd,[1211214,m*n],'int32');
>        write(fd,A,'double');
>
>
> I suggest just using
Dear Jed,

I looked at the MATLAB codes as an example, I should interface my 
matrices which are in the C++ code(they are MTL4 compressed or dense 
matrices), so they are not in MATLAB.

However, I guess I should use the format given on MatLoad page whether 
the matrices are sparse or dense.

I wrote a simple test case which writes the matrix in binary format, 
which in this case a sparse matrix, now I guess I can use the same 
routine to write a dense matrix as far as I could understand from 
MatLoad(There is no separate specification for a dense matrix, right?).

Best,
U.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120410/7d65aa19/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Tue Apr 10 07:38:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 10 Apr 2012 07:38:26 -0500
Subject: [petsc-users] petsc binary matrix file for dense matrices
In-Reply-To: <4F8428F0.9020608@tudelft.nl>
References: <4F83FD0E.2090704@tudelft.nl>
	<CAM9tzSkxUzygZRJgqvqM1ANK9+gTnojEmkLD-yftFFTadkGt1Q@mail.gmail.com>
	<4F8428F0.9020608@tudelft.nl>
Message-ID: <CAM9tzSnGxxpnLzPQoZonFSkFys4fKWC2pQpjoVYZCrJdsVEmaQ@mail.gmail.com>

On Tue, Apr 10, 2012 at 07:34, Umut Tabak <u.tabak at tudelft.nl> wrote:

> However, I guess I should use the format given on MatLoad page whether the
> matrices are sparse or dense.
>
> I wrote a simple test case which writes the matrix in binary format, which
> in this case a sparse matrix, now I guess I can use the same routine to
> write a dense matrix as far as I could understand from MatLoad(There is no
> separate specification for a dense matrix, right?).
>

Yes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120410/f94211bc/attachment.htm>

From mirzadeh at gmail.com  Wed Apr 11 15:24:49 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 11 Apr 2012 13:24:49 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
Message-ID: <CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>

I was reading the FAQ list when I came across the following:

http://www.mcs.anl.gov/petsc/documentation/faq.html#key

When calling MatPartitioningApply() you get a message Error! Key 16615 not
found The graph of the matrix you are using is not symmetric. You must use
symmetric matrices for partitioning.

Is this a limitation on ParMetis side? I set up the adjacency matrix based
on the discretization that I will be performing on the grid which is
non-symmetric; both numerically and structurally. What's the solution here?
Make an "approximate" adjacency matrix that sort of looks like
(structurally) my discretization but is symmetric? What I don't understand
is my matrix IS non-symmetric when the code runs on coarser grids!

Also, I was reading the FAQ hoping I can find something regarding using gdb
in parallel. I found this: http://scicomp.stackexchange.com/a/410/485 but
I'm not sure how I should be using gdb in parallel. Could you (maybe Matt?)
please explain a little bit?

Thanks

On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Just built petsc-dev and it did not help. I'm going to look into the code
> to see if my graph is "ill-formed" in some sense. Just hope the problem is
> from my side not a real bug in ParMetis!
>
>
>
> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>>
>> Thanks everyone.
>>
>>
>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since
>>>> this was not supported with 3.2-p6, and previous versions had bugs, I built
>>>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>>>> flags to build petsc.
>>>>
>>>> Should I switch to petsc-dev?
>>>>
>>>
>>> Yes, and use --download-metis --download-parmetis because the version
>>> upstream has some bugs for which the patches have not been applied.
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120411/3f5db597/attachment.htm>

From knepley at gmail.com  Wed Apr 11 17:07:08 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 11 Apr 2012 22:07:08 +0000
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
Message-ID: <CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>

On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> I was reading the FAQ list when I came across the following:
>
> http://www.mcs.anl.gov/petsc/documentation/faq.html#key
>
> When calling MatPartitioningApply() you get a message Error! Key 16615 not
> found The graph of the matrix you are using is not symmetric. You must
> use symmetric matrices for partitioning.
>
> Is this a limitation on ParMetis side? I set up the adjacency matrix based
> on the discretization that I will be performing on the grid which is
> non-symmetric; both numerically and structurally. What's the solution here?
> Make an "approximate" adjacency matrix that sort of looks like
> (structurally) my discretization but is symmetric? What I don't understand
> is my matrix IS non-symmetric when the code runs on coarser grids!
>

I don't quite understand how you can have a non-symmetric adjacency
description. But Metis/ParMetis partitions undirected graphs, which
by definition have symmetric adjacency matrices. Non-symmetric adjacency
would seem to imply a directed graph of some sort, or more plainly,
something is a adjacent to another thing which is not adjacent to it. That
is a very strange concept.


> Also, I was reading the FAQ hoping I can find something regarding using
> gdb in parallel. I found this: http://scicomp.stackexchange.com/a/410/485but I'm not sure how I should be using gdb in parallel. Could you (maybe
> Matt?) please explain a little bit?
>

-start_in_debugger spawns a gdb windows for EVERY process and attaches it

-debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c

  Thanks,

     Matt


> Thanks
>
> On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Just built petsc-dev and it did not help. I'm going to look into the code
>> to see if my graph is "ill-formed" in some sense. Just hope the problem is
>> from my side not a real bug in ParMetis!
>>
>>
>>
>> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>>>
>>> Thanks everyone.
>>>
>>>
>>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>
>>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>
>>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since
>>>>> this was not supported with 3.2-p6, and previous versions had bugs, I built
>>>>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>>>>> flags to build petsc.
>>>>>
>>>>> Should I switch to petsc-dev?
>>>>>
>>>>
>>>> Yes, and use --download-metis --download-parmetis because the version
>>>> upstream has some bugs for which the patches have not been applied.
>>>>
>>>
>>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120411/e8d3968f/attachment.htm>

From mirzadeh at gmail.com  Wed Apr 11 17:49:46 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 11 Apr 2012 15:49:46 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
Message-ID: <CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>

I see. Well the reason for non-symmetry is that I'm using a finite
difference discretization on quadtree AMR grids. When discretizing at a
hanging node, i.e. a node that is missing a direct neighbor, I use
 interpolation from nodes of the parent cell. When it comes to discretizing
at those nodes (i.e. nodes of the parent), however, they are on a coarser
grid and will have direct neighbors so they would not see the hanging node
in their discretization. This leads to a non-symmetric discretization
matrix which I was using (structurally) to define the adjacency matrix of
the graph. I think I can fix this by requiring the parent nodes to also
have a link to the hanging node (although they are not used in the
discretization anyway)

I'm just surprised that ParMetis does not complain about non-symmetric
graphs at lower resolutions and partitions them just fine!

As for the debugger, -start_in_debugger does not start the debugger for me
:(. I'm using mpirun -np 2 ./a.out -start_in_debugger. Is this the correct
way?




On Wed, Apr 11, 2012 at 3:07 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> I was reading the FAQ list when I came across the following:
>>
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#key
>>
>> When calling MatPartitioningApply() you get a message Error! Key 16615
>> not found The graph of the matrix you are using is not symmetric. You
>> must use symmetric matrices for partitioning.
>>
>> Is this a limitation on ParMetis side? I set up the adjacency matrix
>> based on the discretization that I will be performing on the grid which is
>> non-symmetric; both numerically and structurally. What's the solution here?
>> Make an "approximate" adjacency matrix that sort of looks like
>> (structurally) my discretization but is symmetric? What I don't understand
>> is my matrix IS non-symmetric when the code runs on coarser grids!
>>
>
> I don't quite understand how you can have a non-symmetric adjacency
> description. But Metis/ParMetis partitions undirected graphs, which
> by definition have symmetric adjacency matrices. Non-symmetric adjacency
> would seem to imply a directed graph of some sort, or more plainly,
> something is a adjacent to another thing which is not adjacent to it. That
> is a very strange concept.
>
>
>> Also, I was reading the FAQ hoping I can find something regarding using
>> gdb in parallel. I found this: http://scicomp.stackexchange.com/a/410/485but I'm not sure how I should be using gdb in parallel. Could you (maybe
>> Matt?) please explain a little bit?
>>
>
> -start_in_debugger spawns a gdb windows for EVERY process and attaches it
>
> -debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c
>
>   Thanks,
>
>      Matt
>
>
>> Thanks
>>
>> On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Just built petsc-dev and it did not help. I'm going to look into the
>>> code to see if my graph is "ill-formed" in some sense. Just hope the
>>> problem is from my side not a real bug in ParMetis!
>>>
>>>
>>>
>>> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>>>>
>>>> Thanks everyone.
>>>>
>>>>
>>>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>>
>>>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>>
>>>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since
>>>>>> this was not supported with 3.2-p6, and previous versions had bugs, I built
>>>>>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>>>>>> flags to build petsc.
>>>>>>
>>>>>> Should I switch to petsc-dev?
>>>>>>
>>>>>
>>>>> Yes, and use --download-metis --download-parmetis because the version
>>>>> upstream has some bugs for which the patches have not been applied.
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120411/70bf228c/attachment.htm>

From knepley at gmail.com  Wed Apr 11 19:43:30 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 12 Apr 2012 00:43:30 +0000
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
Message-ID: <CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>

On Wed, Apr 11, 2012 at 10:49 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> I see. Well the reason for non-symmetry is that I'm using a finite
> difference discretization on quadtree AMR grids. When discretizing at a
> hanging node, i.e. a node that is missing a direct neighbor, I use
>  interpolation from nodes of the parent cell. When it comes to discretizing
> at those nodes (i.e. nodes of the parent), however, they are on a coarser
> grid and will have direct neighbors so they would not see the hanging node
> in their discretization. This leads to a non-symmetric discretization
> matrix which I was using (structurally) to define the adjacency matrix of
> the graph. I think I can fix this by requiring the parent nodes to also
> have a link to the hanging node (although they are not used in the
> discretization anyway)
>
> I'm just surprised that ParMetis does not complain about non-symmetric
> graphs at lower resolutions and partitions them just fine!
>

ParMetis does not burden the user with a whole lot of diagnostic output :)


> As for the debugger, -start_in_debugger does not start the debugger for me
> :(. I'm using mpirun -np 2 ./a.out -start_in_debugger. Is this the correct
> way?
>

Send the entire output. It is likely that at the beginning there are lines
showing a failure to open an X window.

   Matt


>
> On Wed, Apr 11, 2012 at 3:07 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> I was reading the FAQ list when I came across the following:
>>>
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#key
>>>
>>> When calling MatPartitioningApply() you get a message Error! Key 16615
>>> not found The graph of the matrix you are using is not symmetric. You
>>> must use symmetric matrices for partitioning.
>>>
>>> Is this a limitation on ParMetis side? I set up the adjacency matrix
>>> based on the discretization that I will be performing on the grid which is
>>> non-symmetric; both numerically and structurally. What's the solution here?
>>> Make an "approximate" adjacency matrix that sort of looks like
>>> (structurally) my discretization but is symmetric? What I don't understand
>>> is my matrix IS non-symmetric when the code runs on coarser grids!
>>>
>>
>> I don't quite understand how you can have a non-symmetric adjacency
>> description. But Metis/ParMetis partitions undirected graphs, which
>> by definition have symmetric adjacency matrices. Non-symmetric adjacency
>> would seem to imply a directed graph of some sort, or more plainly,
>> something is a adjacent to another thing which is not adjacent to it.
>> That is a very strange concept.
>>
>>
>>> Also, I was reading the FAQ hoping I can find something regarding using
>>> gdb in parallel. I found this:
>>> http://scicomp.stackexchange.com/a/410/485 but I'm not sure how I
>>> should be using gdb in parallel. Could you (maybe Matt?) please explain a
>>> little bit?
>>>
>>
>> -start_in_debugger spawns a gdb windows for EVERY process and attaches it
>>
>> -debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks
>>>
>>> On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Just built petsc-dev and it did not help. I'm going to look into the
>>>> code to see if my graph is "ill-formed" in some sense. Just hope the
>>>> problem is from my side not a real bug in ParMetis!
>>>>
>>>>
>>>>
>>>> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>
>>>>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>>>>>
>>>>> Thanks everyone.
>>>>>
>>>>>
>>>>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>
>>>>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>>>
>>>>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since
>>>>>>> this was not supported with 3.2-p6, and previous versions had bugs, I built
>>>>>>> parmetis myself and used --with-parmetis-include and --with-parmetis-lib
>>>>>>> flags to build petsc.
>>>>>>>
>>>>>>> Should I switch to petsc-dev?
>>>>>>>
>>>>>>
>>>>>> Yes, and use --download-metis --download-parmetis because the version
>>>>>> upstream has some bugs for which the patches have not been applied.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/eba9df3c/attachment-0001.htm>

From u.tabak at tudelft.nl  Thu Apr 12 10:19:07 2012
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Thu, 12 Apr 2012 17:19:07 +0200
Subject: [petsc-users] A book advice on numerical computation errors
Message-ID: <4F86F26B.60506@tudelft.nl>

Dear all,

I would like to ask for a reference book on numerical errors in 
computations and ways to tackle with them, such as decreasing round of 
error, cancellation error and so on, probably with some programming 
examples? Are there any good references on this topic?

I would appreciate a pointer.

Best wishes,
Umut

From john.fettig at gmail.com  Thu Apr 12 10:35:03 2012
From: john.fettig at gmail.com (John Fettig)
Date: Thu, 12 Apr 2012 11:35:03 -0400
Subject: [petsc-users] A book advice on numerical computation errors
In-Reply-To: <4F86F26B.60506@tudelft.nl>
References: <4F86F26B.60506@tudelft.nl>
Message-ID: <CAD8Bu=r+v4s9GxfuK3e1ou=J_WDW6pkF0nSNq4khbT_WvY7mvA@mail.gmail.com>

This isn't a book, but may be helpful if you haven't read it:
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

John

On Thu, Apr 12, 2012 at 11:19 AM, Umut Tabak <u.tabak at tudelft.nl> wrote:

> Dear all,
>
> I would like to ask for a reference book on numerical errors in
> computations and ways to tackle with them, such as decreasing round of
> error, cancellation error and so on, probably with some programming
> examples? Are there any good references on this topic?
>
> I would appreciate a pointer.
>
> Best wishes,
> Umut
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/9bb1deb3/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 12 10:40:48 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 12 Apr 2012 10:40:48 -0500
Subject: [petsc-users] A book advice on numerical computation errors
In-Reply-To: <4F86F26B.60506@tudelft.nl>
References: <4F86F26B.60506@tudelft.nl>
Message-ID: <CAM9tzSmM7Q4LGJt=jLKGBHkfeTpmdCRkwETD2QvORZxZfUXoGQ@mail.gmail.com>

On Thu, Apr 12, 2012 at 10:19, Umut Tabak <u.tabak at tudelft.nl> wrote:

> I would like to ask for a reference book on numerical errors in
> computations and ways to tackle with them, such as decreasing round of
> error, cancellation error and so on, probably with some programming
> examples? Are there any good references on this topic?


This book has a good section devoted to numerical stability:

http://people.maths.ox.ac.uk/trefethen/text.html

This book is comprehensive:

http://www.maths.manchester.ac.uk/~higham/asna/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/a5360068/attachment.htm>

From zhenglun.wei at gmail.com  Thu Apr 12 11:37:19 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 12 Apr 2012 11:37:19 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
Message-ID: <4F8704BF.3070008@gmail.com>

Dear All,
      I hope you're having a nice day.
      I came across a problem while using MPI derived data type. 
Although it does not related to PETSc, I hope you guys can give me some 
suggestions or comments.
      The reason I want to do this is that, in my PETSc code, I want to 
send a set of data. It is fairly easy to do that if data is contiguous; 
while it would be comparatively hard if it is not. I searched online and 
found the MPI derived data type may help me. What I learned are 
MPI_TYPE_CONTIGUOUS, MPI_TYPE_VECTOR  and MPI_TYPE_STRUCT. I did a test 
program, which basically combined several example programs in 
"http://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types" with all 
these three MPI derived data types. It works fine. (Please see my 
attachment for AlanFloatStruct.c). Briefly saying, it generates a 4 by 4 
matrix 'a[i][j]', where 'a' is a structure with members of 'u' and 'v':

a[i][j].u is
    01   02   03   04      // 2 digits just for E-mail format. Actually, 
you may find that data type for member '.u' is 'double'
    05   06   07   08
    09   10   11   12
    13   14   15   16
a[i][j].v = 10*a[i][j].u  // I will neglect .v later, because I want to 
focus on .u member

      As I defined, 'j' in 'a[i][j]' is the inner loop. For example, 
a[0][0] = 1,  a[0][1] = 2, a[0][2] = 3 ... So, they are contiguous. 
Then, 'i' is the outer loop, a[0][0] = 1, a[1][0] = 5 ..., which are not 
contiguous.
      My purpose is let rank = 0 to send the 4 contiguous data of 
a[n][m] (m = 0~3) to n'th rank and store them to 'b[n][p]' in the n'th 
rank. Then, I want to send the 4 non-contiguous data of a[m][n] (m = 
0~3) to n'th rank and store them to 'c[p][n]' in the n'th rank. For 
example, if 'b' and 'c' are 4 by 4 matrix and initialized by '-5', after 
communication with rank = 0, I expect in rank = 1

b[i][j].u
    -5   -5   -5   -5
    05   06   07   08
    -5   -5   -5   -5
    -5   -5   -5   -5
c[i][j].v
    -5   02   -5   -5
    -5   06   -5   -5
    -5   10   -5   -5
    -5   14   -5   -5

     Everything is fine in AlanFloatStruct.c. *However, when I was 
trying to use 'calloc' to allocate memory to 'a[][]' rather than 
directly define 'a[][]' as a structure matrix, the problem comes.* (see 
AlanTEST1.c). 'b' matrix is ok, yet, it keeps giving me problems for 'c' 
that
c[i][j].u
    -5   02   -5   -5
    -5   00   -5   -5
    -5   00   -5   -5
    -5   00   -5   -5

     I guess it would be some problem for byte counting or padding of 
MPI data in 'calloc'. I refer to 'Notes' part in 
'www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Type_struct/html' 
and made AlanTEST2.c. The problem is still there.
     I wonder what should I for this? Or, does PETS have any alternative 
way to fix this problem. BTW, if you want to run my code, you need 
change the name of .c file to AlanTEST.c. My script file 'AlanRun' is 
written based on AlanTEST.c.

thank you so much in advance,
Alan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/a4b7493e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MPI_TYPE_STRUCT.zip
Type: application/x-zip-compressed
Size: 3845 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/a4b7493e/attachment.bin>

From knepley at gmail.com  Thu Apr 12 11:51:58 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 12 Apr 2012 16:51:58 +0000
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F8704BF.3070008@gmail.com>
References: <4F8704BF.3070008@gmail.com>
Message-ID: <CAMYG4G=vSqk6oF6u0LX1Vzo+a6HBgdHJ6UcgU5Bm=JaR5+Pp2Q@mail.gmail.com>

On Thu, Apr 12, 2012 at 4:37 PM, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> wrote:

>  Dear All,
>      I hope you're having a nice day.
>      I came across a problem while using MPI derived data type. Although
> it does not related to PETSc, I hope you guys can give me some suggestions
> or comments.
>      The reason I want to do this is that, in my PETSc code, I want to
> send a set of data. It is fairly easy to do that if data is contiguous;
> while it would be comparatively hard if it is not. I searched online and
> found the MPI derived data type may help me. What I learned are
> MPI_TYPE_CONTIGUOUS, MPI_TYPE_VECTOR  and MPI_TYPE_STRUCT. I did a test
> program, which basically combined several example programs in
> "http://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types"<http://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types>with all these three MPI derived data types. It works fine. (Please see my
> attachment for AlanFloatStruct.c). Briefly saying, it generates a 4 by 4
> matrix 'a[i][j]', where 'a' is a structure with members of 'u' and 'v':
>
> a[i][j].u is
>    01   02   03   04      // 2 digits just for E-mail format. Actually,
> you may find that data type for member '.u' is 'double'
>    05   06   07   08
>    09   10   11   12
>    13   14   15   16
> a[i][j].v = 10*a[i][j].u  // I will neglect .v later, because I want to
> focus on .u member
>
>      As I defined, 'j' in 'a[i][j]' is the inner loop. For example,
> a[0][0] = 1,  a[0][1] = 2, a[0][2] = 3 ... So, they are contiguous. Then,
> 'i' is the outer loop, a[0][0] = 1, a[1][0] = 5 ..., which are not
> contiguous.
>      My purpose is let rank = 0 to send the 4 contiguous data of a[n][m]
> (m = 0~3) to n'th rank and store them to 'b[n][p]' in the n'th rank. Then,
> I want to send the 4 non-contiguous data of a[m][n] (m = 0~3) to n'th rank
> and store them to 'c[p][n]' in the n'th rank. For example, if 'b' and 'c'
> are 4 by 4 matrix and initialized by '-5', after communication with rank =
> 0, I expect in rank = 1
>
> b[i][j].u
>    -5   -5   -5   -5
>    05   06   07   08
>    -5   -5   -5   -5
>    -5   -5   -5   -5
> c[i][j].v
>    -5   02   -5   -5
>    -5   06   -5   -5
>    -5   10   -5   -5
>    -5   14   -5   -5
>
>     Everything is fine in AlanFloatStruct.c. *However, when I was trying
> to use 'calloc' to allocate memory to 'a[][]' rather than directly define
> 'a[][]' as a structure matrix, the problem comes.* (see AlanTEST1.c). 'b'
> matrix is ok, yet, it keeps giving me problems for 'c' that
>

If you declare c[][] on the stack, most compilers will lay it out
contiguously. However, if you separately allocate each row, they
need not be contiguous. If you really want to do this, allocate one big
block, and setup the pointers to each row yourself.

  Thanks,

     Matt


> c[i][j].u
>    -5   02   -5   -5
>    -5   00   -5   -5
>    -5   00   -5   -5
>    -5   00   -5   -5
>
>     I guess it would be some problem for byte counting or padding of MPI
> data in 'calloc'. I refer to 'Notes' part in '
> www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Type_struct/html' and
> made AlanTEST2.c. The problem is still there.
>     I wonder what should I for this? Or, does PETS have any alternative
> way to fix this problem. BTW, if you want to run my code, you need change
> the name of .c file to AlanTEST.c. My script file 'AlanRun' is written
> based on AlanTEST.c.
>
> thank you so much in advance,
> Alan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/8591366c/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Thu Apr 12 11:52:33 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 12 Apr 2012 11:52:33 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F8704BF.3070008@gmail.com>
References: <4F8704BF.3070008@gmail.com>
Message-ID: <CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>

On Thu, Apr 12, 2012 at 11:37, Zhenglun (Alan) Wei
<zhenglun.wei at gmail.com>wrote:
>
>     Everything is fine in AlanFloatStruct.c. *However, when I was trying
> to use 'calloc' to allocate memory to 'a[][]' rather than directly define
> 'a[][]' as a structure matrix, the problem comes.* (see AlanTEST1.c).
>

    a = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
    b = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
    c = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
    for(i = 0; i < SIZE; i++) {
      a[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
      b[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
      c[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
    }

At this point, there is no relationship between a[i][j] and a[k][j]. It is
not strided memory. I suggest allocating the entire array in one allocation
(SIZE*SIZE*sizeof(TESTVAR)). You can set up pointers into it if you like,
but MPI will not use them.

    MPI_Type_vector(SIZE, 1, SIZE, structype, &columntype);

This specifies that the array is strided.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/25aaa14b/attachment.htm>

From mirzadeh at gmail.com  Thu Apr 12 11:56:16 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Thu, 12 Apr 2012 09:56:16 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
	<CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
Message-ID: <CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>

Thanks Matt. When I use -start_in_debugger, it just executes the program
without any extra message whatsoever! so its like I did not use the flag at
all.

Also, thought to share my conclusions on using ParMetis with others. The
two important things are:

1- Make sure you have a symmetric adjacency matrix for partitioning.
Chaning my non-symmetric matrix into a symmetric one solved the "Key 10716
not found!" problem.
2- Unlike PETSc, ParMetis does not equal number of nodes/elements to each
processors. The numbers are close, but not the same as you start
partitioning the grid with. It, however, does produce contigious
distribution of nodes/elements. You can use  ISPartitioningCount to figure
out how many nodes/elements are assigned to each processor.

Mohammad


On Wed, Apr 11, 2012 at 5:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Apr 11, 2012 at 10:49 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> I see. Well the reason for non-symmetry is that I'm using a finite
>> difference discretization on quadtree AMR grids. When discretizing at a
>> hanging node, i.e. a node that is missing a direct neighbor, I use
>>  interpolation from nodes of the parent cell. When it comes to discretizing
>> at those nodes (i.e. nodes of the parent), however, they are on a coarser
>> grid and will have direct neighbors so they would not see the hanging node
>> in their discretization. This leads to a non-symmetric discretization
>> matrix which I was using (structurally) to define the adjacency matrix of
>> the graph. I think I can fix this by requiring the parent nodes to also
>> have a link to the hanging node (although they are not used in the
>> discretization anyway)
>>
>> I'm just surprised that ParMetis does not complain about non-symmetric
>> graphs at lower resolutions and partitions them just fine!
>>
>
> ParMetis does not burden the user with a whole lot of diagnostic output :)
>
>
>> As for the debugger, -start_in_debugger does not start the debugger for
>> me :(. I'm using mpirun -np 2 ./a.out -start_in_debugger. Is this the
>> correct way?
>>
>
> Send the entire output. It is likely that at the beginning there are lines
> showing a failure to open an X window.
>
>    Matt
>
>
>>
>> On Wed, Apr 11, 2012 at 3:07 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> I was reading the FAQ list when I came across the following:
>>>>
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#key
>>>>
>>>> When calling MatPartitioningApply() you get a message Error! Key 16615
>>>> not found The graph of the matrix you are using is not symmetric. You
>>>> must use symmetric matrices for partitioning.
>>>>
>>>> Is this a limitation on ParMetis side? I set up the adjacency matrix
>>>> based on the discretization that I will be performing on the grid which is
>>>> non-symmetric; both numerically and structurally. What's the solution here?
>>>> Make an "approximate" adjacency matrix that sort of looks like
>>>> (structurally) my discretization but is symmetric? What I don't understand
>>>> is my matrix IS non-symmetric when the code runs on coarser grids!
>>>>
>>>
>>> I don't quite understand how you can have a non-symmetric adjacency
>>> description. But Metis/ParMetis partitions undirected graphs, which
>>> by definition have symmetric adjacency matrices. Non-symmetric adjacency
>>> would seem to imply a directed graph of some sort, or more plainly,
>>> something is a adjacent to another thing which is not adjacent to it.
>>> That is a very strange concept.
>>>
>>>
>>>> Also, I was reading the FAQ hoping I can find something regarding using
>>>> gdb in parallel. I found this:
>>>> http://scicomp.stackexchange.com/a/410/485 but I'm not sure how I
>>>> should be using gdb in parallel. Could you (maybe Matt?) please explain a
>>>> little bit?
>>>>
>>>
>>> -start_in_debugger spawns a gdb windows for EVERY process and attaches it
>>>
>>> -debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks
>>>>
>>>> On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <mirzadeh at gmail.com
>>>> > wrote:
>>>>
>>>>> Just built petsc-dev and it did not help. I'm going to look into the
>>>>> code to see if my graph is "ill-formed" in some sense. Just hope the
>>>>> problem is from my side not a real bug in ParMetis!
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>>
>>>>>> ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.
>>>>>>
>>>>>> Thanks everyone.
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>>
>>>>>>> On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>>>>
>>>>>>>> Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2.
>>>>>>>> Since this was not supported with 3.2-p6, and previous versions had bugs, I
>>>>>>>> built parmetis myself and used --with-parmetis-include and
>>>>>>>> --with-parmetis-lib flags to build petsc.
>>>>>>>>
>>>>>>>> Should I switch to petsc-dev?
>>>>>>>>
>>>>>>>
>>>>>>> Yes, and use --download-metis --download-parmetis because the
>>>>>>> version upstream has some bugs for which the patches have not been applied.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/c388e7a0/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 12 12:03:11 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 12 Apr 2012 12:03:11 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
	<CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
	<CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>
Message-ID: <CAM9tzSktjb0SdJJWeAZDs1zA9XaFmun62o+mndqOJDOq-SWjCg@mail.gmail.com>

On Thu, Apr 12, 2012 at 11:56, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> 2- Unlike PETSc, ParMetis does not equal number of nodes/elements to each
> processors. The numbers are close, but not the same as you start
> partitioning the grid with. It, however, does produce contigious
> distribution of nodes/elements.


Contiguous partitions are not guaranteed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/ed2902e6/attachment.htm>

From mirzadeh at gmail.com  Thu Apr 12 12:08:30 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Thu, 12 Apr 2012 10:08:30 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAM9tzSktjb0SdJJWeAZDs1zA9XaFmun62o+mndqOJDOq-SWjCg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
	<CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
	<CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>
	<CAM9tzSktjb0SdJJWeAZDs1zA9XaFmun62o+mndqOJDOq-SWjCg@mail.gmail.com>
Message-ID: <CAKsPKwkuAcvznQ6NvRqHyw_cK6tgUwm8H09y8z-r8GuGKCNKYg@mail.gmail.com>

Humm. What I meant by contiguous is in terms of new global numberings. That
is for proc 0: [0, np0), for proc 1: [np0, np1), for proc 2: [np1, np2)
..., for proc p-1: [ np-2, ntotal). I did not mean
(topologically) contiguous partitions. Is this not the case?

On Thu, Apr 12, 2012 at 10:03 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Thu, Apr 12, 2012 at 11:56, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> 2- Unlike PETSc, ParMetis does not equal number of nodes/elements to each
>> processors. The numbers are close, but not the same as you start
>> partitioning the grid with. It, however, does produce contigious
>> distribution of nodes/elements.
>
>
> Contiguous partitions are not guaranteed.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/95c9595d/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 12 12:14:19 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 12 Apr 2012 12:14:19 -0500
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAKsPKwkuAcvznQ6NvRqHyw_cK6tgUwm8H09y8z-r8GuGKCNKYg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
	<CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
	<CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>
	<CAM9tzSktjb0SdJJWeAZDs1zA9XaFmun62o+mndqOJDOq-SWjCg@mail.gmail.com>
	<CAKsPKwkuAcvznQ6NvRqHyw_cK6tgUwm8H09y8z-r8GuGKCNKYg@mail.gmail.com>
Message-ID: <CAM9tzSmrYJ0x4OmgWg5zrEGm3GL_1eaazVN1k+-scj--hqKYsg@mail.gmail.com>

On Thu, Apr 12, 2012 at 12:08, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Humm. What I meant by contiguous is in terms of new global numberings.
> That is for proc 0: [0, np0), for proc 1: [np0, np1), for proc 2: [np1,
> np2) ..., for proc p-1: [ np-2, ntotal). I did not mean
> (topologically) contiguous partitions. Is this not the case?
>

The global numbering produced by ISPartitioningToNumbering() is contiguous,
but each part might not be topologically connected.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/65bfcdf3/attachment.htm>

From mirzadeh at gmail.com  Thu Apr 12 12:21:38 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Thu, 12 Apr 2012 10:21:38 -0700
Subject: [petsc-users] Strange error(?) message
In-Reply-To: <CAM9tzSmrYJ0x4OmgWg5zrEGm3GL_1eaazVN1k+-scj--hqKYsg@mail.gmail.com>
References: <CAKsPKw=OdZ3vMR09funxzKx-Q+oWeEBkHFdi2HsNMqeyb5Z6ng@mail.gmail.com>
	<CAM9tzS=doCQmxc97dZvwJuK36h98EWBiHw2WUam8iUY0aK0tEg@mail.gmail.com>
	<CAKsPKwn9eXMLy3t5=her+qsD+5QxQyyp6aAUG35Gtd7zqWXNzw@mail.gmail.com>
	<CAM9tzSmb6zgMd3EVHrw4z+9nzQ+TjOsjx57iNqBVeWxE1cKpdw@mail.gmail.com>
	<CAKsPKwnmbEHHL9y1MXdf9JL+4ON_pg931aYPRDKwnWYU-Q==Qg@mail.gmail.com>
	<CANH+MCjL_22Lvdv83BCugWLKfioCeqEHXZ=Sn_YozxUScYNnWQ@mail.gmail.com>
	<CAKsPKwmnqg-_GY_qMS9GvUW_A=GKt==R8XG=osMZYsJ+mApxUQ@mail.gmail.com>
	<CAM9tzSkjVJfPK6G7GWdrUE4j0dY5+iRmF_ft5eAqLOhSr40UEg@mail.gmail.com>
	<CAKsPKw=PJqzZnjYvUkZ=niv5pJ+rLB=q=_ctBccb1g3WzRLPMA@mail.gmail.com>
	<CAKsPKwmg7pAO75BG1Y+KMD5kUdpHT-Bt1J-LSou2CzgWpFQJZA@mail.gmail.com>
	<CAKsPKwmq9YM7bsv2kScMLyZB8Eeq-tvVrhRKtwM7DyRBMP9yqw@mail.gmail.com>
	<CAMYG4Gku=pQu4LVS2qVGybK+C3xAbovfR_BzX2R0HOPWwvdsZQ@mail.gmail.com>
	<CAKsPKwnqk+WbkdKagzCZStHJOef+tw-6etsk5_X5okunfA0CPw@mail.gmail.com>
	<CAMYG4GnJXN_veScYWM0Wyyw6Z4-ey_eKC-RXAmtuG1S8MZpCOw@mail.gmail.com>
	<CAKsPKw=O=rBdwUvZMXSjc4GHEdn2sFdt-LQfjOtS2jViMA3K_A@mail.gmail.com>
	<CAM9tzSktjb0SdJJWeAZDs1zA9XaFmun62o+mndqOJDOq-SWjCg@mail.gmail.com>
	<CAKsPKwkuAcvznQ6NvRqHyw_cK6tgUwm8H09y8z-r8GuGKCNKYg@mail.gmail.com>
	<CAM9tzSmrYJ0x4OmgWg5zrEGm3GL_1eaazVN1k+-scj--hqKYsg@mail.gmail.com>
Message-ID: <CAKsPKwmR2mFLCTCcvJq4LZkk+hQgA=RPPuPcjovKhMRW-+4Wmg@mail.gmail.com>

Correct! That's what I thought :)

On Thu, Apr 12, 2012 at 10:14 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Thu, Apr 12, 2012 at 12:08, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Humm. What I meant by contiguous is in terms of new global numberings.
>> That is for proc 0: [0, np0), for proc 1: [np0, np1), for proc 2: [np1,
>> np2) ..., for proc p-1: [ np-2, ntotal). I did not mean
>> (topologically) contiguous partitions. Is this not the case?
>>
>
> The global numbering produced by ISPartitioningToNumbering() is
> contiguous, but each part might not be topologically connected.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/dad1ed86/attachment.htm>

From zhenglun.wei at gmail.com  Thu Apr 12 13:16:46 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 12 Apr 2012 13:16:46 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
Message-ID: <4F871C0E.7030402@gmail.com>

On 4/12/2012 11:52 AM, Jed Brown wrote:
> On Thu, Apr 12, 2012 at 11:37, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>         Everything is fine in AlanFloatStruct.c. *However, when I was
>     trying to use 'calloc' to allocate memory to 'a[][]' rather than
>     directly define 'a[][]' as a structure matrix, the problem comes.*
>     (see AlanTEST1.c).
>
>
>     a = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
>     b = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
>     c = (TESTVAR**) calloc(SIZE, sizeof(TESTVAR*));
>     for(i = 0; i < SIZE; i++) {
>       a[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
>       b[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
>       c[i] = (TESTVAR*) calloc(SIZE, sizeof(TESTVAR));
>     }
>
> At this point, there is no relationship between a[i][j] and a[k][j]. 
> It is not strided memory. I suggest allocating the entire array in one 
> allocation (SIZE*SIZE*sizeof(TESTVAR)). You can set up pointers into 
> it if you like, but MPI will not use them.
>
>     MPI_Type_vector(SIZE, 1, SIZE, structype, &columntype);
>
> This specifies that the array is strided.
Dear Dr. Brown,
      Thank you for your reply. It gives me another way to solve this 
problem. However, I failed several times while I was trying it. Can I 
repeat your idea to make sure that I understand it.
1)  should define a new one-dimensional pointer array, i.e. TESTVAR *array;
2)  I allocate this array with array = (TESTVAR*) calloc(SIZE*SIZE, 
sizeof(TESTVAR));
3)  then, I setup pointer of 'a[i][j]'  to array.
       Are they correct? BTW, does PETSc have any data structure can be 
an alternative so that I do not need to use MPI derived data type?

thanks,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/947d895c/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 12 13:22:01 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 12 Apr 2012 13:22:01 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F871C0E.7030402@gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
Message-ID: <CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>

On Thu, Apr 12, 2012 at 13:16, Zhenglun (Alan) Wei
<zhenglun.wei at gmail.com>wrote:

>      Thank you for your reply. It gives me another way to solve this
> problem. However, I failed several times while I was trying it. Can I
> repeat your idea to make sure that I understand it.
> 1)  should define a new one-dimensional pointer array, i.e. TESTVAR *array;
> 2)  I allocate this array with array = (TESTVAR*) calloc(SIZE*SIZE,
> sizeof(TESTVAR));
> 3)  then, I setup pointer of 'a[i][j]'  to array.
>

the_struct **a = malloc(SIZE*sizeof(the_struct));
for (i=0; i<SIZE; i++) a = &array[i*SIZE];

Now you can use a[i][j] instead of array[i*SIZE+j].

With C99, you can just create a VLA pointer:

Scalar (*a)[SIZE] = (Scalar(*)[SIZE])array;

and then use a[i][j] syntax without needing to allocate those pointers.
It's a shame Microsoft refuses to do any maintenance on their old C
compiler.


>       Are they correct? BTW, does PETSc have any data structure can be an
> alternative so that I do not need to use MPI derived data type?
>

You can use VecScatter, for example.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120412/76ae40fc/attachment-0001.htm>

From recrusader at gmail.com  Fri Apr 13 10:37:15 2012
From: recrusader at gmail.com (recrusader)
Date: Fri, 13 Apr 2012 10:37:15 -0500
Subject: [petsc-users] about array in VecCreateMPIWithArray()
Message-ID: <CAA==Fcvq9_mZWVEy+JcmFfc3VAji6N-53fY6sg=FEetJ5qmnEg@mail.gmail.com>

Dear PETSc developers,

my question is what type of array is need for VecCreateMPIWithArray?
it is parallel distributed, that each processor holds local values of the
array? or each processor has a copy of all the values of the array?

PetscErrorCode  VecCreateMPIWithArray(MPI_Comm comm,PetscInt
n,PetscInt N,const PetscScalar array[],Vec *vv)

I didn't find more information about the array from the manual.
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html

Thanks a lot,
Yujie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/6a9e8aba/attachment.htm>

From knepley at gmail.com  Fri Apr 13 10:38:57 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 13 Apr 2012 11:38:57 -0400
Subject: [petsc-users] about array in VecCreateMPIWithArray()
In-Reply-To: <CAA==Fcvq9_mZWVEy+JcmFfc3VAji6N-53fY6sg=FEetJ5qmnEg@mail.gmail.com>
References: <CAA==Fcvq9_mZWVEy+JcmFfc3VAji6N-53fY6sg=FEetJ5qmnEg@mail.gmail.com>
Message-ID: <CAMYG4GmNP9uOhFp-QEyT9HLSYROKnMrNTv-b6xE7McYV3_OtoA@mail.gmail.com>

On Fri, Apr 13, 2012 at 11:37 AM, recrusader <recrusader at gmail.com> wrote:

> Dear PETSc developers,
>
> my question is what type of array is need for VecCreateMPIWithArray?
> it is parallel distributed, that each processor holds local values of the
> array? or each processor has a copy of all the values of the array?
>
>
Only local values.

   Matt


> PetscErrorCode  VecCreateMPIWithArray(MPI_Comm comm,PetscInt n,PetscInt N,const PetscScalar array[],Vec *vv)
>
> I didn't find more information about the array from the manual.
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html
>
> Thanks a lot,
> Yujie
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/46e78e9f/attachment.htm>

From recrusader at gmail.com  Fri Apr 13 10:59:21 2012
From: recrusader at gmail.com (recrusader)
Date: Fri, 13 Apr 2012 10:59:21 -0500
Subject: [petsc-users] about array in VecCreateMPIWithArray()
In-Reply-To: <CAMYG4GmNP9uOhFp-QEyT9HLSYROKnMrNTv-b6xE7McYV3_OtoA@mail.gmail.com>
References: <CAA==Fcvq9_mZWVEy+JcmFfc3VAji6N-53fY6sg=FEetJ5qmnEg@mail.gmail.com>
	<CAMYG4GmNP9uOhFp-QEyT9HLSYROKnMrNTv-b6xE7McYV3_OtoA@mail.gmail.com>
Message-ID: <CAA==Fcvc_ZL7EOiuf09WaMbkMz0EdOssPoHP6qP3_-xa6eD7nA@mail.gmail.com>

Thanks, Matt.

If I have one array, each processor has a copy of all the values.
one possible option is
VecCreateMPI();
VecSetValues();
VecAssemblyBegin();
VecAssemblyEnd();

any better idea?

if I have one array, only one processor has a copy of all the values.
How to generate a MPI vector?

Best,
Yujie

On Fri, Apr 13, 2012 at 10:38 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Apr 13, 2012 at 11:37 AM, recrusader <recrusader at gmail.com> wrote:
>
>> Dear PETSc developers,
>>
>> my question is what type of array is need for VecCreateMPIWithArray?
>> it is parallel distributed, that each processor holds local values of the
>> array? or each processor has a copy of all the values of the array?
>>
>>
> Only local values.
>
>    Matt
>
>
>> PetscErrorCode  VecCreateMPIWithArray(MPI_Comm comm,PetscInt n,PetscInt N,const PetscScalar array[],Vec *vv)
>>
>> I didn't find more information about the array from the manual.
>>
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html
>>
>> Thanks a lot,
>> Yujie
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/41cedc4b/attachment.htm>

From knepley at gmail.com  Fri Apr 13 11:01:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 13 Apr 2012 12:01:34 -0400
Subject: [petsc-users] about array in VecCreateMPIWithArray()
In-Reply-To: <CAA==Fcvc_ZL7EOiuf09WaMbkMz0EdOssPoHP6qP3_-xa6eD7nA@mail.gmail.com>
References: <CAA==Fcvq9_mZWVEy+JcmFfc3VAji6N-53fY6sg=FEetJ5qmnEg@mail.gmail.com>
	<CAMYG4GmNP9uOhFp-QEyT9HLSYROKnMrNTv-b6xE7McYV3_OtoA@mail.gmail.com>
	<CAA==Fcvc_ZL7EOiuf09WaMbkMz0EdOssPoHP6qP3_-xa6eD7nA@mail.gmail.com>
Message-ID: <CAMYG4GnFEJ6XSxVfXTzT-3DWXd-7ib_iJQrYoM2Wz0Cpk==6NA@mail.gmail.com>

On Fri, Apr 13, 2012 at 11:59 AM, recrusader <recrusader at gmail.com> wrote:

> Thanks, Matt.
>
> If I have one array, each processor has a copy of all the values.
> one possible option is
>

1) If all processors have all values the code is already not scalable

2) If everyone has all values, only set the values you own

   Matt


> VecCreateMPI();
> VecSetValues();
> VecAssemblyBegin();
> VecAssemblyEnd();
>
> any better idea?
>
> if I have one array, only one processor has a copy of all the values.
> How to generate a MPI vector?
>
> Best,
> Yujie
>
> On Fri, Apr 13, 2012 at 10:38 AM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, Apr 13, 2012 at 11:37 AM, recrusader <recrusader at gmail.com>wrote:
>>
>>> Dear PETSc developers,
>>>
>>> my question is what type of array is need for VecCreateMPIWithArray?
>>> it is parallel distributed, that each processor holds local values of
>>> the array? or each processor has a copy of all the values of the array?
>>>
>>>
>> Only local values.
>>
>>    Matt
>>
>>
>>> PetscErrorCode  VecCreateMPIWithArray(MPI_Comm comm,PetscInt n,PetscInt N,const PetscScalar array[],Vec *vv)
>>>
>>> I didn't find more information about the array from the manual.
>>>
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html
>>>
>>> Thanks a lot,
>>> Yujie
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/74e1b144/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 13 17:17:31 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 13 Apr 2012 15:17:31 -0700
Subject: [petsc-users] Petsc CMake conf
Message-ID: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>

Hi,

Are there any cmake conf file that I can include in my CMakeLists.txt that
would automatically detect include/lib directory for my petsc installation?
Right now I'm doing this manually and was wondering if its possibel to
automate it.

Thanks,
Mohammad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/afe50c61/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 13 17:19:07 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 13 Apr 2012 17:19:07 -0500
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
Message-ID: <CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>

On Fri, Apr 13, 2012 at 17:17, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Hi,
>
> Are there any cmake conf file that I can include in my CMakeLists.txt that
> would automatically detect include/lib directory for my petsc installation?
> Right now I'm doing this manually and was wondering if its possibel to
> automate it.
>

https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/3d85d788/attachment.htm>

From zhenglun.wei at gmail.com  Fri Apr 13 17:38:34 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Fri, 13 Apr 2012 17:38:34 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
Message-ID: <4F88AAEA.40406@gmail.com>

Dear Dr. Brown,
     Your brilliant idea works. :) Thank you so much.
     I have a final question on it. Is it taken a lot of memory for 
doing this? As I understand, pointers won't occupy many memories and it 
works like an alias. It will not, to my limit knowledge, take much extra 
memory by doing this.
     Is that true?

thanks,
Alan
On 4/12/2012 1:22 PM, Jed Brown wrote:
> On Thu, Apr 12, 2012 at 13:16, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>          Thank you for your reply. It gives me another way to solve
>     this problem. However, I failed several times while I was trying
>     it. Can I repeat your idea to make sure that I understand it.
>     1)  should define a new one-dimensional pointer array, i.e.
>     TESTVAR *array;
>     2)  I allocate this array with array = (TESTVAR*)
>     calloc(SIZE*SIZE, sizeof(TESTVAR));
>     3)  then, I setup pointer of 'a[i][j]'  to array.
>
>
> the_struct **a = malloc(SIZE*sizeof(the_struct));
> for (i=0; i<SIZE; i++) a = &array[i*SIZE];
>
> Now you can use a[i][j] instead of array[i*SIZE+j].
>
> With C99, you can just create a VLA pointer:
>
> Scalar (*a)[SIZE] = (Scalar(*)[SIZE])array;
>
> and then use a[i][j] syntax without needing to allocate those 
> pointers. It's a shame Microsoft refuses to do any maintenance on 
> their old C compiler.
>
>           Are they correct? BTW, does PETSc have any data structure
>     can be an alternative so that I do not need to use MPI derived
>     data type?
>
>
> You can use VecScatter, for example.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/0c60869a/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 13 17:41:57 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 13 Apr 2012 17:41:57 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F88AAEA.40406@gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
Message-ID: <CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>

On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei
<zhenglun.wei at gmail.com>wrote:

>     I have a final question on it. Is it taken a lot of memory for doing
> this? As I understand, pointers won't occupy many memories and it works
> like an alias. It will not, to my limit knowledge, take much extra memory
> by doing this.


A pointer takes about as much space as a floating point value, so that
array of pointers costs about 1*N compared to the N*N matrix.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/13427abf/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 13 17:47:08 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 13 Apr 2012 15:47:08 -0700
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
Message-ID: <CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>

Sweet :D -- Thanks Jed.

Just a question though. When I run cmake in the build directory I get:

-- Performing Test MULTIPASS_TEST_1_petsc_works_minimal
-- Performing Test MULTIPASS_TEST_1_petsc_works_minimal - Failed
-- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes
-- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes - Failed
-- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries
-- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries - Failed
-- Performing Test MULTIPASS_TEST_4_petsc_works_all
-- Performing Test MULTIPASS_TEST_4_petsc_works_all - Failed
-- PETSc could not be used, maybe the install is broken.
-- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
 (missing:  PETSC_EXECUTABLE_RUNS)
-- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
 (missing:  PETSC_EXECUTABLE_RUNS)


But actually I can make and run the binary without any problem! should I
ignore this? I do set PETSC_DIR and PETSC_ARCH in my CMakeLists.txt.



On Fri, Apr 13, 2012 at 3:19 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

>
>
> On Fri, Apr 13, 2012 at 17:17, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Hi,
>>
>> Are there any cmake conf file that I can include in my CMakeLists.txt
>> that would automatically detect include/lib directory for my petsc
>> installation? Right now I'm doing this manually and was wondering if its
>> possibel to automate it.
>>
>
> https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/c60a9cc5/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 13 17:49:36 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 13 Apr 2012 17:49:36 -0500
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
Message-ID: <CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>

On Fri, Apr 13, 2012 at 17:47, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> ust a question though. When I run cmake in the build directory I get:
>
> -- Performing Test MULTIPASS_TEST_1_petsc_works_minimal
> -- Performing Test MULTIPASS_TEST_1_petsc_works_minimal - Failed
> -- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes
> -- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes - Failed
> -- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries
> -- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries - Failed
> -- Performing Test MULTIPASS_TEST_4_petsc_works_all
> -- Performing Test MULTIPASS_TEST_4_petsc_works_all - Failed
> -- PETSc could not be used, maybe the install is broken.
> -- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
>  (missing:  PETSC_EXECUTABLE_RUNS)
> -- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
>  (missing:  PETSC_EXECUTABLE_RUNS)
>
>
> But actually I can make and run the binary without any problem! should I
> ignore this? I do set PETSC_DIR and PETSC_ARCH in my CMakeLists.txt.
>

Which binary runs? Check the CMake error files and use cmake --trace or
--debug-output.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/46ab8017/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 13 17:57:33 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 13 Apr 2012 15:57:33 -0700
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
Message-ID: <CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>

On Fri, Apr 13, 2012 at 3:49 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Fri, Apr 13, 2012 at 17:47, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> ust a question though. When I run cmake in the build directory I get:
>>
>> -- Performing Test MULTIPASS_TEST_1_petsc_works_minimal
>> -- Performing Test MULTIPASS_TEST_1_petsc_works_minimal - Failed
>> -- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes
>> -- Performing Test MULTIPASS_TEST_2_petsc_works_allincludes - Failed
>> -- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries
>> -- Performing Test MULTIPASS_TEST_3_petsc_works_alllibraries - Failed
>> -- Performing Test MULTIPASS_TEST_4_petsc_works_all
>> -- Performing Test MULTIPASS_TEST_4_petsc_works_all - Failed
>> -- PETSc could not be used, maybe the install is broken.
>> -- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
>>  (missing:  PETSC_EXECUTABLE_RUNS)
>> -- PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
>>  (missing:  PETSC_EXECUTABLE_RUNS)
>>
>>
>> But actually I can make and run the binary without any problem! should I
>> ignore this? I do set PETSC_DIR and PETSC_ARCH in my CMakeLists.txt.
>>
>
> Which binary runs? Check the CMake error files and use cmake --trace or
> --debug-output.
>


my code -- I'm experimenting with CMake to manage my project and link it to
petsc. right now, it seems to link to petsc correctly and I can run the
code just fine. It just produces those messages when generating the
makefile.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/0b41ce9b/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Fri Apr 13 18:01:00 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 13 Apr 2012 18:01:00 -0500
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
Message-ID: <CAM9tzSkrY+40HsehUunAzPb30jMUx+BfnWPvE+y4jXuOwzRd-Q@mail.gmail.com>

On Fri, Apr 13, 2012 at 17:57, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> my code -- I'm experimenting with CMake to manage my project and link it
> to petsc. right now, it seems to link to petsc correctly and I can run the
> code just fine. It just produces those messages when generating the
> makefile.


Feel free to investigate why it was failing and let me know or send a
patch. It shouldn't be hard to figure out from the logs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/3606fc64/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 13 18:03:32 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 13 Apr 2012 16:03:32 -0700
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAM9tzSkrY+40HsehUunAzPb30jMUx+BfnWPvE+y4jXuOwzRd-Q@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<CAM9tzSkrY+40HsehUunAzPb30jMUx+BfnWPvE+y4jXuOwzRd-Q@mail.gmail.com>
Message-ID: <CAKsPKwkQQ7pyALV1EPS6W0TCNXDzC-ZiLOh6PvLVhiOFbC5AFQ@mail.gmail.com>

Sure. Will let you know if I found something.

Once again thanks. One of the best things about PETSc is the community  --
You don't see this often in other packages :)

On Fri, Apr 13, 2012 at 4:01 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Fri, Apr 13, 2012 at 17:57, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> my code -- I'm experimenting with CMake to manage my project and link it
>> to petsc. right now, it seems to link to petsc correctly and I can run the
>> code just fine. It just produces those messages when generating the
>> makefile.
>
>
> Feel free to investigate why it was failing and let me know or send a
> patch. It shouldn't be hard to figure out from the logs.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120413/beb92454/attachment.htm>

From rlmackie862 at gmail.com  Fri Apr 13 18:52:56 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Fri, 13 Apr 2012 16:52:56 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
Message-ID: <8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>

In using ksp_monitor_true_residual_norm, is it possible to change how often this information is printed out?
That is, instead of every iteration, say I only want to see it every 10 or 20 iterations. Is there an easy way
to do this, other than creating my own monitor and doing it myself?

Thanks, Randy M.

From bsmith at mcs.anl.gov  Fri Apr 13 19:01:56 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 13 Apr 2012 19:01:56 -0500
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
Message-ID: <8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>


   Write a one line monitor that calls the true residual monitor when desired:

    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
    {
        if (n % 10) {
           ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
        }
    }

    then call this on the KPS

   ierr = KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);


On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:

> In using ksp_monitor_true_residual_norm, is it possible to change how often this information is printed out?
> That is, instead of every iteration, say I only want to see it every 10 or 20 iterations. Is there an easy way
> to do this, other than creating my own monitor and doing it myself?
> 
> Thanks, Randy M.


From dalcinl at gmail.com  Sat Apr 14 05:08:33 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 14 Apr 2012 13:08:33 +0300
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
Message-ID: <CAEcYPwAEH1+_2NOQDCd3njCaKnKAFJe2q4mQ6MFK4jAhSNxALA@mail.gmail.com>

On 14 April 2012 01:19, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>
> On Fri, Apr 13, 2012 at 17:17, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:
>>
>> Hi,
>>
>> Are there any cmake conf file that I can include in my CMakeLists.txt that
>> would automatically detect include/lib directory for my petsc installation?
>> Right now I'm doing this manually and was wondering if its possibel to
>> automate it.
>
>
> https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake
>

Jed, have you considered adding this to petsc-dev, removing all the
checks for old PETSc releases? Or perhaps better, how about  dumping
an autogenerated PETSc.cmake in
$PETSC_DIR/$PETSC_ARCH/conf/PETSc.cmake, or perhaps other name,
defining all the stuff end users need. For example, I'm using the code
below in a codebase I'm  working on (note that at the end I'm loading
PETScConfig.cmake), but this is not robust, it does not handle builds
with separate libraries. If PETSc provides all these variables, then I
could simply  include (${PETSC_DIR}/${PETSC_ARCH}/conf/PETSc.cmake),
and you can make your FindPETSc.cmake much simpler (at least for the
upcoming version of PETSc)


  find_path (PETSC_DIR include/petsc.h HINTS ENV PETSC_DIR DOC "PETSc
Directory")
  set (PETSC_ARCH $ENV{PETSC_ARCH})
  find_path (PETSC_INCLUDE_DIR  petsc.h HINTS "${PETSC_DIR}"
PATH_SUFFIXES include NO_DEFAULT_PATH)
  find_path (PETSC_INCLUDE_CONF petscconf.h HINTS "${PETSC_DIR}"
PATH_SUFFIXES "${PETSC_ARCH}/include" NO_DEFAULT_PATH)
  mark_as_advanced (PETSC_INCLUDE_DIR PETSC_INCLUDE_CONF)
  set (PETSC_INCLUDES ${PETSC_INCLUDE_CONF} ${PETSC_INCLUDE_DIR} CACHE
PATH "PETSc include paths" FORCE)
  set (PETSC_DEFINITIONS "-D__INSDIR__=" CACHE STRING "PETSc
preprocesor definitions" FORCE)
  find_path (PETSC_LIB_DIR NAMES "" HINTS "${PETSC_DIR}" PATH_SUFFIXES
"${PETSC_ARCH}/lib" NO_DEFAULT_PATH)
  find_library (PETSC_LIBRARY NAMES petsc HINTS "${PETSC_LIB_DIR}"
NO_DEFAULT_PATH)
  set (PETSC_LIBRARIES ${PETSC_LIBRARY} CACHE FILEPATH "PETSc library" FORCE)
  include (${PETSC_DIR}/${PETSC_ARCH}/conf/PETScConfig.cmake)


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From jedbrown at mcs.anl.gov  Sat Apr 14 09:10:52 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 14 Apr 2012 09:10:52 -0500
Subject: [petsc-users] Petsc CMake conf
In-Reply-To: <CAEcYPwAEH1+_2NOQDCd3njCaKnKAFJe2q4mQ6MFK4jAhSNxALA@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAEcYPwAEH1+_2NOQDCd3njCaKnKAFJe2q4mQ6MFK4jAhSNxALA@mail.gmail.com>
Message-ID: <CAM9tzS=3gzBJZn7uwPsfWsbMcOa64M5nGsNNztdfH69D0wBRXw@mail.gmail.com>

On Sat, Apr 14, 2012 at 05:08, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>
> Jed, have you considered adding this to petsc-dev, removing all the
> checks for old PETSc releases?


I get the impression that a lot of projects want to maintain one or two
versions of backward compatibility (e.g. so they can let their users use
the versions in Debian). I definitely think PETSc should write a pkg-config
file, that would be trivial for CMake users to consume as well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120414/5489d674/attachment.htm>

From zhenglun.wei at gmail.com  Sat Apr 14 19:00:03 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Sat, 14 Apr 2012 19:00:03 -0500
Subject: [petsc-users] Any Examples for 3D Poisson Solver with both
 Dirichlet and Neumann BC's
Message-ID: <4F8A0F83.40300@gmail.com>

Dear All,
     Any suggestions on a 3D Poisson Solver with both Dirichlet and 
Neumann BC's?
     Also, does PETSc have any example of Poisson solver (either 2D or 
3D) with periodic boundary conditions.

thanks
Zhenglun

From aron.ahmadia at kaust.edu.sa  Sat Apr 14 19:08:03 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Sun, 15 Apr 2012 03:08:03 +0300
Subject: [petsc-users] Any Examples for 3D Poisson Solver with both
 Dirichlet and Neumann BC's
In-Reply-To: <4F8A0F83.40300@gmail.com>
References: <4F8A0F83.40300@gmail.com>
Message-ID: <CADVoUZSBBAmjZuYmq95+gM1uSTNZ5wnnG_tYG4uKUz1ENn5D4A@mail.gmail.com>

http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex34.c.htmlhas
examples both Neumann and Dirichlet BCs...

A

On Sun, Apr 15, 2012 at 3:00 AM, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> wrote:

> Dear All,
>    Any suggestions on a 3D Poisson Solver with both Dirichlet and Neumann
> BC's?
>    Also, does PETSc have any example of Poisson solver (either 2D or 3D)
> with periodic boundary conditions.
>
> thanks
> Zhenglun
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120415/38a56c9c/attachment.htm>

From jkozdon at stanford.edu  Sun Apr 15 15:24:37 2012
From: jkozdon at stanford.edu (Jeremy Kozdon)
Date: Sun, 15 Apr 2012 13:24:37 -0700
Subject: [petsc-users] multiblock example
Message-ID: <CADXDPriEf8B+unjVbcEKMKiinVya2Dodmz-wk__VvLNjV1ArSQ@mail.gmail.com>

Hello All,

We are considering using petsc for a new code we are developing which
involves multiblock grids and finite difference methods. I am still pretty
new to petsc (still reading the manual), so wondering where a good place to
look for examples on this is (since I assume that we are not the first to
consider this). Not sure if there is a good place in the petsc source to
consider this, one of the libraries examples perhaps, or another opensource
package.

One of the things we want to be able to do is assign different numbers of
processors to different blocks based on the work load we estimate for that
block. Currently we do this by decomposing the global communicator into a
block communicators, but if I understand correctly this is not the correct
approach to take with petsc since we will need to use DMComposite which
relies on collective operations.

Currently everything we are doing is with structured cartesian grids, but
in the future we will also need to be able to handle unstructured grids
which are coupled to the structured grids.

Thanks!
Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120415/d33f86b7/attachment.htm>

From knepley at gmail.com  Sun Apr 15 16:54:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 15 Apr 2012 17:54:34 -0400
Subject: [petsc-users] multiblock example
In-Reply-To: <CADXDPriEf8B+unjVbcEKMKiinVya2Dodmz-wk__VvLNjV1ArSQ@mail.gmail.com>
References: <CADXDPriEf8B+unjVbcEKMKiinVya2Dodmz-wk__VvLNjV1ArSQ@mail.gmail.com>
Message-ID: <CAMYG4GnRChRRn2885Vxx5bRRkhwdE2=kyPJsGZ-8nPFYjdd9Qw@mail.gmail.com>

On Sun, Apr 15, 2012 at 4:24 PM, Jeremy Kozdon <jkozdon at stanford.edu> wrote:

> Hello All,
>
> We are considering using petsc for a new code we are developing which
> involves multiblock grids and finite difference methods. I am still pretty
> new to petsc (still reading the manual), so wondering where a good place to
> look for examples on this is (since I assume that we are not the first to
> consider this). Not sure if there is a good place in the petsc source to
> consider this, one of the libraries examples perhaps, or another opensource
> package.
>
> One of the things we want to be able to do is assign different numbers of
> processors to different blocks based on the work load we estimate for that
> block. Currently we do this by decomposing the global communicator into a
> block communicators, but if I understand correctly this is not the correct
> approach to take with petsc since we will need to use DMComposite which
> relies on collective operations.
>

The Cartesian grid interface is the DMDA class in PETSc. You can prescribe
whatever processor layout you want. The key
question is how will the blocks interact. The DMComposite is fine for
putting blocks together to generate a combined residual,
but the Jacobian interface is not as fleshed out, for example it does not
automatically allocate the off-diagonal blocks from
block interaction. I don't think there are any problems with collectives in
DMComposite. Do you have a specific thing that
does not work for you?


> Currently everything we are doing is with structured cartesian grids, but
> in the future we will also need to be able to handle unstructured grids
> which are coupled to the structured grids.
>

This can also be done with DMComposite, using DMComplex for the
unstructured grids.

   Matt


> Thanks!
> Jeremy
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120415/7fe8e934/attachment.htm>

From ataicher at ices.utexas.edu  Mon Apr 16 11:25:04 2012
From: ataicher at ices.utexas.edu (Abraham Taicher)
Date: Mon, 16 Apr 2012 11:25:04 -0500
Subject: [petsc-users] PETSc Gather
Message-ID: <4F8C47E0.4080900@ices.utexas.edu>

Hi,

I have a parallel vector spread across all processors that I would like
to gather into a single processor, say the root processor.  Is there
PETSc functionality to do this? I tried looking for a PETSc gather
function but the one I found, VecStrideGather
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html>,
does not seem to apply.  Should I simply use the MPI routine MPI_Gather?

Thanks,

Abraham Taicher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/5a845b63/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr 16 11:26:58 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 16 Apr 2012 11:26:58 -0500
Subject: [petsc-users] PETSc Gather
In-Reply-To: <4F8C47E0.4080900@ices.utexas.edu>
References: <4F8C47E0.4080900@ices.utexas.edu>
Message-ID: <CAM9tzSk1dK-0c0qZ6BUoi_MqBXTukQdtfqDHoNVP27Jbtj7RrQ@mail.gmail.com>

On Mon, Apr 16, 2012 at 11:25, Abraham Taicher <ataicher at ices.utexas.edu>wrote:

> **
> Hi,
>
> I have a parallel vector spread across all processors that I would like to
> gather into a single processor, say the root processor.  Is there PETSc
> functionality to do this? I tried looking for a PETSc gather function but
> the one I found, VecStrideGather<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html>,
> does not seem to apply.  Should I simply use the MPI routine MPI_Gather?
>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/cfaa328e/attachment.htm>

From hanangul12 at yahoo.co.uk  Mon Apr 16 12:04:21 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Mon, 16 Apr 2012 18:04:21 +0100 (BST)
Subject: [petsc-users] Complex Number
Message-ID: <1334595861.28342.YahooMailNeo@web28701.mail.ir2.yahoo.com>

Dear users n devlopers,?

My Petsc routine contans a complex number, as a parameter of my problem. 

I need to provide it at the run time options, since I intend to test a variety of parameter.
In this regard, I read that PetscScalar is the way to declare a complex number.?
If I need merely a complex number at run time option to compile a compact routine, 

any details from any examples of libraries ? 


Thanks in anticipation.?

Abdul,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/b4fcf9b5/attachment.htm>

From abhyshr at mcs.anl.gov  Mon Apr 16 12:20:15 2012
From: abhyshr at mcs.anl.gov (Shri)
Date: Mon, 16 Apr 2012 12:20:15 -0500 (CDT)
Subject: [petsc-users] Complex Number
In-Reply-To: <1334595861.28342.YahooMailNeo@web28701.mail.ir2.yahoo.com>
Message-ID: <1714267901.145576.1334596815310.JavaMail.root@zimbra-mb2.anl.gov>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsGetScalar.html ----- Original Message -----
> Dear users n devlopers,
> My Petsc routine contans a complex number, as a parameter of my
> problem.
> I need to provide it at the run time options, since I intend to test a
> variety of parameter.
> In this regard, I read that PetscScalar is the way to declare a
> complex number.
> If I need merely a complex number at run time option to compile a
> compact routine,
> any details from any examples of libraries ?
> Thanks in anticipation.
> Abdul,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/51a5e59b/attachment.htm>

From hanangul12 at yahoo.co.uk  Mon Apr 16 13:05:55 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Mon, 16 Apr 2012 19:05:55 +0100 (BST)
Subject: [petsc-users] Complex Number -  VecScale
In-Reply-To: <1714267901.145576.1334596815310.JavaMail.root@zimbra-mb2.anl.gov>
References: <1334595861.28342.YahooMailNeo@web28701.mail.ir2.yahoo.com>
	<1714267901.145576.1334596815310.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <1334599555.63368.YahooMailNeo@web28702.mail.ir2.yahoo.com>

Thanks! :)?

I have an other query.? 

By using VecScale on vector "x" with scalar "alpha", it overwrites the vector "x" with "alpha \times x" . 

Can we re-use the vector "x" after scaling ? 


thanks, 

?
Abdul




>________________________________
> From: Shri <abhyshr at mcs.anl.gov>
>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
>Sent: Monday, 16 April 2012, 19:20
>Subject: Re: [petsc-users] Complex Number
> 
>
>http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsGetScalar.html
>
>>________________________________
>
>Dear users n devlopers,?
>>
>>
>>My Petsc routine contans a complex number, as a parameter of my problem. 
>>
>>I need to provide it at the run time options, since I intend to test a variety of parameter.
>>In this regard, I read that PetscScalar is the way to declare a complex number.?
>>If I need merely a complex number at run time option to compile a compact routine, 
>>
>>any details from any examples of libraries ? 
>>
>>
>>
>>Thanks in anticipation.?
>>
>>
>>Abdul,
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/05798a7f/attachment.htm>

From knepley at gmail.com  Mon Apr 16 13:18:02 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 16 Apr 2012 14:18:02 -0400
Subject: [petsc-users] Complex Number - VecScale
In-Reply-To: <1334599555.63368.YahooMailNeo@web28702.mail.ir2.yahoo.com>
References: <1334595861.28342.YahooMailNeo@web28701.mail.ir2.yahoo.com>
	<1714267901.145576.1334596815310.JavaMail.root@zimbra-mb2.anl.gov>
	<1334599555.63368.YahooMailNeo@web28702.mail.ir2.yahoo.com>
Message-ID: <CAMYG4Gm+0vEFRmb3=0epf-O6Hk+JNxifdVhL1xgkMXy1hT_dhQ@mail.gmail.com>

On Mon, Apr 16, 2012 at 2:05 PM, Abdul Hanan Sheikh
<hanangul12 at yahoo.co.uk>wrote:

> Thanks! :)
>
> I have an other query.
> By using VecScale on vector "x" with scalar "alpha", it overwrites the
> vector "x" with "alpha \times x" .
> Can we re-use the vector "x" after scaling ?
>

No. If you want the original vector around,  use VecAXPY() instead.

   Matt


> thanks,
>
> Abdul
>
>   ------------------------------
> *From:* Shri <abhyshr at mcs.anl.gov>
> *To:* Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Monday, 16 April 2012, 19:20
> *Subject:* Re: [petsc-users] Complex Number
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsGetScalar.html
>
> ------------------------------
>
> Dear users n devlopers,
>
> My Petsc routine contans a complex number, as a parameter of my problem.
> I need to provide it at the run time options, since I intend to test a
> variety of parameter.
> In this regard, I read that PetscScalar is the way to declare a complex
> number.
> If I need merely a complex number at run time option to compile a compact
> routine,
> any details from any examples of libraries ?
>
> Thanks in anticipation.
>
> Abdul,
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/0c0f08e7/attachment.htm>

From juhaj at iki.fi  Mon Apr 16 16:11:12 2012
From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=)
Date: Mon, 16 Apr 2012 23:11:12 +0200
Subject: [petsc-users] determining dof at runtime
Message-ID: <201204162311.16383.juhaj@iki.fi>

Hi list!

I am trying to create a 3D DA such that I can use DMDAVecGetArray() and access 
it using array indexing. I know how to get several dof in the DA and how using 
a struct to access the members works, when the dof is determined at compile-
time (so accesses like array[1][2][3].member1[2] are computed correctly.

However, I would like to determine dof at run-time. Creating a struct with 
pointers (even of the type member[]) of course gives me a struct whose size 
depends on the *pointer* size, not my dof. So I was wondering if I could work 
around this somehow, but do not seem to be able to.

My DA is created by

DMDACreate3d(PETSC_COMM_WORLD,
             DMDA_BOUNDARY_GHOSTED, DMDA_BOUNDARY_GHOSTED, 
             DMDA_BOUNDARY_GHOSTED, DMDA_STENCIL_BOX,
             2, 2, 2,
             PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE,
             ndof, NGHOSTS,
             PETSC_NULL, PETSC_NULL, PETSC_NULL, 
             &da);

so my Vec does have the correct dof, but I cannot seem to fool C to address 
this as array[z][y][x][n] or something of the like, where the last index is 
the index to ndof. (Usually I would index that as array[z][y][x].member[n], 
but now I do not know how to make that happen since I cannot define a struct 
whose storage size is not known to the compiler.)

Any ideas? Or am I trying to accomplish something which is not possible? (If 
so, why would that be? The vector is of the correct size.)

Cheers,
-Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		 -----------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/4b62ebe5/attachment.pgp>

From knepley at gmail.com  Mon Apr 16 16:15:06 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 16 Apr 2012 17:15:06 -0400
Subject: [petsc-users] determining dof at runtime
In-Reply-To: <201204162311.16383.juhaj@iki.fi>
References: <201204162311.16383.juhaj@iki.fi>
Message-ID: <CAMYG4G=E8vDOioF7=rC6y4hc6QQ4o=CivBJxoQE9=4i-TVzDsA@mail.gmail.com>

On Mon, Apr 16, 2012 at 5:11 PM, Juha J?ykk? <juhaj at iki.fi> wrote:

> Hi list!
>
> I am trying to create a 3D DA such that I can use DMDAVecGetArray() and
> access
> it using array indexing. I know how to get several dof in the DA and how
> using
> a struct to access the members works, when the dof is determined at
> compile-
> time (so accesses like array[1][2][3].member1[2] are computed correctly.
>
> However, I would like to determine dof at run-time. Creating a struct with
> pointers (even of the type member[]) of course gives me a struct whose size
> depends on the *pointer* size, not my dof. So I was wondering if I could
> work
> around this somehow, but do not seem to be able to.
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMDAVecGetArrayDOF.html

   Matt


> My DA is created by
>
> DMDACreate3d(PETSC_COMM_WORLD,
>             DMDA_BOUNDARY_GHOSTED, DMDA_BOUNDARY_GHOSTED,
>             DMDA_BOUNDARY_GHOSTED, DMDA_STENCIL_BOX,
>             2, 2, 2,
>             PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE,
>             ndof, NGHOSTS,
>             PETSC_NULL, PETSC_NULL, PETSC_NULL,
>             &da);
>
> so my Vec does have the correct dof, but I cannot seem to fool C to address
> this as array[z][y][x][n] or something of the like, where the last index is
> the index to ndof. (Usually I would index that as array[z][y][x].member[n],
> but now I do not know how to make that happen since I cannot define a
> struct
> whose storage size is not known to the compiler.)
>
> Any ideas? Or am I trying to accomplish something which is not possible?
> (If
> so, why would that be? The vector is of the correct size.)
>
> Cheers,
> -Juha
>
> --
>                 -----------------------------------------------
>                | Juha J?ykk?, juhaj at iki.fi                     |
>                 -----------------------------------------------
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120416/eb57cd02/attachment-0001.htm>

From zonexo at gmail.com  Tue Apr 17 06:07:38 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 17 Apr 2012 13:07:38 +0200
Subject: [petsc-users] PETSc Gather
In-Reply-To: <CAM9tzSk1dK-0c0qZ6BUoi_MqBXTukQdtfqDHoNVP27Jbtj7RrQ@mail.gmail.com>
References: <4F8C47E0.4080900@ices.utexas.edu>
	<CAM9tzSk1dK-0c0qZ6BUoi_MqBXTukQdtfqDHoNVP27Jbtj7RrQ@mail.gmail.com>
Message-ID: <4F8D4EFA.7040904@gmail.com>

Hi,

I'm not sure if this is a naive question, but is there any speed or 
performance difference when using VecScatterCreateToZero or similar 
routine, compared to MPI standard routines such as mpi_gather, 
mpi_gatherv etc?

Yours sincerely,

TAY wee-beng


On 16/4/2012 6:26 PM, Jed Brown wrote:
>
>
> On Mon, Apr 16, 2012 at 11:25, Abraham Taicher 
> <ataicher at ices.utexas.edu <mailto:ataicher at ices.utexas.edu>> wrote:
>
>     Hi,
>
>     I have a parallel vector spread across all processors that I would
>     like to gather into a single processor, say the root processor. 
>     Is there PETSc functionality to do this? I tried looking for a
>     PETSc gather function but the one I found, VecStrideGather
>     <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html>,
>     does not seem to apply.  Should I simply use the MPI routine
>     MPI_Gather?
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/7fad7999/attachment.htm>

From knepley at gmail.com  Tue Apr 17 06:10:24 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 17 Apr 2012 07:10:24 -0400
Subject: [petsc-users] PETSc Gather
In-Reply-To: <4F8D4EFA.7040904@gmail.com>
References: <4F8C47E0.4080900@ices.utexas.edu>
	<CAM9tzSk1dK-0c0qZ6BUoi_MqBXTukQdtfqDHoNVP27Jbtj7RrQ@mail.gmail.com>
	<4F8D4EFA.7040904@gmail.com>
Message-ID: <CAMYG4GnhN3JQgytN9MYGMgEcPx1j2jiSP99_mxwCD0nUznGzoA@mail.gmail.com>

On Tue, Apr 17, 2012 at 7:07 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I'm not sure if this is a naive question, but is there any speed or
> performance difference when using VecScatterCreateToZero or similar
> routine, compared to MPI standard routines such as mpi_gather, mpi_gatherv
> etc?
>

No.

   Matt


> Yours sincerely,
>
> TAY wee-beng
>
>
> On 16/4/2012 6:26 PM, Jed Brown wrote:
>
>
>
> On Mon, Apr 16, 2012 at 11:25, Abraham Taicher <ataicher at ices.utexas.edu>wrote:
>
>>  Hi,
>>
>> I have a parallel vector spread across all processors that I would like
>> to gather into a single processor, say the root processor.  Is there PETSc
>> functionality to do this? I tried looking for a PETSc gather function but
>> the one I found, VecStrideGather<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html>,
>> does not seem to apply.  Should I simply use the MPI routine MPI_Gather?
>>
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/426b96d6/attachment.htm>

From rlmackie862 at gmail.com  Tue Apr 17 12:27:27 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Tue, 17 Apr 2012 10:27:27 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
Message-ID: <CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>

Hi Barry,

I've tried implementing this in Fortran, following ex2f.F in
/src/ksp/ksp/examples/tutorials, but
it is bombing out with the following message:

---------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
CDT 2011
-------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
CDT 2011


[1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
src/sys/viewer/impls/ascii/filev.c
[1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
src/ksp/ksp/interface/iterativ.c
[1]PETSC ERROR: ourmonitor() line 103 in
src/ksp/ksp/interface/ftn-custom/zitfuncf.c
[1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
[1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c


I am running PETSC 3.2-p5.

I have set my monitor routine as follows:

  subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)

  implicit none

#include "finclude/petsc.h"

  KSP :: ksp
  PetscErrorCode :: ierr
  PetscInt :: n, dummy
  PetscReal :: rnorm

  if (n == 200) then
    call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
  end if

  return
  end subroutine ShellKSPMonitor

And then in the main program I call:

  call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT,
PETSC_NULL_FUNCTION, ierr)

As I have followed the example in the PETSc documentation, I am unsure
where I have made a mistake.

Thanks,

Randy M


On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Write a one line monitor that calls the true residual monitor when
> desired:
>
>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt
> n,PetscReal rnorm,void *dummy)
>    {
>        if (n % 10) {
>           ierr =
> KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>        }
>    }
>
>    then call this on the KPS
>
>   ierr =
> KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
>
>
> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
>
> > In using ksp_monitor_true_residual_norm, is it possible to change how
> often this information is printed out?
> > That is, instead of every iteration, say I only want to see it every 10
> or 20 iterations. Is there an easy way
> > to do this, other than creating my own monitor and doing it myself?
> >
> > Thanks, Randy M.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/43926130/attachment.htm>

From knepley at gmail.com  Tue Apr 17 12:39:32 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 17 Apr 2012 13:39:32 -0400
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
Message-ID: <CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>

On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com>wrote:

> Hi Barry,
>
> I've tried implementing this in Fortran, following ex2f.F in
> /src/ksp/ksp/examples/tutorials, but
> it is bombing out with the following message:
>
> ---------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
> -------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
>
>
> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
> src/sys/viewer/impls/ascii/filev.c
> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
> src/ksp/ksp/interface/iterativ.c
> [1]PETSC ERROR: ourmonitor() line 103 in
> src/ksp/ksp/interface/ftn-custom/zitfuncf.c
> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
>
>
> I am running PETSC 3.2-p5.
>
> I have set my monitor routine as follows:
>
>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
>
>   implicit none
>
> #include "finclude/petsc.h"
>
>   KSP :: ksp
>   PetscErrorCode :: ierr
>   PetscInt :: n, dummy
>   PetscReal :: rnorm
>
>   if (n == 200) then
>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
>

I think it should work if dummy is 0, but we should really do NULL
conversion
here and you should pass PETSC_NULL.

   Matt


>   end if
>
>   return
>   end subroutine ShellKSPMonitor
>
> And then in the main program I call:
>
>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT,
> PETSC_NULL_FUNCTION, ierr)
>
> As I have followed the example in the PETSc documentation, I am unsure
> where I have made a mistake.
>
> Thanks,
>
> Randy M
>
>
> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Write a one line monitor that calls the true residual monitor when
>> desired:
>>
>>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt
>> n,PetscReal rnorm,void *dummy)
>>    {
>>        if (n % 10) {
>>           ierr =
>> KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>>        }
>>    }
>>
>>    then call this on the KPS
>>
>>   ierr =
>> KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
>>
>>
>> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
>>
>> > In using ksp_monitor_true_residual_norm, is it possible to change how
>> often this information is printed out?
>> > That is, instead of every iteration, say I only want to see it every 10
>> or 20 iterations. Is there an easy way
>> > to do this, other than creating my own monitor and doing it myself?
>> >
>> > Thanks, Randy M.
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/a778e32a/attachment.htm>

From rlmackie862 at gmail.com  Tue Apr 17 13:13:18 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Tue, 17 Apr 2012 11:13:18 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
Message-ID: <18197698-3EE2-43E3-9CC6-A61C011E2C79@gmail.com>

Hi Matt,

I'm afraid it didn't work by passing in dummy=0. 

Randy


On Apr 17, 2012, at 10:39 AM, Matthew Knepley wrote:

> On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> Hi Barry,
> 
> I've tried implementing this in Fortran, following ex2f.F in /src/ksp/ksp/examples/tutorials, but
> it is bombing out with the following message:
> 
> ---------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> -------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> 
> 
> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in src/sys/viewer/impls/ascii/filev.c
> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in src/ksp/ksp/interface/iterativ.c
> [1]PETSC ERROR: ourmonitor() line 103 in src/ksp/ksp/interface/ftn-custom/zitfuncf.c
> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
> 
> 
> I am running PETSC 3.2-p5.
> 
> I have set my monitor routine as follows:
> 
>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
> 
>   implicit none
> 
> #include "finclude/petsc.h"
> 
>   KSP :: ksp
>   PetscErrorCode :: ierr
>   PetscInt :: n, dummy
>   PetscReal :: rnorm
> 
>   if (n == 200) then
>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
> 
> I think it should work if dummy is 0, but we should really do NULL conversion
> here and you should pass PETSC_NULL.
> 
>    Matt
>  
>   end if
> 
>   return
>   end subroutine ShellKSPMonitor
> 
> And then in the main program I call:
> 
>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION, ierr)
> 
> As I have followed the example in the PETSc documentation, I am unsure where I have made a mistake.
> 
> Thanks, 
> 
> Randy M
> 
> 
> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Write a one line monitor that calls the true residual monitor when desired:
> 
>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
>    {
>        if (n % 10) {
>           ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>        }
>    }
> 
>    then call this on the KPS
> 
>   ierr = KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
> 
> 
> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
> 
> > In using ksp_monitor_true_residual_norm, is it possible to change how often this information is printed out?
> > That is, instead of every iteration, say I only want to see it every 10 or 20 iterations. Is there an easy way
> > to do this, other than creating my own monitor and doing it myself?
> >
> > Thanks, Randy M.
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/2aa9f48b/attachment.htm>

From mmnasr at gmail.com  Tue Apr 17 15:18:48 2012
From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani)
Date: Tue, 17 Apr 2012 13:18:48 -0700
Subject: [petsc-users] VTK viewer
Message-ID: <CAE_bc5b1=LABFH2-smhi+owXJ1rd9UY6npGQ_2YzL_RB-rhJqg@mail.gmail.com>

Hi,

I realized that in petsc-dev version a new viewer is added to write vtk
binary data in VTS format.
I was wondering if there is a way to use the same viewers and write in VTR
format for the structured orthogonal grids.
Obviously I could use VTS format to represent my data, but for big files,
that would multiply the file size by a factor of 4!

Thanks,
Mohamad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/0171f0e5/attachment-0001.htm>

From knepley at gmail.com  Tue Apr 17 15:43:01 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 17 Apr 2012 16:43:01 -0400
Subject: [petsc-users] VTK viewer
In-Reply-To: <CAE_bc5b1=LABFH2-smhi+owXJ1rd9UY6npGQ_2YzL_RB-rhJqg@mail.gmail.com>
References: <CAE_bc5b1=LABFH2-smhi+owXJ1rd9UY6npGQ_2YzL_RB-rhJqg@mail.gmail.com>
Message-ID: <CAMYG4GnqAiKkyd1PhrianDngOBk3b4RxAgLMF_EV98gO10QJQg@mail.gmail.com>

On Tue, Apr 17, 2012 at 4:18 PM, Mohamad M. Nasr-Azadani
<mmnasr at gmail.com>wrote:

> Hi,
>
> I realized that in petsc-dev version a new viewer is added to write vtk
> binary data in VTS format.
> I was wondering if there is a way to use the same viewers and write in VTR
> format for the structured orthogonal grids.
> Obviously I could use VTS format to represent my data, but for big files,
> that would multiply the file size by a factor of 4!
>

Right now, we are not going to write this. However, if you want to
contribute it, we would certainly take it.

  Thanks,

     Matt


> Thanks,
> Mohamad
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/63e8caff/attachment.htm>

From mmnasr at gmail.com  Tue Apr 17 16:33:35 2012
From: mmnasr at gmail.com (Mohamad M. Nasr-Azadani)
Date: Tue, 17 Apr 2012 14:33:35 -0700
Subject: [petsc-users] VTK viewer
In-Reply-To: <CAMYG4GnqAiKkyd1PhrianDngOBk3b4RxAgLMF_EV98gO10QJQg@mail.gmail.com>
References: <CAE_bc5b1=LABFH2-smhi+owXJ1rd9UY6npGQ_2YzL_RB-rhJqg@mail.gmail.com>
	<CAMYG4GnqAiKkyd1PhrianDngOBk3b4RxAgLMF_EV98gO10QJQg@mail.gmail.com>
Message-ID: <CAE_bc5ZuoRRuX28p=EUiBHP3Bnj0ydGfTid2ky3u7hkk1dufNQ@mail.gmail.com>

Thanks Matt,

I would like to help, but I have not contributed any codes to PETSc before.
To be honest, a little bit overwhelming for me. :-)

Mohamad


On Tue, Apr 17, 2012 at 1:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Apr 17, 2012 at 4:18 PM, Mohamad M. Nasr-Azadani <mmnasr at gmail.com
> > wrote:
>
>> Hi,
>>
>> I realized that in petsc-dev version a new viewer is added to write vtk
>> binary data in VTS format.
>> I was wondering if there is a way to use the same viewers and write in
>> VTR format for the structured orthogonal grids.
>> Obviously I could use VTS format to represent my data, but for big files,
>> that would multiply the file size by a factor of 4!
>>
>
> Right now, we are not going to write this. However, if you want to
> contribute it, we would certainly take it.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Mohamad
>>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120417/db6adb1e/attachment.htm>

From zonexo at gmail.com  Wed Apr 18 08:30:43 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 18 Apr 2012 15:30:43 +0200
Subject: [petsc-users] Any Examples for 3D Poisson Solver with both
 Dirichlet and Neumann BC's
In-Reply-To: <CADVoUZSBBAmjZuYmq95+gM1uSTNZ5wnnG_tYG4uKUz1ENn5D4A@mail.gmail.com>
References: <4F8A0F83.40300@gmail.com>
	<CADVoUZSBBAmjZuYmq95+gM1uSTNZ5wnnG_tYG4uKUz1ENn5D4A@mail.gmail.com>
Message-ID: <4F8EC203.9070700@gmail.com>

Hi,

Does anyone has experience of the performance of this solver with Hypre 
multigrid solvers? I'm now using Hypre struct solver and wonder if it is 
worth trying out other multigrid solvers.

Yours sincerely,

TAY wee-beng


On 15/4/2012 2:08 AM, Aron Ahmadia wrote:
> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex34.c.html 
> has examples both Neumann and Dirichlet BCs...
>
> A
>
> On Sun, Apr 15, 2012 at 3:00 AM, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>     Dear All,
>        Any suggestions on a 3D Poisson Solver with both Dirichlet and
>     Neumann BC's?
>        Also, does PETSc have any example of Poisson solver (either 2D
>     or 3D) with periodic boundary conditions.
>
>     thanks
>     Zhenglun
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/f0bdb98e/attachment.htm>

From christian.staudt at ira.uka.de  Wed Apr 18 11:40:30 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Wed, 18 Apr 2012 18:40:30 +0200
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
Message-ID: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>

Hello PETSc users,

I am new to PETSc (and, coming from the Java world, not an experienced C++ developer). I hope my question is appropriate on this list.

Here's my problem: I'd like to build an application in C++ against PETSc, using Eclipse CDT. However, I have problems including PETSc as a library in my Eclipse CDT project.


I have tried the following for my test project:

In Properties>>C/C++ Build>>Settings>>GCC C++ Compiler >> Includes I add the path

/usr/local/petsc/include/

In Properties>>C/C++ Build>>Settings>>MacOS X C++ Linker >> Libraries, I add 

/usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a 

to "Libraries (-l)" and

/usr/local/petsc/arch-darwin-c-debug/lib/

to "Library search path (-L)"



Now I try to build a simple Hello World program which has the lines

#include <petsc.h>
#include <petscmat.h>

at the beginning of the main file.

I get compiler errors for almost every line of the PETSc header files, indicating that basic types are not found, for example:

/usr/local/petsc/include/petscmat.h:1643: error: 'PetscScalar' has not been declared
/usr/local/petsc/include/petscmat.h:1721: error: 'PetscReal' has not been declared
/usr/local/petsc/include/petscmat.h:1152: error: 'PetscReal' does not name a type
...


Can anyone tell me what might be the problem? Thank you.

Chris




From jedbrown at mcs.anl.gov  Wed Apr 18 11:45:39 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 18 Apr 2012 09:45:39 -0700
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
Message-ID: <CAM9tzSngkV+SsPvEHLObTebYHroWF1JjsgvqMb5bLQCtziSN-Q@mail.gmail.com>

On Wed, Apr 18, 2012 at 09:40, Christian Staudt <christian.staudt at ira.uka.de
> wrote:

> In Properties>>C/C++ Build>>Settings>>GCC C++ Compiler >> Includes I add
> the path
>
> /usr/local/petsc/include/
>
> In Properties>>C/C++ Build>>Settings>>MacOS X C++ Linker >> Libraries, I
> add
>
> /usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
>
> to "Libraries (-l)" and
>
> /usr/local/petsc/arch-darwin-c-debug/lib/
>
> to "Library search path (-L)"
>
>
>
> Now I try to build a simple Hello World program which has the lines
>
> #include <petsc.h>
> #include <petscmat.h>
>
> at the beginning of the main file.
>
> I get compiler errors for almost every line of the PETSc header files,
> indicating that basic types are not found, for example:
>
> /usr/local/petsc/include/petscmat.h:1643: error: 'PetscScalar' has not
> been declared
> /usr/local/petsc/include/petscmat.h:1721: error: 'PetscReal' has not been
> declared
> /usr/local/petsc/include/petscmat.h:1152: error: 'PetscReal' does not name
> a type
>

What was the _first_ error message? Was petscconf.h found?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/46dd3c11/attachment.htm>

From knepley at gmail.com  Wed Apr 18 11:47:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 18 Apr 2012 12:47:34 -0400
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
Message-ID: <CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>

On Wed, Apr 18, 2012 at 12:40 PM, Christian Staudt <
christian.staudt at ira.uka.de> wrote:

> Hello PETSc users,
>
> I am new to PETSc (and, coming from the Java world, not an experienced C++
> developer). I hope my question is appropriate on this list.
>
> Here's my problem: I'd like to build an application in C++ against PETSc,
> using Eclipse CDT. However, I have problems including PETSc as a library in
> my Eclipse CDT project.
>
>
> I have tried the following for my test project:
>
> In Properties>>C/C++ Build>>Settings>>GCC C++ Compiler >> Includes I add
> the path
>
> /usr/local/petsc/include/
>
> In Properties>>C/C++ Build>>Settings>>MacOS X C++ Linker >> Libraries, I
> add
>
> /usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
>
> to "Libraries (-l)" and
>
> /usr/local/petsc/arch-darwin-c-debug/lib/
>
> to "Library search path (-L)"
>
>
>
> Now I try to build a simple Hello World program which has the lines
>
> #include <petsc.h>
> #include <petscmat.h>
>
> at the beginning of the main file.
>
> I get compiler errors for almost every line of the PETSc header files,
> indicating that basic types are not found, for example:
>

I guarantee you there is a "Header not found" error up at the top that you
cut off. ALWAYS send the full error message.

   Matt


> /usr/local/petsc/include/petscmat.h:1643: error: 'PetscScalar' has not
> been declared
> /usr/local/petsc/include/petscmat.h:1721: error: 'PetscReal' has not been
> declared
> /usr/local/petsc/include/petscmat.h:1152: error: 'PetscReal' does not name
> a type
> ...
>
>
> Can anyone tell me what might be the problem? Thank you.
>
> Chris
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/6f295b33/attachment.htm>

From christian.staudt at ira.uka.de  Wed Apr 18 12:14:56 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Wed, 18 Apr 2012 19:14:56 +0200
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
	<CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
Message-ID: <4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>


Am 18.04.2012 um 18:47 schrieb Matthew Knepley:

> I guarantee you there is a "Header not found" error up at the top that you cut off. ALWAYS send the full error message.
> 
>    Matt


You're right, Eclipse Console truncated the output - sorry. Here's the full error message: [1]


> Was petscconf.h found?

/usr/local/petsc/include/petscsys.h:13:23: error: petscconf.h: No such file or directory

petscconf.h was not found. What is petscconf.h and what can I do to fix this?

Thank you

Chris


[1] http://pastebin.com/e7cqVvKg


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/98ebda21/attachment.htm>

From knepley at gmail.com  Wed Apr 18 12:17:59 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 18 Apr 2012 13:17:59 -0400
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
	<CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
	<4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>
Message-ID: <CAMYG4GkYa=ZMGJeXuQnha3Wdech1LbaNXg8=Q_tP_DpQUt8hYQ@mail.gmail.com>

On Wed, Apr 18, 2012 at 1:14 PM, Christian Staudt <
christian.staudt at ira.uka.de> wrote:

>
> Am 18.04.2012 um 18:47 schrieb Matthew Knepley:
>
> I guarantee you there is a "Header not found" error up at the top that you
> cut off. ALWAYS send the full error message.
>
>    Matt
>
>
>
> You're right, Eclipse Console truncated the output - sorry. Here's the
> full error message: [1]
>
>
> Was petscconf.h found?
>
>
The include directories can be obtained using:

  make getincludedirs

and usually amount to

  -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include

but can be modified by configure for certain packages.

   Matt


> /usr/local/petsc/include/petscsys.h:13:23: error: petscconf.h: No such
> file or directory
>
> petscconf.h was not found. What is petscconf.h and what can I do to fix
> this?
>
> Thank you
>
> Chris
>
>
> [1] http://pastebin.com/e7cqVvKg
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/facb5a2d/attachment.htm>

From christian.staudt at ira.uka.de  Wed Apr 18 12:46:25 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Wed, 18 Apr 2012 19:46:25 +0200
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <CAMYG4GkYa=ZMGJeXuQnha3Wdech1LbaNXg8=Q_tP_DpQUt8hYQ@mail.gmail.com>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
	<CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
	<4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>
	<CAMYG4GkYa=ZMGJeXuQnha3Wdech1LbaNXg8=Q_tP_DpQUt8hYQ@mail.gmail.com>
Message-ID: <111F0318-022A-40A0-8183-10E8F1F20541@ira.uka.de>


Am 18.04.2012 um 19:17 schrieb Matthew Knepley:

> The include directories can be obtained using:
> 
>   make getincludedirs
> 
> and usually amount to
> 
>   -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include
> 
> but can be modified by configure for certain packages.
> 
>    Matt



	make getincludedirs 

failed for me with:

cls /usr/local/petsc $ make getincludedirs
conf/petscvariables:3: /usr/local/petsc-3.2-p6/arch-darwin-c-debug/conf/petscvariables: No such file or directory
makefile:15: /usr/local/petsc-3.2-p6/conf/variables: No such file or directory
makefile:16: /usr/local/petsc-3.2-p6/conf/rules: No such file or directory
makefile:17: /usr/local/petsc-3.2-p6/conf/test: No such file or directory
make: *** No rule to make target `/usr/local/petsc-3.2-p6/conf/test'.  Stop.

However, I simply added 

  ${PETSC_DIR}/${PETSC_ARCH}/include

to the includes for my Eclipse project, which fixes the error that petscconf.h was not found.

Now the following error appears:

Building target: PETScTest2
Invoking: MacOS X C++ Linker
g++ -L/usr/local/petsc/arch-darwin-c-debug/lib/ -o "PETScTest2"  ./src/PETScTest2.o   -l/usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
ld: library not found for -l/usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
collect2: ld returned 1 exit status
make: *** [PETScTest2] Error 1

libpetsc.a definitely exists at this location.  Is my PETSc installation broken?

Chris


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/1f7e4576/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 18 12:49:00 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 18 Apr 2012 10:49:00 -0700
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <111F0318-022A-40A0-8183-10E8F1F20541@ira.uka.de>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
	<CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
	<4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>
	<CAMYG4GkYa=ZMGJeXuQnha3Wdech1LbaNXg8=Q_tP_DpQUt8hYQ@mail.gmail.com>
	<111F0318-022A-40A0-8183-10E8F1F20541@ira.uka.de>
Message-ID: <CAM9tzSnDNyMJYbtfux556JRON5c28+1Kj-=vB5J-7o_KdHR2fw@mail.gmail.com>

On Wed, Apr 18, 2012 at 10:46, Christian Staudt <christian.staudt at ira.uka.de
> wrote:

> to the includes for my Eclipse project, which fixes the error that
> petscconf.h was not found.
>
> Now the following error appears:
>
> Building target: PETScTest2
> Invoking: MacOS X C++ Linker
> g++ -L/usr/local/petsc/arch-darwin-c-debug/lib/ -o "PETScTest2"
> ./src/PETScTest2.o   -l/usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
> ld: library not found for
> -l/usr/local/petsc/arch-darwin-c-debug/lib/libpetsc.a
> collect2: ld returned 1 exit status
> make: *** [PETScTest2] Error 1
>
> libpetsc.a definitely exists at this location.  Is my PETSc installation
> broken?
>

This is not the right option with which to link a library, you can do it
this way.

-L/usr/local/petsc/arch-darwin-c-debug/lib/ -lpetsc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/08bdf61d/attachment.htm>

From christian.staudt at ira.uka.de  Wed Apr 18 12:58:48 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Wed, 18 Apr 2012 19:58:48 +0200
Subject: [petsc-users] How to include PETSc in an Eclipse CDT project
In-Reply-To: <CAM9tzSnDNyMJYbtfux556JRON5c28+1Kj-=vB5J-7o_KdHR2fw@mail.gmail.com>
References: <A3DB3CA4-CD53-4BB1-8EEA-AA4C0DA73260@ira.uka.de>
	<CAMYG4GkFv6Ntoh9yRfz5ixx2Cru3C6WP4FY22-n0ieK+raBUeg@mail.gmail.com>
	<4B9F30FB-4D3B-414C-A706-13033E0A462B@ira.uka.de>
	<CAMYG4GkYa=ZMGJeXuQnha3Wdech1LbaNXg8=Q_tP_DpQUt8hYQ@mail.gmail.com>
	<111F0318-022A-40A0-8183-10E8F1F20541@ira.uka.de>
	<CAM9tzSnDNyMJYbtfux556JRON5c28+1Kj-=vB5J-7o_KdHR2fw@mail.gmail.com>
Message-ID: <0A942570-F003-46FB-9617-D460B211CB1A@ira.uka.de>


Am 18.04.2012 um 19:49 schrieb Jed Brown:

> This is not the right option with which to link a library, you can do it this way.
> 
> -L/usr/local/petsc/arch-darwin-c-debug/lib/ -lpetsc


Thanks for pointing out the error. Now I can include PETSc in my project.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/f1308218/attachment.htm>

From zhenglun.wei at gmail.com  Wed Apr 18 14:52:30 2012
From: zhenglun.wei at gmail.com (Alan Wei)
Date: Wed, 18 Apr 2012 14:52:30 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
Message-ID: <CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>

Dear all,
    I hope you're having a nice day. I have a further question on this
issue in 3D.
1, Following the idea of Dr. Brown and Dr. Knepley, I finished a 2D test,
which works very fine. Here, I did it in 3D by
"
    TESTVAR ***a, ***b, ***c;
    TESTVAR **aa, **bb, **cc;
    TESTVAR *arraya, *arrayb, *arrayc;

    arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
    arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
    arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));

    aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
    bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
    cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));

    for(i = 0; i < SIZE*SIZE; i++) {
      aa[i] = &arraya[i*SIZE];
      bb[i] = &arrayb[i*SIZE];
      cc[i] = &arrayc[i*SIZE];
    }

    a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
    b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
    c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));

    for(i = 0; i < SIZE; i++) {
      a[i] = &aa[i*SIZE];
      b[i] = &bb[i*SIZE];
      c[i] = &cc[i*SIZE];
    }
"
  It works. However, I wonder if there is any other good ideas for 3D
problem other than this kinda of 'two-layer' approach.

2, I have a little question on PETSc about 3D processor ordering. Does
PETSc have any function giving me the nodes/rank number of neighboring
nodes/ranks? Are those 'Application Ordering' functions applicable for my
case?

thanks,
Alan

On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> > wrote:
>
>>     I have a final question on it. Is it taken a lot of memory for doing
>> this? As I understand, pointers won't occupy many memories and it works
>> like an alias. It will not, to my limit knowledge, take much extra memory
>> by doing this.
>
>
> A pointer takes about as much space as a floating point value, so that
> array of pointers costs about 1*N compared to the N*N matrix.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/683f1034/attachment-0001.htm>

From kevin.richard.green at gmail.com  Wed Apr 18 16:26:32 2012
From: kevin.richard.green at gmail.com (Kevin Green)
Date: Wed, 18 Apr 2012 17:26:32 -0400
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
Message-ID: <CAObUYLPknATNUG985Kqqs4MMifJMXT-k72R+PrtetC1L74ODYQ@mail.gmail.com>

If I could just wedge myself in here for a second... I recall fiddling with
a 3d contiguous allocation problem a while back.  My solution is
essentially the same, but the pointers to pointers are allocated in the
space before the array data... Nice thing is, you only need  double ***M
(or whatever type you want, I think I even have a templated version sitting
around somewhere...) outside of the function.

double*** contiguous3(int icount,
              int jcount,
              int kcount)
// Generates a contiguously allocated (row major) array for 3 dimensions.
// Array can be accessed as M[i][j][k], and also taking
// m = &M[0][0][0] (for pointer m) allows m[i]
// Allocated memory can be freed with a single call to free(M)
{
  int ii, jj;
  int typeSize = sizeof(double);
  void*** iret = (void***)malloc(icount*sizeof(void***) +
                 icount*jcount*sizeof(void**) +
                 icount*jcount*kcount*typeSize);
  void** jret = (void**)(iret + icount);
  char* kret = (char*)(jret + icount*jcount);
  for(ii=0;ii<icount;++ii){
    iret[ii] = &jret[ii*jcount];
  }
  for(ii=0;ii<icount;++ii){
    for(jj=0;jj<jcount;++jj){
      jret[ii*jcount+jj] = &kret[ii*jcount*kcount*typeSize +
                 jj*kcount*typeSize];
    }
  }
  return (double***)iret;
}

Cheers,
Kevin



On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com> wrote:

> Dear all,
>     I hope you're having a nice day. I have a further question on this
> issue in 3D.
> 1, Following the idea of Dr. Brown and Dr. Knepley, I finished a 2D test,
> which works very fine. Here, I did it in 3D by
> "
>     TESTVAR ***a, ***b, ***c;
>     TESTVAR **aa, **bb, **cc;
>     TESTVAR *arraya, *arrayb, *arrayc;
>
>     arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>
>     aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>
>     for(i = 0; i < SIZE*SIZE; i++) {
>       aa[i] = &arraya[i*SIZE];
>       bb[i] = &arrayb[i*SIZE];
>       cc[i] = &arrayc[i*SIZE];
>     }
>
>     a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>
>     for(i = 0; i < SIZE; i++) {
>       a[i] = &aa[i*SIZE];
>       b[i] = &bb[i*SIZE];
>       c[i] = &cc[i*SIZE];
>     }
> "
>   It works. However, I wonder if there is any other good ideas for 3D
> problem other than this kinda of 'two-layer' approach.
>
> 2, I have a little question on PETSc about 3D processor ordering. Does
> PETSc have any function giving me the nodes/rank number of neighboring
> nodes/ranks? Are those 'Application Ordering' functions applicable for my
> case?
>
> thanks,
> Alan
>
> On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei <
>> zhenglun.wei at gmail.com> wrote:
>>
>>>     I have a final question on it. Is it taken a lot of memory for doing
>>> this? As I understand, pointers won't occupy many memories and it works
>>> like an alias. It will not, to my limit knowledge, take much extra memory
>>> by doing this.
>>
>>
>> A pointer takes about as much space as a floating point value, so that
>> array of pointers costs about 1*N compared to the N*N matrix.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120418/8c2aee5f/attachment.htm>

From knepley at gmail.com  Thu Apr 19 04:52:46 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Apr 2012 05:52:46 -0400
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
Message-ID: <CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>

On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com> wrote:

> Dear all,
>     I hope you're having a nice day. I have a further question on this
> issue in 3D.
> 1, Following the idea of Dr. Brown and Dr. Knepley, I finished a 2D test,
> which works very fine. Here, I did it in 3D by
> "
>     TESTVAR ***a, ***b, ***c;
>     TESTVAR **aa, **bb, **cc;
>     TESTVAR *arraya, *arrayb, *arrayc;
>
>     arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>
>     aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>
>     for(i = 0; i < SIZE*SIZE; i++) {
>       aa[i] = &arraya[i*SIZE];
>       bb[i] = &arrayb[i*SIZE];
>       cc[i] = &arrayc[i*SIZE];
>     }
>
>     a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>
>     for(i = 0; i < SIZE; i++) {
>       a[i] = &aa[i*SIZE];
>       b[i] = &bb[i*SIZE];
>       c[i] = &cc[i*SIZE];
>     }
> "
>   It works. However, I wonder if there is any other good ideas for 3D
> problem other than this kinda of 'two-layer' approach.
>

What is the reason for not using DMDA?


> 2, I have a little question on PETSc about 3D processor ordering. Does
> PETSc have any function giving me the nodes/rank number of neighboring
> nodes/ranks? Are those 'Application Ordering' functions applicable for my
> case?
>

What do you mean by neighboring? If it is jsut stencil neighbors, then use
a local vector.

   Matt


> thanks,
> Alan
>
> On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei <
>> zhenglun.wei at gmail.com> wrote:
>>
>>>     I have a final question on it. Is it taken a lot of memory for doing
>>> this? As I understand, pointers won't occupy many memories and it works
>>> like an alias. It will not, to my limit knowledge, take much extra memory
>>> by doing this.
>>
>>
>> A pointer takes about as much space as a floating point value, so that
>> array of pointers costs about 1*N compared to the N*N matrix.
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/328d61f9/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr 19 05:07:46 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 19 Apr 2012 12:07:46 +0200
Subject: [petsc-users] Separate MatAssemblyBegin/MatAssemblyEnd
Message-ID: <4F8FE3F2.906@gfz-potsdam.de>

I'm wondering what is the main idea behind making 
XXXAssemblyBegin/XXXAssemblyEnd separate routines?
I've never used and seen any example where these routines don't follow 
each other.

-- 
Regards,
Alexander


From knepley at gmail.com  Thu Apr 19 05:10:23 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Apr 2012 06:10:23 -0400
Subject: [petsc-users] Separate MatAssemblyBegin/MatAssemblyEnd
In-Reply-To: <4F8FE3F2.906@gfz-potsdam.de>
References: <4F8FE3F2.906@gfz-potsdam.de>
Message-ID: <CAMYG4G=X+gyjwJ7QvAsYNaEC+49-jDQMKpEw1NdX4ExnU4OvLQ@mail.gmail.com>

On Thu, Apr 19, 2012 at 6:07 AM, Alexander Grayver
<agrayver at gfz-potsdam.de>wrote:

> I'm wondering what is the main idea behind making XXXAssemblyBegin/**XXXAssemblyEnd
> separate routines?
> I've never used and seen any example where these routines don't follow
> each other.


You can overlap communication and computation by putting flops in between
these calls.

   Matt

-- 
> Regards,
> Alexander
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/d19685e0/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr 19 05:46:38 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 19 Apr 2012 12:46:38 +0200
Subject: [petsc-users] Separate MatAssemblyBegin/MatAssemblyEnd
In-Reply-To: <CAMYG4G=X+gyjwJ7QvAsYNaEC+49-jDQMKpEw1NdX4ExnU4OvLQ@mail.gmail.com>
References: <4F8FE3F2.906@gfz-potsdam.de>
	<CAMYG4G=X+gyjwJ7QvAsYNaEC+49-jDQMKpEw1NdX4ExnU4OvLQ@mail.gmail.com>
Message-ID: <4F8FED0E.609@gfz-potsdam.de>

Is it used somewhere in PETSc?

On 19.04.2012 12:10, Matthew Knepley wrote:
> On Thu, Apr 19, 2012 at 6:07 AM, Alexander Grayver 
> <agrayver at gfz-potsdam.de <mailto:agrayver at gfz-potsdam.de>> wrote:
>
>     I'm wondering what is the main idea behind making
>     XXXAssemblyBegin/XXXAssemblyEnd separate routines?
>     I've never used and seen any example where these routines don't
>     follow each other.
>
>
> You can overlap communication and computation by putting flops in 
> between these calls.
>
>    Matt
>
>     -- 
>     Regards,
>     Alexander
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
Regards,
Alexander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/7d78ea43/attachment.htm>

From knepley at gmail.com  Thu Apr 19 05:52:49 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Apr 2012 06:52:49 -0400
Subject: [petsc-users] Separate MatAssemblyBegin/MatAssemblyEnd
In-Reply-To: <4F8FED0E.609@gfz-potsdam.de>
References: <4F8FE3F2.906@gfz-potsdam.de>
	<CAMYG4G=X+gyjwJ7QvAsYNaEC+49-jDQMKpEw1NdX4ExnU4OvLQ@mail.gmail.com>
	<4F8FED0E.609@gfz-potsdam.de>
Message-ID: <CAMYG4G=jvvk0RRYHE7k7cSdMArFAXRus5x6vLqzfXSTC3KNZrQ@mail.gmail.com>

On Thu, Apr 19, 2012 at 6:46 AM, Alexander Grayver
<agrayver at gfz-potsdam.de>wrote:

> **
> Is it used somewhere in PETSc?
>

PETSc is the library. Assembly happens in the application.

   Matt


> On 19.04.2012 12:10, Matthew Knepley wrote:
>
> On Thu, Apr 19, 2012 at 6:07 AM, Alexander Grayver <
> agrayver at gfz-potsdam.de> wrote:
>
>> I'm wondering what is the main idea behind making
>> XXXAssemblyBegin/XXXAssemblyEnd separate routines?
>> I've never used and seen any example where these routines don't follow
>> each other.
>
>
>  You can overlap communication and computation by putting flops in
> between these calls.
>
>     Matt
>
>  --
>> Regards,
>> Alexander
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> Regards,
> Alexander
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/6bb23a0c/attachment-0001.htm>

From agrayver at gfz-potsdam.de  Thu Apr 19 08:25:26 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 19 Apr 2012 15:25:26 +0200
Subject: [petsc-users] PetscBinaryRead in matlab
Message-ID: <4F901246.9030206@gfz-potsdam.de>

Trying to read double complex matrix output from PETSc using matlab 
function PetscBinaryRead I encountered a problem that is related to the 
rounding problem inside linspace function:

These are sizes of a matrix:
n = 1776;
m = 54400;

Trying to execute the code:
iReal = linspace(1,n*m*2-1,n*m);
iImag = iReal +1;

% check if all elements are integers, otherwise matlab can't use them as 
indices:
aa=ceil(iReal) == floor(iReal);
size(find(aa==0))

I get:

ans =

            1     1693033

That means there 1693033 in iReal array that are viewed as real numbers 
by matlab and can't use further as indices:

??? Subscript indices must either be real positive integers or logicals.

Error in ==> PetscBinaryRead at 92
         A     = complex(reshape(s(iReal),n,m)',reshape(s(iImag),n,m)') ;

  Simple workaround that worked for me was just use floor function.

-- 
Regards,
Alexander


From jedbrown at mcs.anl.gov  Thu Apr 19 08:39:48 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 06:39:48 -0700
Subject: [petsc-users] PetscBinaryRead in matlab
In-Reply-To: <4F901246.9030206@gfz-potsdam.de>
References: <4F901246.9030206@gfz-potsdam.de>
Message-ID: <CAM9tzSkyU2e_yz8942Z66+Bu_Oc=1fTc4YJU7j72LeZBgJ8Qqw@mail.gmail.com>

On Thu, Apr 19, 2012 at 06:25, Alexander Grayver <agrayver at gfz-potsdam.de>wrote:

> That means there 1693033 in iReal array that are viewed as real numbers by
> matlab and can't use further as indices:
>
> ??? Subscript indices must either be real positive integers or logicals.
>
> Error in ==> PetscBinaryRead at 92
>        A     = complex(reshape(s(iReal),n,m)'**,reshape(s(iImag),n,m)') ;
>
>  Simple workaround that worked for me was just use floor function.
>

Does replacing linspace with (1:2:2*m*n-1) work without floor?

There isn't a guarantee that whatever rounding error committed by linspace
always rounds away from zero (so that floor sends it in the right
direction).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/cdbd3b18/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr 19 08:41:42 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Thu, 19 Apr 2012 15:41:42 +0200
Subject: [petsc-users] PetscBinaryRead in matlab
In-Reply-To: <CAM9tzSkyU2e_yz8942Z66+Bu_Oc=1fTc4YJU7j72LeZBgJ8Qqw@mail.gmail.com>
References: <4F901246.9030206@gfz-potsdam.de>
	<CAM9tzSkyU2e_yz8942Z66+Bu_Oc=1fTc4YJU7j72LeZBgJ8Qqw@mail.gmail.com>
Message-ID: <4F901616.4020403@gfz-potsdam.de>

On 19.04.2012 15:39, Jed Brown wrote:
> Does replacing linspace with (1:2:2*m*n-1) work without floor?
>
> There isn't a guarantee that whatever rounding error committed by 
> linspace always rounds away from zero (so that floor sends it in the 
> right direction).

You're right, it is undefined. Your variant works.

-- 
Regards,
Alexander


From zhenglun.wei at gmail.com  Thu Apr 19 09:18:45 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 19 Apr 2012 09:18:45 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
	<CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
Message-ID: <4F901EC5.5050307@gmail.com>

"
     TESTVAR ***a, ***b, ***c;
     TESTVAR **aa, **bb, **cc;
     TESTVAR *arraya, *arrayb, *arrayc;

     arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));

     aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
     for(i = 0; i < SIZE*SIZE; i++) {
       aa[i] = &arraya[i*SIZE];
       bb[i] = &arrayb[i*SIZE];
       cc[i] = &arrayc[i*SIZE];
     }

     a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
     for(i = 0; i < SIZE; i++) {
       a[i] = &aa[i*SIZE];
       b[i] = &bb[i*SIZE];
       c[i] = &cc[i*SIZE];
     }
"
   It works. However, I wonder if there is any other good ideas for 3D 
problem other than this kinda of 'two-layer' approach.

*_What is the reason for not using DMDA?_
*In 2D, I established a 2D array for data communication between nodes by 
using MPI derived data type. It allows me to easily communicate both 
contiguous (i.e. MPI_TYPE_CONTIGUOUS) and non-contiguous (i.e. 
MPI_TYPE_VECTOR) data. That is why I use this similar approach in 3D, 
though an additional data type, i.e. MPI_TYPE_INDEXED, need to be used. 
Does DMDA have those type of function or derived data type?

    "2, I have a little question on PETSc about 3D processor ordering.
    Does PETSc have any function giving me the nodes/rank number of
    neighboring nodes/ranks? Are those 'Application Ordering' functions
    applicable for my case?"


_*What do you mean by neighboring? If it is jsut stencil neighbors, then 
use a local vector.*_
When I send and receive data with MPI_Send and MPI_RECV, I need provide 
the 'destination' (in MPI_Send refer 
to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html') 
and 'source' (in MPI_RECV refer 
to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html'). In 
a 2D problem with Cartesian grid, 4 processes divide the whole domain to 
4 sub-domain.
----------------------------
       2      |      3      |
----------------------------
       0      |      1      |
---------------------------
  Then, for node 1, the neighboring nodes are '0' and '3', which '0' is 
the left node and '3' is the top node. I wonder if PETSc has any 
function that I can call to obtain those neighboring nodes so that I do 
not need to construct my function.

I'm sorry for confusing you.

thanks in advance,
Alan

On 4/19/2012 4:52 AM, Matthew Knepley wrote:
> On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com 
> <mailto:zhenglun.wei at gmail.com>> wrote:
>
>     Dear all,
>         I hope you're having a nice day. I have a further question on
>     this issue in 3D.
>     1, Following the idea of Dr. Brown and Dr. Knepley, I finished a
>     2D test, which works very fine. Here, I did it in 3D by
>     "
>         TESTVAR ***a, ***b, ***c;
>         TESTVAR **aa, **bb, **cc;
>         TESTVAR *arraya, *arrayb, *arrayc;
>
>         arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>         arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>         arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>
>         aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         for(i = 0; i < SIZE*SIZE; i++) {
>           aa[i] = &arraya[i*SIZE];
>           bb[i] = &arrayb[i*SIZE];
>           cc[i] = &arrayc[i*SIZE];
>         }
>
>         a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         for(i = 0; i < SIZE; i++) {
>           a[i] = &aa[i*SIZE];
>           b[i] = &bb[i*SIZE];
>           c[i] = &cc[i*SIZE];
>         }
>     "
>       It works. However, I wonder if there is any other good ideas for
>     3D problem other than this kinda of 'two-layer' approach.
>
>
> What is the reason for not using DMDA?
>
>     2, I have a little question on PETSc about 3D processor ordering.
>     Does PETSc have any function giving me the nodes/rank number of
>     neighboring nodes/ranks? Are those 'Application Ordering'
>     functions applicable for my case?
>
>
> What do you mean by neighboring? If it is jsut stencil neighbors, then 
> use a local vector.
>
>    Matt
>
>     thanks,
>     Alan
>
>     On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov
>     <mailto:jedbrown at mcs.anl.gov>> wrote:
>
>         On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei
>         <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>                 I have a final question on it. Is it taken a lot of
>             memory for doing this? As I understand, pointers won't
>             occupy many memories and it works like an alias. It will
>             not, to my limit knowledge, take much extra memory by
>             doing this. 
>
>
>         A pointer takes about as much space as a floating point value,
>         so that array of pointers costs about 1*N compared to the N*N
>         matrix.
>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/c9669ec4/attachment.htm>

From knepley at gmail.com  Thu Apr 19 09:35:23 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Apr 2012 10:35:23 -0400
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F901EC5.5050307@gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
	<CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
	<4F901EC5.5050307@gmail.com>
Message-ID: <CAMYG4GmMBoTo2igMhwZOhYUsi9w6Nn+RW1Ss5cSOuhspMhxvNg@mail.gmail.com>

On Thu, Apr 19, 2012 at 10:18 AM, Zhenglun (Alan) Wei <
zhenglun.wei at gmail.com> wrote:

>   "
>     TESTVAR ***a, ***b, ***c;
>     TESTVAR **aa, **bb, **cc;
>     TESTVAR *arraya, *arrayb, *arrayc;
>
>      arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>
>      aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>
>     for(i = 0; i < SIZE*SIZE; i++) {
>       aa[i] = &arraya[i*SIZE];
>       bb[i] = &arrayb[i*SIZE];
>       cc[i] = &arrayc[i*SIZE];
>     }
>
>      a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>
>     for(i = 0; i < SIZE; i++) {
>       a[i] = &aa[i*SIZE];
>       b[i] = &bb[i*SIZE];
>       c[i] = &cc[i*SIZE];
>     }
> "
>   It works. However, I wonder if there is any other good ideas for 3D
> problem other than this kinda of 'two-layer' approach.
>
>  *What is the reason for not using DMDA?
> *In 2D, I established a 2D array for data communication between nodes by
> using MPI derived data type. It allows me to easily communicate both
> contiguous (i.e. MPI_TYPE_CONTIGUOUS) and non-contiguous (i.e.
> MPI_TYPE_VECTOR) data. That is why I use this similar approach in 3D,
> though an additional data type, i.e. MPI_TYPE_INDEXED, need to be used.
> Does DMDA have those type of function or derived data type?
>

It definitely does communication between the local pieces. Do you want
something else?


>  "2, I have a little question on PETSc about 3D processor ordering. Does
>> PETSc have any function giving me the nodes/rank number of neighboring
>> nodes/ranks? Are those 'Application Ordering' functions applicable for my
>> case?"
>>
>
>  *What do you mean by neighboring? If it is jsut stencil neighbors, then
> use a local vector.*
> When I send and receive data with MPI_Send and MPI_RECV, I need provide
> the 'destination' (in MPI_Send refer to'
> http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html') and
> 'source' (in MPI_RECV refer to'
> http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html'). In
> a 2D problem with Cartesian grid, 4 processes divide the whole domain to 4
> sub-domain.
> ----------------------------
>       2      |      3      |
> ----------------------------
>       0      |      1      |
> ---------------------------
>  Then, for node 1, the neighboring nodes are '0' and '3', which '0' is the
> left node and '3' is the top node. I wonder if PETSc has any function that
> I can call to obtain those neighboring nodes so that I do not need to
> construct my function.
>

Yes, it looks like you should just use a DMDA. See the manual section.

   Matt


> I'm sorry for confusing you.
>
> thanks in advance,
> Alan
>
> On 4/19/2012 4:52 AM, Matthew Knepley wrote:
>
> On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com> wrote:
>
>> Dear all,
>>     I hope you're having a nice day. I have a further question on this
>> issue in 3D.
>> 1, Following the idea of Dr. Brown and Dr. Knepley, I finished a 2D test,
>> which works very fine. Here, I did it in 3D by
>>  "
>>     TESTVAR ***a, ***b, ***c;
>>     TESTVAR **aa, **bb, **cc;
>>     TESTVAR *arraya, *arrayb, *arrayc;
>>
>>      arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>
>>      aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>
>>     for(i = 0; i < SIZE*SIZE; i++) {
>>       aa[i] = &arraya[i*SIZE];
>>       bb[i] = &arrayb[i*SIZE];
>>       cc[i] = &arrayc[i*SIZE];
>>     }
>>
>>      a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>
>>     for(i = 0; i < SIZE; i++) {
>>       a[i] = &aa[i*SIZE];
>>       b[i] = &bb[i*SIZE];
>>       c[i] = &cc[i*SIZE];
>>     }
>> "
>>   It works. However, I wonder if there is any other good ideas for 3D
>> problem other than this kinda of 'two-layer' approach.
>>
>
>  What is the reason for not using DMDA?
>
>
>>  2, I have a little question on PETSc about 3D processor ordering. Does
>> PETSc have any function giving me the nodes/rank number of neighboring
>> nodes/ranks? Are those 'Application Ordering' functions applicable for my
>> case?
>>
>
>  What do you mean by neighboring? If it is jsut stencil neighbors, then
> use a local vector.
>
>     Matt
>
>
>>  thanks,
>> Alan
>>
>> On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>>  On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei <
>>> zhenglun.wei at gmail.com> wrote:
>>>
>>>>     I have a final question on it. Is it taken a lot of memory for
>>>> doing this? As I understand, pointers won't occupy many memories and it
>>>> works like an alias. It will not, to my limit knowledge, take much extra
>>>> memory by doing this.
>>>
>>>
>>>  A pointer takes about as much space as a floating point value, so that
>>> array of pointers costs about 1*N compared to the N*N matrix.
>>>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/afc3f27f/attachment-0001.htm>

From zhenglun.wei at gmail.com  Thu Apr 19 09:49:06 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 19 Apr 2012 09:49:06 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAMYG4GmMBoTo2igMhwZOhYUsi9w6Nn+RW1Ss5cSOuhspMhxvNg@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
	<CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
	<4F901EC5.5050307@gmail.com>
	<CAMYG4GmMBoTo2igMhwZOhYUsi9w6Nn+RW1Ss5cSOuhspMhxvNg@mail.gmail.com>
Message-ID: <4F9025E2.6010600@gmail.com>

Dear Dr. Knepley,
      It is very nice to hear that. I will read the manual. Do we have 
any examples showing its functions?

thank you so much,
Alan
On 4/19/2012 9:35 AM, Matthew Knepley wrote:
> On Thu, Apr 19, 2012 at 10:18 AM, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>     "
>         TESTVAR ***a, ***b, ***c;
>         TESTVAR **aa, **bb, **cc;
>         TESTVAR *arraya, *arrayb, *arrayc;
>
>         arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>         arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>         arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>
>         aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>         for(i = 0; i < SIZE*SIZE; i++) {
>           aa[i] = &arraya[i*SIZE];
>           bb[i] = &arrayb[i*SIZE];
>           cc[i] = &arrayc[i*SIZE];
>         }
>
>         a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>         for(i = 0; i < SIZE; i++) {
>           a[i] = &aa[i*SIZE];
>           b[i] = &bb[i*SIZE];
>           c[i] = &cc[i*SIZE];
>         }
>     "
>       It works. However, I wonder if there is any other good ideas for
>     3D problem other than this kinda of 'two-layer' approach.
>
>     *_What is the reason for not using DMDA?_
>     *In 2D, I established a 2D array for data communication between
>     nodes by using MPI derived data type. It allows me to easily
>     communicate both contiguous (i.e. MPI_TYPE_CONTIGUOUS) and
>     non-contiguous (i.e. MPI_TYPE_VECTOR) data. That is why I use this
>     similar approach in 3D, though an additional data type, i.e.
>     MPI_TYPE_INDEXED, need to be used. Does DMDA have those type of
>     function or derived data type?
>
>
> It definitely does communication between the local pieces. Do you want 
> something else?
>
>         "2, I have a little question on PETSc about 3D processor
>         ordering. Does PETSc have any function giving me the
>         nodes/rank number of neighboring nodes/ranks? Are those
>         'Application Ordering' functions applicable for my case?"
>
>
>     _*What do you mean by neighboring? If it is jsut stencil
>     neighbors, then use a local vector.*_
>     When I send and receive data with MPI_Send and MPI_RECV, I need
>     provide the 'destination' (in MPI_Send refer
>     to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html')
>     and 'source' (in MPI_RECV refer
>     to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html').
>     In a 2D problem with Cartesian grid, 4 processes divide the whole
>     domain to 4 sub-domain.
>     ----------------------------
>           2      |      3      |
>     ----------------------------
>           0      |      1      |
>     ---------------------------
>      Then, for node 1, the neighboring nodes are '0' and '3', which
>     '0' is the left node and '3' is the top node. I wonder if PETSc
>     has any function that I can call to obtain those neighboring nodes
>     so that I do not need to construct my function.
>
>
> Yes, it looks like you should just use a DMDA. See the manual section.
>
>    Matt
>
>     I'm sorry for confusing you.
>
>     thanks in advance,
>     Alan
>
>     On 4/19/2012 4:52 AM, Matthew Knepley wrote:
>>     On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com
>>     <mailto:zhenglun.wei at gmail.com>> wrote:
>>
>>         Dear all,
>>             I hope you're having a nice day. I have a further
>>         question on this issue in 3D.
>>         1, Following the idea of Dr. Brown and Dr. Knepley, I
>>         finished a 2D test, which works very fine. Here, I did it in
>>         3D by
>>         "
>>             TESTVAR ***a, ***b, ***c;
>>             TESTVAR **aa, **bb, **cc;
>>             TESTVAR *arraya, *arrayb, *arrayc;
>>
>>             arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>             arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>             arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>
>>             aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             for(i = 0; i < SIZE*SIZE; i++) {
>>               aa[i] = &arraya[i*SIZE];
>>               bb[i] = &arrayb[i*SIZE];
>>               cc[i] = &arrayc[i*SIZE];
>>             }
>>
>>             a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             for(i = 0; i < SIZE; i++) {
>>               a[i] = &aa[i*SIZE];
>>               b[i] = &bb[i*SIZE];
>>               c[i] = &cc[i*SIZE];
>>             }
>>         "
>>           It works. However, I wonder if there is any other good
>>         ideas for 3D problem other than this kinda of 'two-layer'
>>         approach.
>>
>>
>>     What is the reason for not using DMDA?
>>
>>         2, I have a little question on PETSc about 3D processor
>>         ordering. Does PETSc have any function giving me the
>>         nodes/rank number of neighboring nodes/ranks? Are those
>>         'Application Ordering' functions applicable for my case?
>>
>>
>>     What do you mean by neighboring? If it is jsut stencil neighbors,
>>     then use a local vector.
>>
>>        Matt
>>
>>         thanks,
>>         Alan
>>
>>         On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown
>>         <jedbrown at mcs.anl.gov <mailto:jedbrown at mcs.anl.gov>> wrote:
>>
>>             On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei
>>             <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>>
>>             wrote:
>>
>>                     I have a final question on it. Is it taken a lot
>>                 of memory for doing this? As I understand, pointers
>>                 won't occupy many memories and it works like an
>>                 alias. It will not, to my limit knowledge, take much
>>                 extra memory by doing this. 
>>
>>
>>             A pointer takes about as much space as a floating point
>>             value, so that array of pointers costs about 1*N compared
>>             to the N*N matrix.
>>
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/bd48cf91/attachment.htm>

From knepley at gmail.com  Thu Apr 19 09:51:07 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 19 Apr 2012 10:51:07 -0400
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <4F9025E2.6010600@gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
	<CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
	<4F901EC5.5050307@gmail.com>
	<CAMYG4GmMBoTo2igMhwZOhYUsi9w6Nn+RW1Ss5cSOuhspMhxvNg@mail.gmail.com>
	<4F9025E2.6010600@gmail.com>
Message-ID: <CAMYG4GnuYfynEgPb1Zt7ZHAQ1OO8CO6HVNBEcAbaL2zF3O1Pug@mail.gmail.com>

On Thu, Apr 19, 2012 at 10:49 AM, Zhenglun (Alan) Wei <
zhenglun.wei at gmail.com> wrote:

>  Dear Dr. Knepley,
>      It is very nice to hear that. I will read the manual. Do we have any
> examples showing its functions?
>

Lots of examples. Start with SNES ex5 and ex19.

   Matt


> thank you so much,
> Alan
> On 4/19/2012 9:35 AM, Matthew Knepley wrote:
>
> On Thu, Apr 19, 2012 at 10:18 AM, Zhenglun (Alan) Wei <
> zhenglun.wei at gmail.com> wrote:
>
>>   "
>>     TESTVAR ***a, ***b, ***c;
>>     TESTVAR **aa, **bb, **cc;
>>     TESTVAR *arraya, *arrayb, *arrayc;
>>
>>      arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>
>>      aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>
>>     for(i = 0; i < SIZE*SIZE; i++) {
>>       aa[i] = &arraya[i*SIZE];
>>       bb[i] = &arrayb[i*SIZE];
>>       cc[i] = &arrayc[i*SIZE];
>>     }
>>
>>      a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>
>>     for(i = 0; i < SIZE; i++) {
>>       a[i] = &aa[i*SIZE];
>>       b[i] = &bb[i*SIZE];
>>       c[i] = &cc[i*SIZE];
>>     }
>> "
>>   It works. However, I wonder if there is any other good ideas for 3D
>> problem other than this kinda of 'two-layer' approach.
>>
>>  *What is the reason for not using DMDA?
>> *In 2D, I established a 2D array for data communication between nodes by
>> using MPI derived data type. It allows me to easily communicate both
>> contiguous (i.e. MPI_TYPE_CONTIGUOUS) and non-contiguous (i.e.
>> MPI_TYPE_VECTOR) data. That is why I use this similar approach in 3D,
>> though an additional data type, i.e. MPI_TYPE_INDEXED, need to be used.
>> Does DMDA have those type of function or derived data type?
>>
>
>  It definitely does communication between the local pieces. Do you want
> something else?
>
>
>>   "2, I have a little question on PETSc about 3D processor ordering.
>>> Does PETSc have any function giving me the nodes/rank number of neighboring
>>> nodes/ranks? Are those 'Application Ordering' functions applicable for my
>>> case?"
>>>
>>
>>  *What do you mean by neighboring? If it is jsut stencil neighbors, then
>> use a local vector.*
>> When I send and receive data with MPI_Send and MPI_RECV, I need provide
>> the 'destination' (in MPI_Send refer to'
>> http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html')
>> and 'source' (in MPI_RECV refer to'
>> http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html').
>> In a 2D problem with Cartesian grid, 4 processes divide the whole domain to
>> 4 sub-domain.
>> ----------------------------
>>       2      |      3      |
>> ----------------------------
>>       0      |      1      |
>> ---------------------------
>>  Then, for node 1, the neighboring nodes are '0' and '3', which '0' is
>> the left node and '3' is the top node. I wonder if PETSc has any function
>> that I can call to obtain those neighboring nodes so that I do not need to
>> construct my function.
>>
>
>  Yes, it looks like you should just use a DMDA. See the manual section.
>
>     Matt
>
>
>>  I'm sorry for confusing you.
>>
>> thanks in advance,
>> Alan
>>
>> On 4/19/2012 4:52 AM, Matthew Knepley wrote:
>>
>> On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei <zhenglun.wei at gmail.com> wrote:
>>
>>> Dear all,
>>>     I hope you're having a nice day. I have a further question on this
>>> issue in 3D.
>>> 1, Following the idea of Dr. Brown and Dr. Knepley, I finished a 2D
>>> test, which works very fine. Here, I did it in 3D by
>>>  "
>>>     TESTVAR ***a, ***b, ***c;
>>>     TESTVAR **aa, **bb, **cc;
>>>     TESTVAR *arraya, *arrayb, *arrayc;
>>>
>>>      arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>>     arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>>     arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>>
>>>      aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>     bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>     cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>
>>>     for(i = 0; i < SIZE*SIZE; i++) {
>>>       aa[i] = &arraya[i*SIZE];
>>>       bb[i] = &arrayb[i*SIZE];
>>>       cc[i] = &arrayc[i*SIZE];
>>>     }
>>>
>>>      a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>     b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>     c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>
>>>     for(i = 0; i < SIZE; i++) {
>>>       a[i] = &aa[i*SIZE];
>>>       b[i] = &bb[i*SIZE];
>>>       c[i] = &cc[i*SIZE];
>>>     }
>>> "
>>>   It works. However, I wonder if there is any other good ideas for 3D
>>> problem other than this kinda of 'two-layer' approach.
>>>
>>
>>  What is the reason for not using DMDA?
>>
>>
>>>  2, I have a little question on PETSc about 3D processor ordering. Does
>>> PETSc have any function giving me the nodes/rank number of neighboring
>>> nodes/ranks? Are those 'Application Ordering' functions applicable for my
>>> case?
>>>
>>
>>  What do you mean by neighboring? If it is jsut stencil neighbors, then
>> use a local vector.
>>
>>     Matt
>>
>>
>>>  thanks,
>>> Alan
>>>
>>> On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>
>>>>  On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei <
>>>> zhenglun.wei at gmail.com> wrote:
>>>>
>>>>>     I have a final question on it. Is it taken a lot of memory for
>>>>> doing this? As I understand, pointers won't occupy many memories and it
>>>>> works like an alias. It will not, to my limit knowledge, take much extra
>>>>> memory by doing this.
>>>>
>>>>
>>>>  A pointer takes about as much space as a floating point value, so
>>>> that array of pointers costs about 1*N compared to the N*N matrix.
>>>>
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/acf3560b/attachment-0001.htm>

From zhenglun.wei at gmail.com  Thu Apr 19 09:52:04 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 19 Apr 2012 09:52:04 -0500
Subject: [petsc-users] A problem with MPI Derived Data Type and 'calloc'
In-Reply-To: <CAMYG4GnuYfynEgPb1Zt7ZHAQ1OO8CO6HVNBEcAbaL2zF3O1Pug@mail.gmail.com>
References: <4F8704BF.3070008@gmail.com>
	<CAM9tzSmsWUpp5-Ltng=c=RMQVOJvPzAMy280pggbp+h547DqGw@mail.gmail.com>
	<4F871C0E.7030402@gmail.com>
	<CAM9tzSmBk4TMrc-0QaM=cjW3-hUDqMLy=L_Yqeoo-yDkWq4XCg@mail.gmail.com>
	<4F88AAEA.40406@gmail.com>
	<CAM9tzS=yTpP_UOa-P9NOp6=z+S6ATk_J7axDySR=gjfg5bdq4Q@mail.gmail.com>
	<CAD7ekpBQfw0Du1Nszev--qKvYtdAZ=ekh7Vg=CUaA_1De3SZZg@mail.gmail.com>
	<CAMYG4G=Jvy51jJw3_2U2gGphWWcA1esgp60TQJP4kYS0k=oPeQ@mail.gmail.com>
	<4F901EC5.5050307@gmail.com>
	<CAMYG4GmMBoTo2igMhwZOhYUsi9w6Nn+RW1Ss5cSOuhspMhxvNg@mail.gmail.com>
	<4F9025E2.6010600@gmail.com>
	<CAMYG4GnuYfynEgPb1Zt7ZHAQ1OO8CO6HVNBEcAbaL2zF3O1Pug@mail.gmail.com>
Message-ID: <4F902694.7000505@gmail.com>

Thank you so much, Dr. Knepley.

Alan

On 4/19/2012 9:51 AM, Matthew Knepley wrote:
> On Thu, Apr 19, 2012 at 10:49 AM, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>     Dear Dr. Knepley,
>          It is very nice to hear that. I will read the manual. Do we
>     have any examples showing its functions?
>
>
> Lots of examples. Start with SNES ex5 and ex19.
>
>    Matt
>
>     thank you so much,
>     Alan
>     On 4/19/2012 9:35 AM, Matthew Knepley wrote:
>>     On Thu, Apr 19, 2012 at 10:18 AM, Zhenglun (Alan) Wei
>>     <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>>
>>         "
>>             TESTVAR ***a, ***b, ***c;
>>             TESTVAR **aa, **bb, **cc;
>>             TESTVAR *arraya, *arrayb, *arrayc;
>>
>>             arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>             arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>             arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE, sizeof(TESTVAR));
>>
>>             aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>             for(i = 0; i < SIZE*SIZE; i++) {
>>               aa[i] = &arraya[i*SIZE];
>>               bb[i] = &arrayb[i*SIZE];
>>               cc[i] = &arrayc[i*SIZE];
>>             }
>>
>>             a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>             for(i = 0; i < SIZE; i++) {
>>               a[i] = &aa[i*SIZE];
>>               b[i] = &bb[i*SIZE];
>>               c[i] = &cc[i*SIZE];
>>             }
>>         "
>>           It works. However, I wonder if there is any other good
>>         ideas for 3D problem other than this kinda of 'two-layer'
>>         approach.
>>
>>         *_What is the reason for not using DMDA?_
>>         *In 2D, I established a 2D array for data communication
>>         between nodes by using MPI derived data type. It allows me to
>>         easily communicate both contiguous (i.e. MPI_TYPE_CONTIGUOUS)
>>         and non-contiguous (i.e. MPI_TYPE_VECTOR) data. That is why I
>>         use this similar approach in 3D, though an additional data
>>         type, i.e. MPI_TYPE_INDEXED, need to be used. Does DMDA have
>>         those type of function or derived data type?
>>
>>
>>     It definitely does communication between the local pieces. Do you
>>     want something else?
>>
>>             "2, I have a little question on PETSc about 3D processor
>>             ordering. Does PETSc have any function giving me the
>>             nodes/rank number of neighboring nodes/ranks? Are those
>>             'Application Ordering' functions applicable for my case?"
>>
>>
>>         _*What do you mean by neighboring? If it is jsut stencil
>>         neighbors, then use a local vector.*_
>>         When I send and receive data with MPI_Send and MPI_RECV, I
>>         need provide the 'destination' (in MPI_Send refer
>>         to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Send.html')
>>         and 'source' (in MPI_RECV refer
>>         to'http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Recv.html').
>>         In a 2D problem with Cartesian grid, 4 processes divide the
>>         whole domain to 4 sub-domain.
>>         ----------------------------
>>               2      |      3      |
>>         ----------------------------
>>               0      |      1      |
>>         ---------------------------
>>          Then, for node 1, the neighboring nodes are '0' and '3',
>>         which '0' is the left node and '3' is the top node. I wonder
>>         if PETSc has any function that I can call to obtain those
>>         neighboring nodes so that I do not need to construct my
>>         function.
>>
>>
>>     Yes, it looks like you should just use a DMDA. See the manual
>>     section.
>>
>>        Matt
>>
>>         I'm sorry for confusing you.
>>
>>         thanks in advance,
>>         Alan
>>
>>         On 4/19/2012 4:52 AM, Matthew Knepley wrote:
>>>         On Wed, Apr 18, 2012 at 3:52 PM, Alan Wei
>>>         <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>>>
>>>             Dear all,
>>>                 I hope you're having a nice day. I have a further
>>>             question on this issue in 3D.
>>>             1, Following the idea of Dr. Brown and Dr. Knepley, I
>>>             finished a 2D test, which works very fine. Here, I did
>>>             it in 3D by
>>>             "
>>>                 TESTVAR ***a, ***b, ***c;
>>>                 TESTVAR **aa, **bb, **cc;
>>>                 TESTVAR *arraya, *arrayb, *arrayc;
>>>
>>>                 arraya = (TESTVAR*) calloc(SIZE*SIZE*SIZE,
>>>             sizeof(TESTVAR));
>>>                 arrayb = (TESTVAR*) calloc(SIZE*SIZE*SIZE,
>>>             sizeof(TESTVAR));
>>>                 arrayc = (TESTVAR*) calloc(SIZE*SIZE*SIZE,
>>>             sizeof(TESTVAR));
>>>
>>>                 aa =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>                 bb =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>                 cc =(TESTVAR**) calloc(SIZE*SIZE, sizeof(TESTVAR*));
>>>                 for(i = 0; i < SIZE*SIZE; i++) {
>>>                   aa[i] = &arraya[i*SIZE];
>>>                   bb[i] = &arrayb[i*SIZE];
>>>                   cc[i] = &arrayc[i*SIZE];
>>>                 }
>>>
>>>                 a =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>                 b =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>                 c =(TESTVAR***) calloc(SIZE*SIZE, sizeof(TESTVAR**));
>>>                 for(i = 0; i < SIZE; i++) {
>>>                   a[i] = &aa[i*SIZE];
>>>                   b[i] = &bb[i*SIZE];
>>>                   c[i] = &cc[i*SIZE];
>>>                 }
>>>             "
>>>               It works. However, I wonder if there is any other good
>>>             ideas for 3D problem other than this kinda of
>>>             'two-layer' approach.
>>>
>>>
>>>         What is the reason for not using DMDA?
>>>
>>>             2, I have a little question on PETSc about 3D processor
>>>             ordering. Does PETSc have any function giving me the
>>>             nodes/rank number of neighboring nodes/ranks? Are those
>>>             'Application Ordering' functions applicable for my case?
>>>
>>>
>>>         What do you mean by neighboring? If it is jsut stencil
>>>         neighbors, then use a local vector.
>>>
>>>            Matt
>>>
>>>             thanks,
>>>             Alan
>>>
>>>             On Fri, Apr 13, 2012 at 5:41 PM, Jed Brown
>>>             <jedbrown at mcs.anl.gov <mailto:jedbrown at mcs.anl.gov>> wrote:
>>>
>>>                 On Fri, Apr 13, 2012 at 17:38, Zhenglun (Alan) Wei
>>>                 <zhenglun.wei at gmail.com
>>>                 <mailto:zhenglun.wei at gmail.com>> wrote:
>>>
>>>                         I have a final question on it. Is it taken a
>>>                     lot of memory for doing this? As I understand,
>>>                     pointers won't occupy many memories and it works
>>>                     like an alias. It will not, to my limit
>>>                     knowledge, take much extra memory by doing this. 
>>>
>>>
>>>                 A pointer takes about as much space as a floating
>>>                 point value, so that array of pointers costs about
>>>                 1*N compared to the N*N matrix.
>>>
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/66877bae/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 10:12:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 08:12:26 -0700
Subject: [petsc-users] PetscBinaryRead in matlab
In-Reply-To: <4F901616.4020403@gfz-potsdam.de>
References: <4F901246.9030206@gfz-potsdam.de>
	<CAM9tzSkyU2e_yz8942Z66+Bu_Oc=1fTc4YJU7j72LeZBgJ8Qqw@mail.gmail.com>
	<4F901616.4020403@gfz-potsdam.de>
Message-ID: <CAM9tzSk3=iVC2Z7NnAET8_uKozmYfJ=nDn=sOpbdZMxTd2uTCA@mail.gmail.com>

On Thu, Apr 19, 2012 at 06:41, Alexander Grayver <agrayver at gfz-potsdam.de>wrote:

> You're right, it is undefined. Your variant works.


http://petsc.cs.iit.edu/petsc/petsc-dev/rev/71a0e13876ec
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/376905c9/attachment.htm>

From rlmackie862 at gmail.com  Thu Apr 19 12:12:18 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 19 Apr 2012 10:12:18 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
Message-ID: <CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>

Hi Matt and Barry,

I tried this again, but this time I used a c subroutine like Barry
suggested, which is this:

#include "petsc.h"

PetscErrorCode  shellkspmonitor_(KSP ksp,PetscInt n,PetscReal rnorm,void
*dummy)
{
PetscErrorCode ierr=0;
    if (n % 10) {
       ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
    }

return ierr;
}


In my Fortran code, I declared shellkspmonitor as external, then I called
KSPSetMonitor as follows:

  call KSPMonitorSet(ksp,shellkspmonitor,PETSC_NULL_OBJECT,
PETSC_NULL_FUNCTION, ierr)


But I am still getting the same error as before:

[0]PETSC ERROR: [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR:
--------------------- Error Message ------------------------------------
--------------------- Error Message ------------------------------------
[3]PETSC ERROR: [2]PETSC ERROR: Invalid argument!
Invalid argument!
[3]PETSC ERROR: Wrong type of object: Parameter # 1!
[3]PETSC ERROR: [2]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
CDT 2011
[3]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
src/sys/viewer/impls/ascii/filev.c
[0]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: shellkspmonitor_() line 7 in custom_monitor.c
[3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[3]PETSC ERROR: See docs/index.html for manual pages.

So, there is either a problem in one of the PETSc routines, or the call
from Fortran to KSPMonitorSet
is wrong, although I have copied the calling sequence from the example in
/src/ksp/ksp/examples/tutorials/ex2f.F

In fact, you can reproduce this error using ex2f.F, simply swapping in the
custom monitor above for
the MyKSPMonitor of that example. I can send you the modified code and c
subroutine to test if you want.

Thanks, Randy


On Tue, Apr 17, 2012 at 10:39 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com>wrote:
>
>> Hi Barry,
>>
>> I've tried implementing this in Fortran, following ex2f.F in
>> /src/ksp/ksp/examples/tutorials, but
>> it is bombing out with the following message:
>>
>> ---------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Invalid argument!
>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
>> CDT 2011
>> -------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Invalid argument!
>>  [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
>> CDT 2011
>>
>>
>> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
>> src/sys/viewer/impls/ascii/filev.c
>> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
>> src/ksp/ksp/interface/iterativ.c
>> [1]PETSC ERROR: ourmonitor() line 103 in
>> src/ksp/ksp/interface/ftn-custom/zitfuncf.c
>> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
>> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
>>
>>
>> I am running PETSC 3.2-p5.
>>
>> I have set my monitor routine as follows:
>>
>>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
>>
>>   implicit none
>>
>> #include "finclude/petsc.h"
>>
>>   KSP :: ksp
>>    PetscErrorCode :: ierr
>>   PetscInt :: n, dummy
>>   PetscReal :: rnorm
>>
>>   if (n == 200) then
>>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
>>
>
> I think it should work if dummy is 0, but we should really do NULL
> conversion
> here and you should pass PETSC_NULL.
>
>    Matt
>
>
>>   end if
>>
>>   return
>>   end subroutine ShellKSPMonitor
>>
>> And then in the main program I call:
>>
>>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT,
>> PETSC_NULL_FUNCTION, ierr)
>>
>> As I have followed the example in the PETSc documentation, I am unsure
>> where I have made a mistake.
>>
>> Thanks,
>>
>> Randy M
>>
>>
>> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Write a one line monitor that calls the true residual monitor when
>>> desired:
>>>
>>>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt
>>> n,PetscReal rnorm,void *dummy)
>>>    {
>>>        if (n % 10) {
>>>           ierr =
>>> KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>>>        }
>>>    }
>>>
>>>    then call this on the KPS
>>>
>>>   ierr =
>>> KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
>>>
>>>
>>> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
>>>
>>> > In using ksp_monitor_true_residual_norm, is it possible to change how
>>> often this information is printed out?
>>> > That is, instead of every iteration, say I only want to see it every
>>> 10 or 20 iterations. Is there an easy way
>>> > to do this, other than creating my own monitor and doing it myself?
>>> >
>>> > Thanks, Randy M.
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/7698cd47/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 12:40:58 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 10:40:58 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
Message-ID: <CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>

It looks like this null object check was missing from the Fortran bindings.
Can you try with this patch included?

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/05b5b9325f55

On Thu, Apr 19, 2012 at 10:12, Randall Mackie <rlmackie862 at gmail.com> wrote:

> Hi Matt and Barry,
>
> I tried this again, but this time I used a c subroutine like Barry
> suggested, which is this:
>
> #include "petsc.h"
>
> PetscErrorCode  shellkspmonitor_(KSP ksp,PetscInt n,PetscReal rnorm,void
> *dummy)
> {
> PetscErrorCode ierr=0;
>     if (n % 10) {
>        ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>     }
>
> return ierr;
> }
>
>
> In my Fortran code, I declared shellkspmonitor as external, then I called
> KSPSetMonitor as follows:
>
>   call KSPMonitorSet(ksp,shellkspmonitor,PETSC_NULL_OBJECT,
> PETSC_NULL_FUNCTION, ierr)
>
>
> But I am still getting the same error as before:
>
> [0]PETSC ERROR: [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR:
> --------------------- Error Message ------------------------------------
> --------------------- Error Message ------------------------------------
> [3]PETSC ERROR: [2]PETSC ERROR: Invalid argument!
> Invalid argument!
> [3]PETSC ERROR: Wrong type of object: Parameter # 1!
> [3]PETSC ERROR: [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
> [3]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
> src/sys/viewer/impls/ascii/filev.c
> [0]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
> src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: shellkspmonitor_() line 7 in custom_monitor.c
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
>
> So, there is either a problem in one of the PETSc routines, or the call
> from Fortran to KSPMonitorSet
> is wrong, although I have copied the calling sequence from the example in
> /src/ksp/ksp/examples/tutorials/ex2f.F
>
> In fact, you can reproduce this error using ex2f.F, simply swapping in the
> custom monitor above for
> the MyKSPMonitor of that example. I can send you the modified code and c
> subroutine to test if you want.
>
> Thanks, Randy
>
>
> On Tue, Apr 17, 2012 at 10:39 AM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com>wrote:
>>
>>> Hi Barry,
>>>
>>> I've tried implementing this in Fortran, following ex2f.F in
>>> /src/ksp/ksp/examples/tutorials, but
>>> it is bombing out with the following message:
>>>
>>> ---------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Invalid argument!
>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29
>>> 13:45:54 CDT 2011
>>> -------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Invalid argument!
>>>  [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29
>>> 13:45:54 CDT 2011
>>>
>>>
>>> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in
>>> src/sys/viewer/impls/ascii/filev.c
>>> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in
>>> src/ksp/ksp/interface/iterativ.c
>>> [1]PETSC ERROR: ourmonitor() line 103 in
>>> src/ksp/ksp/interface/ftn-custom/zitfuncf.c
>>> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
>>> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
>>>
>>>
>>> I am running PETSC 3.2-p5.
>>>
>>> I have set my monitor routine as follows:
>>>
>>>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
>>>
>>>   implicit none
>>>
>>> #include "finclude/petsc.h"
>>>
>>>   KSP :: ksp
>>>    PetscErrorCode :: ierr
>>>   PetscInt :: n, dummy
>>>   PetscReal :: rnorm
>>>
>>>   if (n == 200) then
>>>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
>>>
>>
>> I think it should work if dummy is 0, but we should really do NULL
>> conversion
>> here and you should pass PETSC_NULL.
>>
>>    Matt
>>
>>
>>>   end if
>>>
>>>   return
>>>   end subroutine ShellKSPMonitor
>>>
>>> And then in the main program I call:
>>>
>>>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT,
>>> PETSC_NULL_FUNCTION, ierr)
>>>
>>> As I have followed the example in the PETSc documentation, I am unsure
>>> where I have made a mistake.
>>>
>>> Thanks,
>>>
>>> Randy M
>>>
>>>
>>> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>>   Write a one line monitor that calls the true residual monitor when
>>>> desired:
>>>>
>>>>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt
>>>> n,PetscReal rnorm,void *dummy)
>>>>    {
>>>>        if (n % 10) {
>>>>           ierr =
>>>> KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>>>>        }
>>>>    }
>>>>
>>>>    then call this on the KPS
>>>>
>>>>   ierr =
>>>> KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
>>>>
>>>>
>>>> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
>>>>
>>>> > In using ksp_monitor_true_residual_norm, is it possible to change how
>>>> often this information is printed out?
>>>> > That is, instead of every iteration, say I only want to see it every
>>>> 10 or 20 iterations. Is there an easy way
>>>> > to do this, other than creating my own monitor and doing it myself?
>>>> >
>>>> > Thanks, Randy M.
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/70eb17f9/attachment.htm>

From agrayver at gfz-potsdam.de  Thu Apr 19 13:22:05 2012
From: agrayver at gfz-potsdam.de (=?utf-8?B?QWxleGFuZGVyIEdyYXl2ZXI=?=)
Date: Thu, 19 Apr 2012 20:22:05 +0200
Subject: [petsc-users] =?utf-8?q?PetscBinaryRead_in_matlab?=
Message-ID: <auto-000008142513@cgp2.gfz-potsdam.de>

Jed, 

Thanks.
One more thing, the function doesn't close file properly. I guess one should use fclose for that.

Regards,
Alexander

----- Reply message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Subject: [petsc-users] PetscBinaryRead in matlab
Date: Thu, Apr 19, 2012 17:12
On Thu, Apr 19, 2012 at 06:41, Alexander Grayver <agrayver at gfz-potsdam.de> wrote:

You're right, it is undefined. Your variant works.
http://petsc.cs.iit.edu/petsc/petsc-dev/rev/71a0e13876ec
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/c2f06931/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 13:27:40 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 11:27:40 -0700
Subject: [petsc-users] PetscBinaryRead in matlab
In-Reply-To: <auto-000008142513@cgp2.gfz-potsdam.de>
References: <auto-000008142513@cgp2.gfz-potsdam.de>
Message-ID: <CAM9tzSnX1t1abstTEvUpvp47HTHPxiGr-8d9t2ridwphv2b=Ng@mail.gmail.com>

On Thu, Apr 19, 2012 at 11:22, Alexander Grayver <agrayver at gfz-potsdam.de>wrote:

> Jed,
>
> Thanks.
> One more thing, the function doesn't close file properly. I guess one
> should use fclose for that.
>

This should be calling @PetscOpenFile/close.m

if ischar(inarg) || isinteger(inarg)
    close(fd)
end


Is your path arranged so that it's calling something else?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/c9592525/attachment.htm>

From rlmackie862 at gmail.com  Thu Apr 19 13:32:39 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 19 Apr 2012 11:32:39 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
Message-ID: <FF24D6A4-A80C-4661-A5FF-7FD61AFE1282@gmail.com>

I still get the same error message. Can you try /src/ksp/ksp/examples/tutorials/ex2f.F, but replace the
call to KSPBuildSolution line 348 to

        call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)

That will reproduce the error.

Thanks, Randy M


On Apr 19, 2012, at 10:40 AM, Jed Brown wrote:

> It looks like this null object check was missing from the Fortran bindings. Can you try with this patch included?
> 
> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/05b5b9325f55
> 
> On Thu, Apr 19, 2012 at 10:12, Randall Mackie <rlmackie862 at gmail.com> wrote:
> Hi Matt and Barry,
> 
> I tried this again, but this time I used a c subroutine like Barry suggested, which is this:
> 
> #include "petsc.h"
> 
> PetscErrorCode  shellkspmonitor_(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
> {
> 	PetscErrorCode ierr=0;
>     if (n % 10) {
>        ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>     }
>     
> 	return ierr;
> }
> 
> 
> In my Fortran code, I declared shellkspmonitor as external, then I called KSPSetMonitor as follows:
> 
>   call KSPMonitorSet(ksp,shellkspmonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION, ierr)
> 
> 
> But I am still getting the same error as before:
> 
> [0]PETSC ERROR: [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> --------------------- Error Message ------------------------------------
> [3]PETSC ERROR: [2]PETSC ERROR: Invalid argument!
> Invalid argument!
> [3]PETSC ERROR: Wrong type of object: Parameter # 1!
> [3]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in src/sys/viewer/impls/ascii/filev.c
> [0]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: shellkspmonitor_() line 7 in custom_monitor.c
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
> 
> So, there is either a problem in one of the PETSc routines, or the call from Fortran to KSPMonitorSet 
> is wrong, although I have copied the calling sequence from the example in /src/ksp/ksp/examples/tutorials/ex2f.F
> 
> In fact, you can reproduce this error using ex2f.F, simply swapping in the custom monitor above for
> the MyKSPMonitor of that example. I can send you the modified code and c subroutine to test if you want.
> 
> Thanks, Randy 
> 
> 
> On Tue, Apr 17, 2012 at 10:39 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> Hi Barry,
> 
> I've tried implementing this in Fortran, following ex2f.F in /src/ksp/ksp/examples/tutorials, but
> it is bombing out with the following message:
> 
> ---------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> -------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> 
> 
> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in src/sys/viewer/impls/ascii/filev.c
> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in src/ksp/ksp/interface/iterativ.c
> [1]PETSC ERROR: ourmonitor() line 103 in src/ksp/ksp/interface/ftn-custom/zitfuncf.c
> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
> 
> 
> I am running PETSC 3.2-p5.
> 
> I have set my monitor routine as follows:
> 
>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
> 
>   implicit none
> 
> #include "finclude/petsc.h"
> 
>   KSP :: ksp
>   PetscErrorCode :: ierr
>   PetscInt :: n, dummy
>   PetscReal :: rnorm
> 
>   if (n == 200) then
>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
> 
> I think it should work if dummy is 0, but we should really do NULL conversion
> here and you should pass PETSC_NULL.
> 
>    Matt
>  
>   end if
> 
>   return
>   end subroutine ShellKSPMonitor
> 
> And then in the main program I call:
> 
>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION, ierr)
> 
> As I have followed the example in the PETSc documentation, I am unsure where I have made a mistake.
> 
> Thanks, 
> 
> Randy M
> 
> 
> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Write a one line monitor that calls the true residual monitor when desired:
> 
>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
>    {
>        if (n % 10) {
>           ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>        }
>    }
> 
>    then call this on the KPS
> 
>   ierr = KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
> 
> 
> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
> 
> > In using ksp_monitor_true_residual_norm, is it possible to change how often this information is printed out?
> > That is, instead of every iteration, say I only want to see it every 10 or 20 iterations. Is there an easy way
> > to do this, other than creating my own monitor and doing it myself?
> >
> > Thanks, Randy M.
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/9921fa28/attachment.htm>

From rlmackie862 at gmail.com  Thu Apr 19 13:53:00 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 19 Apr 2012 11:53:00 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
Message-ID: <37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>

Jed,

It is working now - I think my first try I didn't apply the patch correctly.

Thanks for the help.

Randy

 
On Apr 19, 2012, at 10:40 AM, Jed Brown wrote:

> It looks like this null object check was missing from the Fortran bindings. Can you try with this patch included?
> 
> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/05b5b9325f55
> 
> On Thu, Apr 19, 2012 at 10:12, Randall Mackie <rlmackie862 at gmail.com> wrote:
> Hi Matt and Barry,
> 
> I tried this again, but this time I used a c subroutine like Barry suggested, which is this:
> 
> #include "petsc.h"
> 
> PetscErrorCode  shellkspmonitor_(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
> {
> 	PetscErrorCode ierr=0;
>     if (n % 10) {
>        ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>     }
>     
> 	return ierr;
> }
> 
> 
> In my Fortran code, I declared shellkspmonitor as external, then I called KSPSetMonitor as follows:
> 
>   call KSPMonitorSet(ksp,shellkspmonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION, ierr)
> 
> 
> But I am still getting the same error as before:
> 
> [0]PETSC ERROR: [1]PETSC ERROR: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> --------------------- Error Message ------------------------------------
> [3]PETSC ERROR: [2]PETSC ERROR: Invalid argument!
> Invalid argument!
> [3]PETSC ERROR: Wrong type of object: Parameter # 1!
> [3]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in src/sys/viewer/impls/ascii/filev.c
> [0]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: shellkspmonitor_() line 7 in custom_monitor.c
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
> 
> So, there is either a problem in one of the PETSc routines, or the call from Fortran to KSPMonitorSet 
> is wrong, although I have copied the calling sequence from the example in /src/ksp/ksp/examples/tutorials/ex2f.F
> 
> In fact, you can reproduce this error using ex2f.F, simply swapping in the custom monitor above for
> the MyKSPMonitor of that example. I can send you the modified code and c subroutine to test if you want.
> 
> Thanks, Randy 
> 
> 
> On Tue, Apr 17, 2012 at 10:39 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Apr 17, 2012 at 1:27 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> Hi Barry,
> 
> I've tried implementing this in Fortran, following ex2f.F in /src/ksp/ksp/examples/tutorials, but
> it is bombing out with the following message:
> 
> ---------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> -------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 
> 
> 
> [1]PETSC ERROR: PetscViewerASCIIAddTab() line 277 in src/sys/viewer/impls/ascii/filev.c
> [1]PETSC ERROR: KSPMonitorTrueResidualNorm() line 233 in src/ksp/ksp/interface/iterativ.c
> [1]PETSC ERROR: ourmonitor() line 103 in src/ksp/ksp/interface/ftn-custom/zitfuncf.c
> [1]PETSC ERROR: KSPMonitor() line 1429 in src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: KSPSolve_CG() line 255 in src/ksp/ksp/impls/cg/cg.c
> [1]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c
> 
> 
> I am running PETSC 3.2-p5.
> 
> I have set my monitor routine as follows:
> 
>   subroutine ShellKSPMonitor(ksp, n, rnorm, dummy, ierr)
> 
>   implicit none
> 
> #include "finclude/petsc.h"
> 
>   KSP :: ksp
>   PetscErrorCode :: ierr
>   PetscInt :: n, dummy
>   PetscReal :: rnorm
> 
>   if (n == 200) then
>     call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
> 
> I think it should work if dummy is 0, but we should really do NULL conversion
> here and you should pass PETSC_NULL.
> 
>    Matt
>  
>   end if
> 
>   return
>   end subroutine ShellKSPMonitor
> 
> And then in the main program I call:
> 
>   call KSPMonitorSet(ksp,ShellKSPMonitor,PETSC_NULL_OBJECT, PETSC_NULL_FUNCTION, ierr)
> 
> As I have followed the example in the PETSc documentation, I am unsure where I have made a mistake.
> 
> Thanks, 
> 
> Randy M
> 
> 
> On Fri, Apr 13, 2012 at 5:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Write a one line monitor that calls the true residual monitor when desired:
> 
>    PetscErrorCode  myKSPMonitorTrueResidualNorm(KSP ksp,PetscInt n,PetscReal rnorm,void *dummy)
>    {
>        if (n % 10) {
>           ierr = KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy);CHKERRQ(ierr);
>        }
>    }
> 
>    then call this on the KPS
> 
>   ierr = KSPMonitorSet(ksp,myKSPMonitorTrueResidualNorm,PETSC_VIEWER_STDOUT,0);CHKERRQ(ierr);
> 
> 
> On Apr 13, 2012, at 6:52 PM, Randall Mackie wrote:
> 
> > In using ksp_monitor_true_residual_norm, is it possible to change how often this information is printed out?
> > That is, instead of every iteration, say I only want to see it every 10 or 20 iterations. Is there an easy way
> > to do this, other than creating my own monitor and doing it myself?
> >
> > Thanks, Randy M.
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/c07d6eaa/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 14:00:03 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 12:00:03 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <FF24D6A4-A80C-4661-A5FF-7FD61AFE1282@gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<FF24D6A4-A80C-4661-A5FF-7FD61AFE1282@gmail.com>
Message-ID: <CAM9tzS=E4_+FJM_o84e-girZ_EPXNPeobuFeY080L0UrnbnFNw@mail.gmail.com>

On Thu, Apr 19, 2012 at 11:32, Randall Mackie <rlmackie862 at gmail.com> wrote:

> I still get the same error message. Can you try
> /src/ksp/ksp/examples/tutorials/ex2f.F, but replace the
> call to KSPBuildSolution line 348 to
>
>         call KSPMonitorTrueResidualNorm(ksp,n,rnorm,dummy,ierr)
>
> That will reproduce the error.
>

Okay, this is much more clear. I do not know a way to pass
PETSC_NULL_OBJECT through Fortran without interaction from the Fortran
side. The reason is that (dummy .eq. PETSC_NULL_OBJECT) compares by value,
but the reference is actually passed. So

dummy = PETSC_NULL_OBJECT
call Anything(dummy)

is different from

call Anything(PETSC_NULL_OBJECT)


The current code is dangerous because (in the NULL case), the address of
"dummy" comes into the Fortran routine as 0x0, so anything that uses its
value will dereference that bad pointer. We can pass
mctx?mctx:PETSC_NULL_OBJECT_Fortran so that we never pass raw 0x0, but the
Fortran side would still have to test for equality to PETSC_NULL_OBJECT and
make the call differently. Can anyone think of a better way?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/30ab3266/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 14:00:31 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 12:00:31 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
Message-ID: <CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>

On Thu, Apr 19, 2012 at 11:53, Randall Mackie <rlmackie862 at gmail.com> wrote:

> It is working now - I think my first try I didn't apply the patch
> correctly.


Are you making the call from C now? Or did you create a viewer so that the
Fortran NULL issues could be avoided?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/149c35e2/attachment.htm>

From rlmackie862 at gmail.com  Thu Apr 19 14:02:26 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 19 Apr 2012 12:02:26 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
	<CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>
Message-ID: <7F7D7638-3394-48A0-A1FF-9B151F3FED51@gmail.com>

Jed,

I am still calling from Fortran, AND using my original Fortran ShellKSPMonitor that calls KSPMonitorTrueResidual.
Seems to work okay now.

Thanks again,

Randy


On Apr 19, 2012, at 12:00 PM, Jed Brown wrote:

> On Thu, Apr 19, 2012 at 11:53, Randall Mackie <rlmackie862 at gmail.com> wrote:
> It is working now - I think my first try I didn't apply the patch correctly.
> 
> Are you making the call from C now? Or did you create a viewer so that the Fortran NULL issues could be avoided?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/a6de70f4/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 14:05:01 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 12:05:01 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <7F7D7638-3394-48A0-A1FF-9B151F3FED51@gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
	<CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>
	<7F7D7638-3394-48A0-A1FF-9B151F3FED51@gmail.com>
Message-ID: <CAM9tzSnA6F_ZxnEv87hSwEqzw4pdTnnd4UCG30TrV=TZGg8skA@mail.gmail.com>

On Thu, Apr 19, 2012 at 12:02, Randall Mackie <rlmackie862 at gmail.com> wrote:

> I am still calling from Fortran, AND using my original Fortran
> ShellKSPMonitor that calls KSPMonitorTrueResidual.
> Seems to work okay now.
>

Did you create a PetscViewer context? I don't think it can work correctly
passing PETSC_NULL_OBJECT, it certainly doesn't in ex2f.F
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/37a4c6c4/attachment.htm>

From rlmackie862 at gmail.com  Thu Apr 19 14:07:08 2012
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 19 Apr 2012 12:07:08 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <CAM9tzSnA6F_ZxnEv87hSwEqzw4pdTnnd4UCG30TrV=TZGg8skA@mail.gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
	<CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>
	<7F7D7638-3394-48A0-A1FF-9B151F3FED51@gmail.com>
	<CAM9tzSnA6F_ZxnEv87hSwEqzw4pdTnnd4UCG30TrV=TZGg8skA@mail.gmail.com>
Message-ID: <BC1323FC-64D8-464F-AD42-B6BDD939F78F@gmail.com>

No I did not, but then ex2f.F is also trying to call VecView, which I am not doing in my custom KSP monitor.
I can call KSPMonitorTrueResidualNorm from a c subroutine so that there are no problems, though.

Randy


On Apr 19, 2012, at 12:05 PM, Jed Brown wrote:

> On Thu, Apr 19, 2012 at 12:02, Randall Mackie <rlmackie862 at gmail.com> wrote:
> I am still calling from Fortran, AND using my original Fortran ShellKSPMonitor that calls KSPMonitorTrueResidual.
> Seems to work okay now.
> 
> Did you create a PetscViewer context? I don't think it can work correctly passing PETSC_NULL_OBJECT, it certainly doesn't in ex2f.F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/89850a8b/attachment.htm>

From jedbrown at mcs.anl.gov  Thu Apr 19 14:14:22 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 19 Apr 2012 12:14:22 -0700
Subject: [petsc-users] ksp_monitor_true_residual_norm
In-Reply-To: <BC1323FC-64D8-464F-AD42-B6BDD939F78F@gmail.com>
References: <CAKsPKwkpHz=50tvE=aCF1ZcV4oe6qgFq9_7gVW1UFW9yHFnyww@mail.gmail.com>
	<CAM9tzSmw8MqtZ-iAAuTH01-eyyfryZQqFDaJ9JTwBaQEjLVh9Q@mail.gmail.com>
	<CAKsPKwnwf3iD3g6qwSEXOdoMDu05b7jPX=y2karFy23izHm3zw@mail.gmail.com>
	<CAM9tzSmA66rqcQeh-fzNxf987K+oftrZTnAQ6yNkTw3=5D2R=w@mail.gmail.com>
	<CAKsPKwmBvhqW9H4UBMzSwDnf-9qK+CDRx3s_KTWCr-viuTPdbA@mail.gmail.com>
	<8D43C964-1CBF-43D1-9DF6-DEC6ADD0941E@gmail.com>
	<8632C07E-B095-476A-845D-EBD52D331FFB@mcs.anl.gov>
	<CADwseAYjWS_CkLonZOjMkW4zGr49g7wB-EZ_r1NPQc4=EVjFVQ@mail.gmail.com>
	<CAMYG4Gno9oess2_Qq7gOiM2oSLxuFo5rBW0kCjtBjSP6U-oUSw@mail.gmail.com>
	<CADwseAYxtHW_+eDU+dEv9VnFXZ6u1eAnoc50FpPkEd5W=J0Oxg@mail.gmail.com>
	<CAM9tzSkJ+JbUeCRzbnBtvARRz+MWjh=cpHjArTAJBg6m5zvO8A@mail.gmail.com>
	<37B86187-D1AA-4451-99E1-ADAD5B7F63FC@gmail.com>
	<CAM9tzSmiPA8FTF0Hti-8SO+bT-OR6iGu5+cr645fES3WQyJdXQ@mail.gmail.com>
	<7F7D7638-3394-48A0-A1FF-9B151F3FED51@gmail.com>
	<CAM9tzSnA6F_ZxnEv87hSwEqzw4pdTnnd4UCG30TrV=TZGg8skA@mail.gmail.com>
	<BC1323FC-64D8-464F-AD42-B6BDD939F78F@gmail.com>
Message-ID: <CAM9tzS=C1Q7NP5-dOb05sUvdbA7-2ns3f53pCpBBD8ZQibhYig@mail.gmail.com>

On Thu, Apr 19, 2012 at 12:07, Randall Mackie <rlmackie862 at gmail.com> wrote:

> No I did not, but then ex2f.F is also trying to call VecView, which I am
> not doing in my custom KSP monitor.
> I can call KSPMonitorTrueResidualNorm from a c subroutine so that there
> are no problems, though.
>

Okay, great.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120419/d8ebb92f/attachment.htm>

From dominik at itis.ethz.ch  Fri Apr 20 08:53:43 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Fri, 20 Apr 2012 15:53:43 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
Message-ID: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>

I am using SNES to solve a simple nonlinear system.
My vector of unknowns (x) is ghosted, and before (i.e. with my own
home brewed nonlinear solver) I had to always update the ghosted
values after a linear solve with ksp. How can I now force the update,
if I am not calling ksp myself?

Thanks for any hints,

Dominik

From knepley at gmail.com  Fri Apr 20 09:08:27 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 20 Apr 2012 10:08:27 -0400
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
Message-ID: <CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>

On Fri, Apr 20, 2012 at 9:53 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I am using SNES to solve a simple nonlinear system.
> My vector of unknowns (x) is ghosted, and before (i.e. with my own
> home brewed nonlinear solver) I had to always update the ghosted
> values after a linear solve with ksp. How can I now force the update,
> if I am not calling ksp myself?
>

You can use


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html

Also, you could investigate using DM, where this should be automatic, but
that is probably
a bigger change to your code.

   Matt


> Thanks for any hints,
>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/6b441931/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 20 09:08:25 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 20 Apr 2012 07:08:25 -0700
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
Message-ID: <CAM9tzSkMrtU=cveMV5gzEs1KEw-eW8JEoqb4v3OTELSyRDWPOA@mail.gmail.com>

On Fri, Apr 20, 2012 at 06:53, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I am using SNES to solve a simple nonlinear system.
> My vector of unknowns (x) is ghosted, and before (i.e. with my own
> home brewed nonlinear solver) I had to always update the ghosted
> values after a linear solve with ksp. How can I now force the update,
> if I am not calling ksp myself?
>

You should do the update in your residual evaluation routine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/ad7d419b/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 20 09:13:41 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 20 Apr 2012 07:13:41 -0700
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
Message-ID: <CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>

On Fri, Apr 20, 2012 at 07:08, Matthew Knepley <knepley at gmail.com> wrote:

> You can use
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html
>

Please don't do it this way, line searches and other methods will not work.
A ghosted vector is defined by the global entries, not the local form.
Update the local form when you want it to be current.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/c3da2053/attachment.htm>

From christian.staudt at ira.uka.de  Fri Apr 20 10:17:31 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Fri, 20 Apr 2012 17:17:31 +0200
Subject: [petsc-users] undefined symbols in libpetsc.a
Message-ID: <4DDDCE75-59E7-4207-BE09-4A207CD40037@ira.uka.de>

Hi PETSc users,

I am trying out PETSc with something very simple from the tutorials [3], creating a vector. My source code:  [1]
The compiler (via Eclipse CDT) fails with the output [2] - undefined symbols "_dswap" "_dnrm2" etc.. in libpetsc.a

I have no idea where to start looking for the error and would be thankful for some hints.

Chris

[1] http://pastebin.com/eMdpAbHy
[2] http://pastebin.com/AB6UynTV
[3] http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex1.c.html




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/0f58b357/attachment.htm>

From balay at mcs.anl.gov  Fri Apr 20 10:23:31 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 20 Apr 2012 10:23:31 -0500 (CDT)
Subject: [petsc-users] undefined symbols in libpetsc.a
In-Reply-To: <4DDDCE75-59E7-4207-BE09-4A207CD40037@ira.uka.de>
References: <4DDDCE75-59E7-4207-BE09-4A207CD40037@ira.uka.de>
Message-ID: <alpine.LFD.2.02.1204201022300.2188@asterix>

Try compiling a PETSc example [with a petsc makefile] to see all the
include and library options needed to compile.

In this case - looks like you are missing linking with blas/lapack

Satish


On Fri, 20 Apr 2012, Christian Staudt wrote:

> Hi PETSc users,
> 
> I am trying out PETSc with something very simple from the tutorials [3], creating a vector. My source code:  [1]
> The compiler (via Eclipse CDT) fails with the output [2] - undefined symbols "_dswap" "_dnrm2" etc.. in libpetsc.a
> 
> I have no idea where to start looking for the error and would be thankful for some hints.
> 
> Chris
> 
> [1] http://pastebin.com/eMdpAbHy
> [2] http://pastebin.com/AB6UynTV
> [3] http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex1.c.html
> 
> 
> 
> 
> 


From christian.staudt at ira.uka.de  Fri Apr 20 12:23:29 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Fri, 20 Apr 2012 19:23:29 +0200
Subject: [petsc-users] undefined symbols in libpetsc.a
In-Reply-To: <mailman.69.1334941272.1626.petsc-users@mcs.anl.gov>
References: <mailman.69.1334941272.1626.petsc-users@mcs.anl.gov>
Message-ID: <64ECFF1E-4CDE-444D-808D-6F9F3F341AA8@ira.uka.de>


Am 20.04.2012 um 19:01 schrieb petsc-users-request at mcs.anl.gov:

> In this case - looks like you are missing linking with blas/lapack
> 
> Satish

Thanks for the hint. I added "blas" and "lapack" as an -l option to the linker. Now my code snippet compiles. 

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/c4892670/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 20 13:31:52 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 20 Apr 2012 11:31:52 -0700
Subject: [petsc-users] Advice on OpenMP/PETSc mix
Message-ID: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>

Hi guys,

I have seen multiple emails regarding this in the mailing list and I'm
afraid you might have already answered this question but I'm not quite sure!

I have objects in my code that are hard(er) to parallelize using MPI and so
far my strategy has been to just handle them in serial such that each
process has a copy of the whole thing. This object is related to my grid
generation/information  etc so it only needs to be done once at
the beginning (no moving mesh for NOW). As a result I do not care much
about the speed since its nothing compared to the overall solution time.
However, I do care about the memory that this object consumes and can limit
my problem size.

So I had the following idea the other day. Is it possible/good idea to
paralleize the grid generation using OpenMP so that each node (as opposed
to core) would share the data structure? This can save me a lot since
memory on nodes are shared among cores (e.g. 32 GB/node vs 2GB/core on
Ranger). What I'm not quite sure about is how the job is scheduled when
running the code via mpirun -n Np. Should Np be the total number of cores
or nodes?

If I use, say Np = 16 processes on one node, MPI is running 16 versions of
the code on a single node (which has 16 cores). How does OpenMP figure out
how to fork? Does it fork a total of 16 threads/MPI process = 256 threads
or is it smart to just fork a total of 16 threads/node = 1 thread/core = 16
threads? I'm a bit confused here how the job is scheduled when MPI and
OpenMP are mixed?

Do I make any sense at all?!

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/cd466a9c/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Fri Apr 20 13:44:59 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Fri, 20 Apr 2012 21:44:59 +0300
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
Message-ID: <CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>

>
> If I use, say Np = 16 processes on one node, MPI is running 16 versions of
> the code on a single node (which has 16 cores). How does OpenMP figure out
> how to fork? Does it fork a total of 16 threads/MPI process = 256 threads
> or is it smart to just fork a total of 16 threads/node = 1 thread/core = 16
> threads? I'm a bit confused here how the job is scheduled when MPI and
> OpenMP are mixed?
>

This is one important use for OMP_NUM_THREADS.   If you're trying to
increase the amount of memory per process, you should map one process per
node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
 There are lots of tutorials and even textbooks now that discuss hybrid
programming techniques that you should look to for more information (or you
could try scicomp.stackexchange.com).

Cheers,
Aron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/efa98e56/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 20 13:49:47 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 20 Apr 2012 11:49:47 -0700
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
Message-ID: <CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>

Thanks Aron. That would work but at the cost of wasting idle cores when
threads join and the rest of MPI-based code is running, correct?

On Fri, Apr 20, 2012 at 11:44 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:

> If I use, say Np = 16 processes on one node, MPI is running 16 versions of
>> the code on a single node (which has 16 cores). How does OpenMP figure out
>> how to fork? Does it fork a total of 16 threads/MPI process = 256 threads
>> or is it smart to just fork a total of 16 threads/node = 1 thread/core = 16
>> threads? I'm a bit confused here how the job is scheduled when MPI and
>> OpenMP are mixed?
>>
>
> This is one important use for OMP_NUM_THREADS.   If you're trying to
> increase the amount of memory per process, you should map one process per
> node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
>  There are lots of tutorials and even textbooks now that discuss hybrid
> programming techniques that you should look to for more information (or you
> could try scicomp.stackexchange.com).
>
> Cheers,
> Aron
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/1059d81f/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Fri Apr 20 13:55:49 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Fri, 20 Apr 2012 21:55:49 +0300
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
	<CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
Message-ID: <CADVoUZQeqbXE-firinQBKoQ+5NSSz2EeXyfwq3n1NpzC9=6FUg@mail.gmail.com>

Yes, that's right.  Though if you're memory-bound you may not notice as
great of an effect.

A

On Fri, Apr 20, 2012 at 9:49 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Thanks Aron. That would work but at the cost of wasting idle cores when
> threads join and the rest of MPI-based code is running, correct?
>
>
> On Fri, Apr 20, 2012 at 11:44 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:
>
>> If I use, say Np = 16 processes on one node, MPI is running 16 versions
>>> of the code on a single node (which has 16 cores). How does OpenMP figure
>>> out how to fork? Does it fork a total of 16 threads/MPI process = 256
>>> threads or is it smart to just fork a total of 16 threads/node = 1
>>> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
>>> when MPI and OpenMP are mixed?
>>>
>>
>> This is one important use for OMP_NUM_THREADS.   If you're trying to
>> increase the amount of memory per process, you should map one process per
>> node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
>>  There are lots of tutorials and even textbooks now that discuss hybrid
>> programming techniques that you should look to for more information (or you
>> could try scicomp.stackexchange.com).
>>
>> Cheers,
>> Aron
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/24b3a01c/attachment.htm>

From five9a2 at gmail.com  Fri Apr 20 13:58:39 2012
From: five9a2 at gmail.com (Jed Brown)
Date: Fri, 20 Apr 2012 13:58:39 -0500
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
	<CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
Message-ID: <CAM9tzSkdqAab4pTF5zd+w_0ZOBhMJA31Zd8_HNBKs4ocvrV8dQ@mail.gmail.com>

PETSc can use threads or the "HMPI" approach. We are in the process of
unifying the code for all the threading models, at which point PETSc will
be able to use your OpenMP threads.
On Apr 20, 2012 1:50 PM, "Mohammad Mirzadeh" <mirzadeh at gmail.com> wrote:

> Thanks Aron. That would work but at the cost of wasting idle cores when
> threads join and the rest of MPI-based code is running, correct?
>
> On Fri, Apr 20, 2012 at 11:44 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:
>
>> If I use, say Np = 16 processes on one node, MPI is running 16 versions
>>> of the code on a single node (which has 16 cores). How does OpenMP figure
>>> out how to fork? Does it fork a total of 16 threads/MPI process = 256
>>> threads or is it smart to just fork a total of 16 threads/node = 1
>>> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
>>> when MPI and OpenMP are mixed?
>>>
>>
>> This is one important use for OMP_NUM_THREADS.   If you're trying to
>> increase the amount of memory per process, you should map one process per
>> node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
>>  There are lots of tutorials and even textbooks now that discuss hybrid
>> programming techniques that you should look to for more information (or you
>> could try scicomp.stackexchange.com).
>>
>> Cheers,
>> Aron
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/282cadff/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 20 14:42:55 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 20 Apr 2012 14:42:55 -0500
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAM9tzSkdqAab4pTF5zd+w_0ZOBhMJA31Zd8_HNBKs4ocvrV8dQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
	<CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
	<CAM9tzSkdqAab4pTF5zd+w_0ZOBhMJA31Zd8_HNBKs4ocvrV8dQ@mail.gmail.com>
Message-ID: <CAM9tzSkO=pg7K2QCkow2OkYEh-_amX=brSk-dg06aNBUZdUgEg@mail.gmail.com>

PETSc can use threads or the "HMPI" approach. We are in the process of
unifying the code for all the threading models, at which point PETSc will
be able to use your OpenMP threads.

> On Apr 20, 2012 1:50 PM, "Mohammad Mirzadeh" <mirzadeh at gmail.com> wrote:
>>
>> Thanks Aron. That would work but at the cost of wasting idle cores when
threads join and the rest of MPI-based code is running, correct?
>>
>> On Fri, Apr 20, 2012 at 11:44 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
wrote:
>>>>
>>>> If I use, say Np = 16 processes on one node, MPI is running 16
versions of the code on a single node (which has 16 cores). How does OpenMP
figure out how to fork? Does it fork a total of 16 threads/MPI process =
256 threads or is it smart to just fork a total of 16 threads/node = 1
thread/core = 16 threads? I'm a bit confused here how the job is scheduled
when MPI and OpenMP are mixed?
>>>
>>>
>>> This is one important use for OMP_NUM_THREADS.   If you're trying to
increase the amount of memory per process, you should map one process per
node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
 There are lots of tutorials and even textbooks now that discuss hybrid
programming techniques that you should look to for more information (or you
could try scicomp.stackexchange.com).
>>>
>>> Cheers,
>>> Aron
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/e0e7086a/attachment-0001.htm>

From mirzadeh at gmail.com  Fri Apr 20 14:56:45 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 20 Apr 2012 12:56:45 -0700
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAM9tzSkO=pg7K2QCkow2OkYEh-_amX=brSk-dg06aNBUZdUgEg@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<CADVoUZR6VQNmRwJqs2WjaqRvBgKXNbeoC57ZgpvEBzgdPgRdOQ@mail.gmail.com>
	<CAKsPKwnVaNReKjU=GSr0cDLt3b8VETaiOrPxy4u9Jtocg0X_WQ@mail.gmail.com>
	<CAM9tzSkdqAab4pTF5zd+w_0ZOBhMJA31Zd8_HNBKs4ocvrV8dQ@mail.gmail.com>
	<CAM9tzSkO=pg7K2QCkow2OkYEh-_amX=brSk-dg06aNBUZdUgEg@mail.gmail.com>
Message-ID: <CAKsPKwkDFz_=H+hPFn5unLo3-xNs1WPrWgiVGdPhhEJO3j3nmA@mail.gmail.com>

> PETSc can use threads or the "HMPI" approach. We are in the process of
unifying the code for all the threading models, at which point PETSc will
be able to use your OpenMP threads.

Jed, does this mean (in future) PETSc can automatically recognize my OpenMP
threads  at the core level and use them for parallelization  even if the
binary is run as 1 MPI process/node?

On Fri, Apr 20, 2012 at 12:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

>  PETSc can use threads or the "HMPI" approach. We are in the process of
> unifying the code for all the threading models, at which point PETSc will
> be able to use your OpenMP threads.
>
> > On Apr 20, 2012 1:50 PM, "Mohammad Mirzadeh" <mirzadeh at gmail.com> wrote:
> >>
> >> Thanks Aron. That would work but at the cost of wasting idle cores when
> threads join and the rest of MPI-based code is running, correct?
> >>
> >> On Fri, Apr 20, 2012 at 11:44 AM, Aron Ahmadia <
> aron.ahmadia at kaust.edu.sa> wrote:
> >>>>
> >>>> If I use, say Np = 16 processes on one node, MPI is running 16
> versions of the code on a single node (which has 16 cores). How does OpenMP
> figure out how to fork? Does it fork a total of 16 threads/MPI process =
> 256 threads or is it smart to just fork a total of 16 threads/node = 1
> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
> when MPI and OpenMP are mixed?
> >>>
> >>>
> >>> This is one important use for OMP_NUM_THREADS.   If you're trying to
> increase the amount of memory per process, you should map one process per
> node and set OMP_NUM_THREADS to the number of OpenMP threads you'd like.
>  There are lots of tutorials and even textbooks now that discuss hybrid
> programming techniques that you should look to for more information (or you
> could try scicomp.stackexchange.com).
> >>>
> >>> Cheers,
> >>> Aron
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/611f8d39/attachment.htm>

From bsmith at mcs.anl.gov  Fri Apr 20 16:25:16 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 20 Apr 2012 16:25:16 -0500
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
Message-ID: <F44DB50F-48EA-45A8-B7C6-D209C0E0BE36@mcs.anl.gov>


   Mohammad,

     Short term for what you can do NOW. 

     PETSc wants to have one MPI process for core; so start up the program that way, (say there are 16 cores per node.  In the block of code that does your "non-MPI stuff" do 

     if ((rank % 16) == 0) {  /* this code is only running on one MPI process per node */
          build your mesh grid data structure and process it, use MPI pramas whatever to parallelize that computation, have a big data structure  */
     }
     have the (rank % 16) == 0  MPI processes send the grid information to each (rank % 16) == j MPI process the part of the grid information they need. 
     have the (rank % 16) == 0  MPI processes delete the global big data structure it built.

     The rest of the program runs as a regular MPI PETSc program

     The advantage of this approach is 1) it will run today 2) it doesn't depend on any fragile os features or software. The disadvantage is that you need to figure out what part of the grid data each process needs and ship it from the (rank % 16) == 0  MPI processes.


    Barry


    


On Apr 20, 2012, at 1:31 PM, Mohammad Mirzadeh wrote:

> Hi guys,
> 
> I have seen multiple emails regarding this in the mailing list and I'm afraid you might have already answered this question but I'm not quite sure!
> 
> I have objects in my code that are hard(er) to parallelize using MPI and so far my strategy has been to just handle them in serial such that each process has a copy of the whole thing. This object is related to my grid generation/information  etc so it only needs to be done once at the beginning (no moving mesh for NOW). As a result I do not care much about the speed since its nothing compared to the overall solution time. However, I do care about the memory that this object consumes and can limit my problem size.
> 
> So I had the following idea the other day. Is it possible/good idea to paralleize the grid generation using OpenMP so that each node (as opposed to core) would share the data structure? This can save me a lot since memory on nodes are shared among cores (e.g. 32 GB/node vs 2GB/core on Ranger). What I'm not quite sure about is how the job is scheduled when running the code via mpirun -n Np. Should Np be the total number of cores or nodes? 
> 
> If I use, say Np = 16 processes on one node, MPI is running 16 versions of the code on a single node (which has 16 cores). How does OpenMP figure out how to fork? Does it fork a total of 16 threads/MPI process = 256 threads or is it smart to just fork a total of 16 threads/node = 1 thread/core = 16 threads? I'm a bit confused here how the job is scheduled when MPI and OpenMP are mixed? 
> 
> Do I make any sense at all?! 
> 
> Thanks


From mirzadeh at gmail.com  Fri Apr 20 16:39:22 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 20 Apr 2012 14:39:22 -0700
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <F44DB50F-48EA-45A8-B7C6-D209C0E0BE36@mcs.anl.gov>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<F44DB50F-48EA-45A8-B7C6-D209C0E0BE36@mcs.anl.gov>
Message-ID: <CAKsPKwmbOK2HZs6uo8J=+s3v=LeHWbHtiQ8595wi_qk7jTq9bQ@mail.gmail.com>

Barry,

That's quite smart and I like it. Aside from the disadvantage that you
mentioned (which to be honest is not quite straight forward but doable) I
have the following question.

When I do computations on, say (rank%16 == 0) processes, do they have
access to the whole 32 GB memory or are they still bounded by the 2GB/core?
... Or in a more broad sense, am I mistaken to assume that each core only
has access to 2GB?

Thanks for your support.

On Fri, Apr 20, 2012 at 2:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Mohammad,
>
>     Short term for what you can do NOW.
>
>     PETSc wants to have one MPI process for core; so start up the program
> that way, (say there are 16 cores per node.  In the block of code that does
> your "non-MPI stuff" do
>
>     if ((rank % 16) == 0) {  /* this code is only running on one MPI
> process per node */
>          build your mesh grid data structure and process it, use MPI
> pramas whatever to parallelize that computation, have a big data structure
>  */
>     }
>     have the (rank % 16) == 0  MPI processes send the grid information to
> each (rank % 16) == j MPI process the part of the grid information they
> need.
>     have the (rank % 16) == 0  MPI processes delete the global big data
> structure it built.
>
>     The rest of the program runs as a regular MPI PETSc program
>
>     The advantage of this approach is 1) it will run today 2) it doesn't
> depend on any fragile os features or software. The disadvantage is that you
> need to figure out what part of the grid data each process needs and ship
> it from the (rank % 16) == 0  MPI processes.
>
>
>    Barry
>
>
>
>
>
> On Apr 20, 2012, at 1:31 PM, Mohammad Mirzadeh wrote:
>
> > Hi guys,
> >
> > I have seen multiple emails regarding this in the mailing list and I'm
> afraid you might have already answered this question but I'm not quite sure!
> >
> > I have objects in my code that are hard(er) to parallelize using MPI and
> so far my strategy has been to just handle them in serial such that each
> process has a copy of the whole thing. This object is related to my grid
> generation/information  etc so it only needs to be done once at the
> beginning (no moving mesh for NOW). As a result I do not care much about
> the speed since its nothing compared to the overall solution time. However,
> I do care about the memory that this object consumes and can limit my
> problem size.
> >
> > So I had the following idea the other day. Is it possible/good idea to
> paralleize the grid generation using OpenMP so that each node (as opposed
> to core) would share the data structure? This can save me a lot since
> memory on nodes are shared among cores (e.g. 32 GB/node vs 2GB/core on
> Ranger). What I'm not quite sure about is how the job is scheduled when
> running the code via mpirun -n Np. Should Np be the total number of cores
> or nodes?
> >
> > If I use, say Np = 16 processes on one node, MPI is running 16 versions
> of the code on a single node (which has 16 cores). How does OpenMP figure
> out how to fork? Does it fork a total of 16 threads/MPI process = 256
> threads or is it smart to just fork a total of 16 threads/node = 1
> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
> when MPI and OpenMP are mixed?
> >
> > Do I make any sense at all?!
> >
> > Thanks
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/21d609b4/attachment.htm>

From jedbrown at mcs.anl.gov  Fri Apr 20 17:05:57 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 20 Apr 2012 17:05:57 -0500
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAKsPKwmbOK2HZs6uo8J=+s3v=LeHWbHtiQ8595wi_qk7jTq9bQ@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<F44DB50F-48EA-45A8-B7C6-D209C0E0BE36@mcs.anl.gov>
	<CAKsPKwmbOK2HZs6uo8J=+s3v=LeHWbHtiQ8595wi_qk7jTq9bQ@mail.gmail.com>
Message-ID: <CAM9tzS=xqsBFTWyt+butHjvreRiu2L1eKj4m+GEYWSjnym9Kdw@mail.gmail.com>

You have access to all the memory, though it may help do distribute it
carefully. For example, by interleaving across NUMA regions to avoid
filling up a single memory bus with the shared data structure that would
later displace the smaller independent data structures that will be
accessed in parallel. You can use libnuma on Linux for this, other
operating systems do not currently provide decent ways to do this.

Write the code without worrying too much about this first.
On Apr 20, 2012 4:39 PM, "Mohammad Mirzadeh" <mirzadeh at gmail.com> wrote:

> Barry,
>
> That's quite smart and I like it. Aside from the disadvantage that you
> mentioned (which to be honest is not quite straight forward but doable) I
> have the following question.
>
> When I do computations on, say (rank%16 == 0) processes, do they have
> access to the whole 32 GB memory or are they still bounded by the 2GB/core?
> ... Or in a more broad sense, am I mistaken to assume that each core only
> has access to 2GB?
>
> Thanks for your support.
>
> On Fri, Apr 20, 2012 at 2:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Mohammad,
>>
>>     Short term for what you can do NOW.
>>
>>     PETSc wants to have one MPI process for core; so start up the program
>> that way, (say there are 16 cores per node.  In the block of code that does
>> your "non-MPI stuff" do
>>
>>     if ((rank % 16) == 0) {  /* this code is only running on one MPI
>> process per node */
>>          build your mesh grid data structure and process it, use MPI
>> pramas whatever to parallelize that computation, have a big data structure
>>  */
>>     }
>>     have the (rank % 16) == 0  MPI processes send the grid information to
>> each (rank % 16) == j MPI process the part of the grid information they
>> need.
>>     have the (rank % 16) == 0  MPI processes delete the global big data
>> structure it built.
>>
>>     The rest of the program runs as a regular MPI PETSc program
>>
>>     The advantage of this approach is 1) it will run today 2) it doesn't
>> depend on any fragile os features or software. The disadvantage is that you
>> need to figure out what part of the grid data each process needs and ship
>> it from the (rank % 16) == 0  MPI processes.
>>
>>
>>    Barry
>>
>>
>>
>>
>>
>> On Apr 20, 2012, at 1:31 PM, Mohammad Mirzadeh wrote:
>>
>> > Hi guys,
>> >
>> > I have seen multiple emails regarding this in the mailing list and I'm
>> afraid you might have already answered this question but I'm not quite sure!
>> >
>> > I have objects in my code that are hard(er) to parallelize using MPI
>> and so far my strategy has been to just handle them in serial such that
>> each process has a copy of the whole thing. This object is related to my
>> grid generation/information  etc so it only needs to be done once at the
>> beginning (no moving mesh for NOW). As a result I do not care much about
>> the speed since its nothing compared to the overall solution time. However,
>> I do care about the memory that this object consumes and can limit my
>> problem size.
>> >
>> > So I had the following idea the other day. Is it possible/good idea to
>> paralleize the grid generation using OpenMP so that each node (as opposed
>> to core) would share the data structure? This can save me a lot since
>> memory on nodes are shared among cores (e.g. 32 GB/node vs 2GB/core on
>> Ranger). What I'm not quite sure about is how the job is scheduled when
>> running the code via mpirun -n Np. Should Np be the total number of cores
>> or nodes?
>> >
>> > If I use, say Np = 16 processes on one node, MPI is running 16 versions
>> of the code on a single node (which has 16 cores). How does OpenMP figure
>> out how to fork? Does it fork a total of 16 threads/MPI process = 256
>> threads or is it smart to just fork a total of 16 threads/node = 1
>> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
>> when MPI and OpenMP are mixed?
>> >
>> > Do I make any sense at all?!
>> >
>> > Thanks
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/839a20a6/attachment.htm>

From mirzadeh at gmail.com  Fri Apr 20 17:18:23 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 20 Apr 2012 15:18:23 -0700
Subject: [petsc-users] Advice on OpenMP/PETSc mix
In-Reply-To: <CAM9tzS=xqsBFTWyt+butHjvreRiu2L1eKj4m+GEYWSjnym9Kdw@mail.gmail.com>
References: <CAKsPKwn49PhKR_HcAu58_d3Bh61+wPgvgdMTL_AqR09RBUo0zQ@mail.gmail.com>
	<F44DB50F-48EA-45A8-B7C6-D209C0E0BE36@mcs.anl.gov>
	<CAKsPKwmbOK2HZs6uo8J=+s3v=LeHWbHtiQ8595wi_qk7jTq9bQ@mail.gmail.com>
	<CAM9tzS=xqsBFTWyt+butHjvreRiu2L1eKj4m+GEYWSjnym9Kdw@mail.gmail.com>
Message-ID: <CAKsPKwkMQCYBguz5BtRjAD-LT4iNyBW4QzyNRK6A+-hapYdrPQ@mail.gmail.com>

Thanks Jed; I did not know I can access the whole memory.

Mohammad

On Fri, Apr 20, 2012 at 3:05 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> You have access to all the memory, though it may help do distribute it
> carefully. For example, by interleaving across NUMA regions to avoid
> filling up a single memory bus with the shared data structure that would
> later displace the smaller independent data structures that will be
> accessed in parallel. You can use libnuma on Linux for this, other
> operating systems do not currently provide decent ways to do this.
>
> Write the code without worrying too much about this first.
> On Apr 20, 2012 4:39 PM, "Mohammad Mirzadeh" <mirzadeh at gmail.com> wrote:
>
>> Barry,
>>
>> That's quite smart and I like it. Aside from the disadvantage that you
>> mentioned (which to be honest is not quite straight forward but doable) I
>> have the following question.
>>
>> When I do computations on, say (rank%16 == 0) processes, do they have
>> access to the whole 32 GB memory or are they still bounded by the 2GB/core?
>> ... Or in a more broad sense, am I mistaken to assume that each core only
>> has access to 2GB?
>>
>> Thanks for your support.
>>
>> On Fri, Apr 20, 2012 at 2:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   Mohammad,
>>>
>>>     Short term for what you can do NOW.
>>>
>>>     PETSc wants to have one MPI process for core; so start up the
>>> program that way, (say there are 16 cores per node.  In the block of code
>>> that does your "non-MPI stuff" do
>>>
>>>     if ((rank % 16) == 0) {  /* this code is only running on one MPI
>>> process per node */
>>>          build your mesh grid data structure and process it, use MPI
>>> pramas whatever to parallelize that computation, have a big data structure
>>>  */
>>>     }
>>>     have the (rank % 16) == 0  MPI processes send the grid information
>>> to each (rank % 16) == j MPI process the part of the grid information they
>>> need.
>>>     have the (rank % 16) == 0  MPI processes delete the global big data
>>> structure it built.
>>>
>>>     The rest of the program runs as a regular MPI PETSc program
>>>
>>>     The advantage of this approach is 1) it will run today 2) it doesn't
>>> depend on any fragile os features or software. The disadvantage is that you
>>> need to figure out what part of the grid data each process needs and ship
>>> it from the (rank % 16) == 0  MPI processes.
>>>
>>>
>>>    Barry
>>>
>>>
>>>
>>>
>>>
>>> On Apr 20, 2012, at 1:31 PM, Mohammad Mirzadeh wrote:
>>>
>>> > Hi guys,
>>> >
>>> > I have seen multiple emails regarding this in the mailing list and I'm
>>> afraid you might have already answered this question but I'm not quite sure!
>>> >
>>> > I have objects in my code that are hard(er) to parallelize using MPI
>>> and so far my strategy has been to just handle them in serial such that
>>> each process has a copy of the whole thing. This object is related to my
>>> grid generation/information  etc so it only needs to be done once at the
>>> beginning (no moving mesh for NOW). As a result I do not care much about
>>> the speed since its nothing compared to the overall solution time. However,
>>> I do care about the memory that this object consumes and can limit my
>>> problem size.
>>> >
>>> > So I had the following idea the other day. Is it possible/good idea to
>>> paralleize the grid generation using OpenMP so that each node (as opposed
>>> to core) would share the data structure? This can save me a lot since
>>> memory on nodes are shared among cores (e.g. 32 GB/node vs 2GB/core on
>>> Ranger). What I'm not quite sure about is how the job is scheduled when
>>> running the code via mpirun -n Np. Should Np be the total number of cores
>>> or nodes?
>>> >
>>> > If I use, say Np = 16 processes on one node, MPI is running 16
>>> versions of the code on a single node (which has 16 cores). How does OpenMP
>>> figure out how to fork? Does it fork a total of 16 threads/MPI process =
>>> 256 threads or is it smart to just fork a total of 16 threads/node = 1
>>> thread/core = 16 threads? I'm a bit confused here how the job is scheduled
>>> when MPI and OpenMP are mixed?
>>> >
>>> > Do I make any sense at all?!
>>> >
>>> > Thanks
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120420/a4947f9a/attachment-0001.htm>

From dominik at itis.ethz.ch  Sat Apr 21 12:01:34 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Sat, 21 Apr 2012 19:01:34 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
Message-ID: <CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>

Thank a lot for the hints, but I still one little clarification. The
FormFunction function has a signature

PetscErrorCode ScalarNonLinearSolver::FormFunction(SNES snes, Vec x,
Vec f, void *ctx)

where I am supposed to compute A(x), b(x) and then fill f with f(x) =
A(x)x - b(x)

The question is: should I call the ghost update on the passed argument
x, or on my own x that I maintain in my application as a part of a KSP
system? Or are they supposed to be actually the same?

Thanks a lot
Dominik

On Fri, Apr 20, 2012 at 4:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Fri, Apr 20, 2012 at 07:08, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> You can use
>>
>>
>> ??http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html
>
>
> Please don't do it this way, line searches and other methods will not work.
> A ghosted vector is defined by the global entries, not the local form.
> Update the local form when you want it to be current.

From bsmith at mcs.anl.gov  Sat Apr 21 12:10:37 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 21 Apr 2012 12:10:37 -0500
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
Message-ID: <148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>


   Domink,

     Sounds like you want to do a "Picard" iteration. You should use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetPicard.html  in petsc-dev 
     Note with this you never compute f(x) = A(x)x - b(x)   you provide a function to compute b(x) and A(x). 

     Please let us know if this new functionality doesn't serve your purpose. 

      You would do the ghost point update in the function that computes b().


    Barry


On Apr 21, 2012, at 12:01 PM, Dominik Szczerba wrote:

> Thank a lot for the hints, but I still one little clarification. The
> FormFunction function has a signature
> 
> PetscErrorCode ScalarNonLinearSolver::FormFunction(SNES snes, Vec x,
> Vec f, void *ctx)
> 
> where I am supposed to compute A(x), b(x) and then fill f with f(x) =
> A(x)x - b(x)
> 
> The question is: should I call the ghost update on the passed argument
> x, or on my own x that I maintain in my application as a part of a KSP
> system? Or are they supposed to be actually the same?
> 
> Thanks a lot
> Dominik
> 
> On Fri, Apr 20, 2012 at 4:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> On Fri, Apr 20, 2012 at 07:08, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>> You can use
>>> 
>>> 
>>>   http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html
>> 
>> 
>> Please don't do it this way, line searches and other methods will not work.
>> A ghosted vector is defined by the global entries, not the local form.
>> Update the local form when you want it to be current.


From dominik at itis.ethz.ch  Sat Apr 21 14:54:02 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Sat, 21 Apr 2012 21:54:02 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
Message-ID: <CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>

Thanks Barry for the hint.
I am indeed solving my nonlinear system using the Picard scheme.
I was once advised to set the Jacobian to my coefficient matrix "A" in
the FormJacobian function passed to SNES to achieve this goal (I am
still trying to get it to work with limited success though).
Are you suggesting I should abandon this approach in favor of the new
functionality in petsc-dev?

Regards,
Dominik

On Sat, Apr 21, 2012 at 7:10 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> ? Domink,
>
> ? ? Sounds like you want to do a "Picard" iteration. You should use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetPicard.html ?in petsc-dev
> ? ? Note with this you never compute f(x) = A(x)x - b(x) ? you provide a function to compute b(x) and A(x).
>
> ? ? Please let us know if this new functionality doesn't serve your purpose.
>
> ? ? ?You would do the ghost point update in the function that computes b().
>
>
> ? ?Barry
>
>
> On Apr 21, 2012, at 12:01 PM, Dominik Szczerba wrote:
>
>> Thank a lot for the hints, but I still one little clarification. The
>> FormFunction function has a signature
>>
>> PetscErrorCode ScalarNonLinearSolver::FormFunction(SNES snes, Vec x,
>> Vec f, void *ctx)
>>
>> where I am supposed to compute A(x), b(x) and then fill f with f(x) =
>> A(x)x - b(x)
>>
>> The question is: should I call the ghost update on the passed argument
>> x, or on my own x that I maintain in my application as a part of a KSP
>> system? Or are they supposed to be actually the same?
>>
>> Thanks a lot
>> Dominik
>>
>> On Fri, Apr 20, 2012 at 4:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>> On Fri, Apr 20, 2012 at 07:08, Matthew Knepley <knepley at gmail.com> wrote:
>>>>
>>>> You can use
>>>>
>>>>
>>>> ? http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html
>>>
>>>
>>> Please don't do it this way, line searches and other methods will not work.
>>> A ghosted vector is defined by the global entries, not the local form.
>>> Update the local form when you want it to be current.
>

From bsmith at mcs.anl.gov  Sat Apr 21 15:02:02 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 21 Apr 2012 15:02:02 -0500
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
Message-ID: <43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>


On Apr 21, 2012, at 2:54 PM, Dominik Szczerba wrote:

> Thanks Barry for the hint.
> I am indeed solving my nonlinear system using the Picard scheme.
> I was once advised to set the Jacobian to my coefficient matrix "A" in
> the FormJacobian function passed to SNES to achieve this goal (I am
> still trying to get it to work with limited success though).
> Are you suggesting I should abandon this approach in favor of the new
> functionality in petsc-dev?

  Yes if you want to do Picard then you should use SNESSetPicard() that is what it designed for.

   Barry

> Regards,
> Dominik
> 
> On Sat, Apr 21, 2012 at 7:10 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>   Domink,
>> 
>>     Sounds like you want to do a "Picard" iteration. You should use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetPicard.html  in petsc-dev
>>     Note with this you never compute f(x) = A(x)x - b(x)   you provide a function to compute b(x) and A(x).
>> 
>>     Please let us know if this new functionality doesn't serve your purpose.
>> 
>>      You would do the ghost point update in the function that computes b().
>> 
>> 
>>    Barry
>> 
>> 
>> On Apr 21, 2012, at 12:01 PM, Dominik Szczerba wrote:
>> 
>>> Thank a lot for the hints, but I still one little clarification. The
>>> FormFunction function has a signature
>>> 
>>> PetscErrorCode ScalarNonLinearSolver::FormFunction(SNES snes, Vec x,
>>> Vec f, void *ctx)
>>> 
>>> where I am supposed to compute A(x), b(x) and then fill f with f(x) =
>>> A(x)x - b(x)
>>> 
>>> The question is: should I call the ghost update on the passed argument
>>> x, or on my own x that I maintain in my application as a part of a KSP
>>> system? Or are they supposed to be actually the same?
>>> 
>>> Thanks a lot
>>> Dominik
>>> 
>>> On Fri, Apr 20, 2012 at 4:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>> On Fri, Apr 20, 2012 at 07:08, Matthew Knepley <knepley at gmail.com> wrote:
>>>>> 
>>>>> You can use
>>>>> 
>>>>> 
>>>>>   http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetUpdate.html
>>>> 
>>>> 
>>>> Please don't do it this way, line searches and other methods will not work.
>>>> A ghosted vector is defined by the global entries, not the local form.
>>>> Update the local form when you want it to be current.
>> 


From zonexo at gmail.com  Sat Apr 21 17:31:02 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 00:31:02 +0200
Subject: [petsc-users] Questions abt ex22f
Message-ID: <4F933526.9070304@gmail.com>

Hi,

May I know if ex22f is complete? I can't find :

call KSPGetSolution(ksp,x,ierr)

If I entered it, it says x not found.

Thank you!

-- 
Yours sincerely,

TAY wee-beng


From knepley at gmail.com  Sat Apr 21 18:14:36 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 21 Apr 2012 19:14:36 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F933526.9070304@gmail.com>
References: <4F933526.9070304@gmail.com>
Message-ID: <CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>

On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com> wrote:

> Hi,
>
> May I know if ex22f is complete? I can't find :
>
> call KSPGetSolution(ksp,x,ierr)
>
> If I entered it, it says x not found.
>

This is correct in petsc-dev. What version are you using?

  Thanks,

    Matt


> Thank you!
>
> --
> Yours sincerely,
>
> TAY wee-beng
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120421/b37c7f06/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 02:31:29 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 09:31:29 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
Message-ID: <4F93B3D1.2060304@gmail.com>

Hi,

I am using petsc-dev 2012-04-20.

Btw, I'm referring to :

http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html

Part of the code is :

call KSPSetFromOptions(ksp,ierr)
       call KSPSetUp(ksp,ierr)
       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
       call KSPDestroy(ksp,ierr)
       call DMDestroy(da,ierr)
       call PetscFinalize(ierr)



Unlike other codes like ex29c or ex45c, there isn't a "call 
KSPGetSolution(ksp,x,ierr)"

Also I want to add "call VecView(x,ierr)" to print out the results, 
which is usally added after the above.

Thank you

Yours sincerely,

TAY wee-beng


On 22/4/2012 1:14 AM, Matthew Knepley wrote:
> On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     May I know if ex22f is complete? I can't find :
>
>     call KSPGetSolution(ksp,x,ierr)
>
>     If I entered it, it says x not found.
>
>
> This is correct in petsc-dev. What version are you using?
>
>   Thanks,
>
>     Matt
>
>     Thank you!
>
>     -- 
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/433fdb62/attachment.htm>

From knepley at gmail.com  Sun Apr 22 07:53:56 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 08:53:56 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F93B3D1.2060304@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
Message-ID: <CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>

On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I am using petsc-dev 2012-04-20.
>
> Btw, I'm referring to :
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>
> Part of the code is :
>
> call KSPSetFromOptions(ksp,ierr)
>       call KSPSetUp(ksp,ierr)
>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>       call KSPDestroy(ksp,ierr)
>       call DMDestroy(da,ierr)
>       call PetscFinalize(ierr)
>
>
>
> Unlike other codes like ex29c or ex45c, there isn't a "call
> KSPGetSolution(ksp,x,ierr)"
>

You need to declare "Vec x", and then you can call KSPGetSolution(ksp, x,
ierr)

   Matt


> Also I want to add "call VecView(x,ierr)" to print out the results, which
> is usally added after the above.
>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>
> On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Hi,
>>
>> May I know if ex22f is complete? I can't find :
>>
>> call KSPGetSolution(ksp,x,ierr)
>>
>> If I entered it, it says x not found.
>>
>
>  This is correct in petsc-dev. What version are you using?
>
>    Thanks,
>
>      Matt
>
>
>> Thank you!
>>
>> --
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/678d2258/attachment-0001.htm>

From dominik at itis.ethz.ch  Sun Apr 22 10:35:52 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Sun, 22 Apr 2012 17:35:52 +0200
Subject: [petsc-users] VecCopy on ghosted vectors
Message-ID: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>

I do

VecGhostUpdateBegin(x1, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateEnd(x1, INSERT_VALUES, SCATTER_FORWARD);

then

VecCopy(x1,x2)

to find that x2 does not have correct ghost values and I need to explicitly call

VecGhostUpdateBegin(x2, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateEnd(x2, INSERT_VALUES, SCATTER_FORWARD);

Is there a rationale for this behavior? I would suggest to add a note
to the VecCopy man page about it.

Thanks
Dominik

From knepley at gmail.com  Sun Apr 22 11:01:54 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 12:01:54 -0400
Subject: [petsc-users] VecCopy on ghosted vectors
In-Reply-To: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
References: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
Message-ID: <CAMYG4Gn_AjLO9=Ax66Pp78oYDvq=E7ZmpVeumpEC6HTDa5vb3g@mail.gmail.com>

On Sun, Apr 22, 2012 at 11:35 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I do
>
> VecGhostUpdateBegin(x1, INSERT_VALUES, SCATTER_FORWARD);
> VecGhostUpdateEnd(x1, INSERT_VALUES, SCATTER_FORWARD);
>
> then
>
> VecCopy(x1,x2)
>
> to find that x2 does not have correct ghost values and I need to
> explicitly call
>
> VecGhostUpdateBegin(x2, INSERT_VALUES, SCATTER_FORWARD);
> VecGhostUpdateEnd(x2, INSERT_VALUES, SCATTER_FORWARD);
>
> Is there a rationale for this behavior? I would suggest to add a note
> to the VecCopy man page about it.
>

VecCopy works for all vector types, and thus does not get extra information
for
a specific type, and a Vec is defined by the global values, not local
copies. Moreover,
although VecGhost is available, the recommended paradigm is to use local
Vecs and
GlobalToLocal().

   Matt


> Thanks
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/045f06bf/attachment.htm>

From bsmith at mcs.anl.gov  Sun Apr 22 11:03:03 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 22 Apr 2012 11:03:03 -0500
Subject: [petsc-users] VecCopy on ghosted vectors
In-Reply-To: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
References: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
Message-ID: <AF2AD772-81BE-4BD7-A2C9-3007F91EE175@mcs.anl.gov>


Dominik,

   If you want to copy the ghost point values, which DO NO exist in the x1 and x2 vectors, you need to use VecCopy(xlocal1,xlocal2) where xlocal1 and xlocal2 are obtained with VecGetLocalForm() on x1 and x2.

   The rational is that x1 and x2 are just regular parallel vectors, copying their entries is just copying their entries, not the "extra" slots that are ONLY available via the local form.

    You are mis-thinking about how "ghosted" vectors work.


   Barry

  
On Apr 22, 2012, at 10:35 AM, Dominik Szczerba wrote:

> I do
> 
> VecGhostUpdateBegin(x1, INSERT_VALUES, SCATTER_FORWARD);
> VecGhostUpdateEnd(x1, INSERT_VALUES, SCATTER_FORWARD);
> 
> then
> 
> VecCopy(x1,x2)
> 
> to find that x2 does not have correct ghost values and I need to explicitly call
> 
> VecGhostUpdateBegin(x2, INSERT_VALUES, SCATTER_FORWARD);
> VecGhostUpdateEnd(x2, INSERT_VALUES, SCATTER_FORWARD);
> 
> Is there a rationale for this behavior? I would suggest to add a note
> to the VecCopy man page about it.
> 
> Thanks
> Dominik


From bsmith at mcs.anl.gov  Sun Apr 22 11:07:32 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 22 Apr 2012 11:07:32 -0500
Subject: [petsc-users] VecCopy on ghosted vectors
In-Reply-To: <AF2AD772-81BE-4BD7-A2C9-3007F91EE175@mcs.anl.gov>
References: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
	<AF2AD772-81BE-4BD7-A2C9-3007F91EE175@mcs.anl.gov>
Message-ID: <C615FC7C-F8DC-4674-8D65-137CB31AAAF1@mcs.anl.gov>


   Think of a ghosted vector and its local form as two completely different vectors. 

   Barry

  They just happen to have some common memory area for efficiency, but that is only for efficiency.


On Apr 22, 2012, at 11:03 AM, Barry Smith wrote:

> 
> Dominik,
> 
>   If you want to copy the ghost point values, which DO NO exist in the x1 and x2 vectors, you need to use VecCopy(xlocal1,xlocal2) where xlocal1 and xlocal2 are obtained with VecGetLocalForm() on x1 and x2.
> 
>   The rational is that x1 and x2 are just regular parallel vectors, copying their entries is just copying their entries, not the "extra" slots that are ONLY available via the local form.
> 
>    You are mis-thinking about how "ghosted" vectors work.
> 
> 
>   Barry
> 
> 
> On Apr 22, 2012, at 10:35 AM, Dominik Szczerba wrote:
> 
>> I do
>> 
>> VecGhostUpdateBegin(x1, INSERT_VALUES, SCATTER_FORWARD);
>> VecGhostUpdateEnd(x1, INSERT_VALUES, SCATTER_FORWARD);
>> 
>> then
>> 
>> VecCopy(x1,x2)
>> 
>> to find that x2 does not have correct ghost values and I need to explicitly call
>> 
>> VecGhostUpdateBegin(x2, INSERT_VALUES, SCATTER_FORWARD);
>> VecGhostUpdateEnd(x2, INSERT_VALUES, SCATTER_FORWARD);
>> 
>> Is there a rationale for this behavior? I would suggest to add a note
>> to the VecCopy man page about it.
>> 
>> Thanks
>> Dominik
> 


From dominik at itis.ethz.ch  Sun Apr 22 11:13:12 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Sun, 22 Apr 2012 18:13:12 +0200
Subject: [petsc-users] VecCopy on ghosted vectors
In-Reply-To: <C615FC7C-F8DC-4674-8D65-137CB31AAAF1@mcs.anl.gov>
References: <CALm7otoZnRrCkV8+Z+A857T3-9O2ALKOyjY-J2O-2O_VZLbHDQ@mail.gmail.com>
	<AF2AD772-81BE-4BD7-A2C9-3007F91EE175@mcs.anl.gov>
	<C615FC7C-F8DC-4674-8D65-137CB31AAAF1@mcs.anl.gov>
Message-ID: <CALm7otpANkjMmZVE355vCK4gLcVQyeqsv8Hrpt6pmTWmPt974w@mail.gmail.com>

Thank you for your explanations, it makes sense. I would suggest to
add a note of warning to the man page about this behavior.

Regards,
Dominik

From zonexo at gmail.com  Sun Apr 22 14:03:16 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 21:03:16 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
Message-ID: <4F9455F4.9050704@gmail.com>

Hi,

I added "Vec x,b" after "KSP ksp"
and then "call KSPGetSolution(ksp, x, ierr)"

I wanted to see the output so I added "call VecView(x,ierr)" but I got 
this error:

[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 
13:45:54 CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22 
21:02:14 2012
[0]PETSC ERROR: Libraries linked from 
/home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
[0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
[0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ 
--with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/ 
--with-debugging=0 --download-hypre=1 
--prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3 
FOPTFLAGS=-O3 --download-mumps=1 --download-parmetis=1 
--download-scalapack=1 --download-blacs=1
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.


Yours sincerely,

TAY wee-beng


On 22/4/2012 2:53 PM, Matthew Knepley wrote:
> On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I am using petsc-dev 2012-04-20.
>
>     Btw, I'm referring to :
>
>     http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>
>     Part of the code is :
>
>     call KSPSetFromOptions(ksp,ierr)
>            call KSPSetUp(ksp,ierr)
>            call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>            call KSPDestroy(ksp,ierr)
>            call DMDestroy(da,ierr)
>            call PetscFinalize(ierr)
>
>
>
>     Unlike other codes like ex29c or ex45c, there isn't a "call
>     KSPGetSolution(ksp,x,ierr)"
>
>
> You need to declare "Vec x", and then you can call KSPGetSolution(ksp, 
> x, ierr)
>
>    Matt
>
>     Also I want to add "call VecView(x,ierr)" to print out the
>     results, which is usally added after the above.
>
>     Thank you
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>     On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         May I know if ex22f is complete? I can't find :
>>
>>         call KSPGetSolution(ksp,x,ierr)
>>
>>         If I entered it, it says x not found.
>>
>>
>>     This is correct in petsc-dev. What version are you using?
>>
>>       Thanks,
>>
>>         Matt
>>
>>         Thank you!
>>
>>         -- 
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/dc9d912b/attachment.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 14:06:47 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 14:06:47 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F9455F4.9050704@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
Message-ID: <CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>

Run in a debugger and/or use --with-debugging=1 so that the error trace has
more information. You could also show us the exact code that you used.
On Apr 22, 2012 2:03 PM, "TAY wee-beng" <zonexo at gmail.com> wrote:

>  Hi,
>
> I added "Vec x,b" after "KSP ksp"
> and then "call KSPGetSolution(ksp, x, ierr)"
>
> I wanted to see the output so I added "call VecView(x,ierr)" but I got
> this error:
>
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22
> 21:02:14 2012
> [0]PETSC ERROR: Libraries linked from
> /home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
> [0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
> --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
> --with-debugging=0 --download-hypre=1
> --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3 FOPTFLAGS=-O3
> --download-mumps=1 --download-parmetis=1 --download-scalapack=1
> --download-blacs=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 22/4/2012 2:53 PM, Matthew Knepley wrote:
>
> On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I am using petsc-dev 2012-04-20.
>>
>> Btw, I'm referring to :
>>
>>
>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>>
>> Part of the code is :
>>
>> call KSPSetFromOptions(ksp,ierr)
>>       call KSPSetUp(ksp,ierr)
>>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>       call KSPDestroy(ksp,ierr)
>>       call DMDestroy(da,ierr)
>>       call PetscFinalize(ierr)
>>
>>
>>
>> Unlike other codes like ex29c or ex45c, there isn't a "call
>> KSPGetSolution(ksp,x,ierr)"
>>
>
>  You need to declare "Vec x", and then you can call KSPGetSolution(ksp,
> x, ierr)
>
>     Matt
>
>
>>  Also I want to add "call VecView(x,ierr)" to print out the results,
>> which is usally added after the above.
>>
>> Thank you
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>
>> On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> May I know if ex22f is complete? I can't find :
>>>
>>> call KSPGetSolution(ksp,x,ierr)
>>>
>>> If I entered it, it says x not found.
>>>
>>
>>  This is correct in petsc-dev. What version are you using?
>>
>>    Thanks,
>>
>>      Matt
>>
>>
>>> Thank you!
>>>
>>> --
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/faa2d036/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 14:18:55 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 21:18:55 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
Message-ID: <4F94599F.3040504@gmail.com>

Hi,

I have attached the ex22f.F file. The changes I added are given in bold:

...

PetscErrorCode   ierr
       DM               da
       KSP              ksp
*Vec              x,b*
       external         ComputeRHS,ComputeMatrix

....

call KSPSetUp(ksp,ierr)
       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
*call KSPGetSolution(ksp,x,ierr)
       call VecView(x,ierr)*
       call KSPDestroy(ksp,ierr)
       call DMDestroy(da,ierr)

The error is:

[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecView line 735 
/home/wtay/Codes/petsc-3.2-p5/src/vec/vec/interface/vector.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 
13:45:54 CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22 
21:11:39 2012
[0]PETSC ERROR: Libraries linked from 
/home/wtay/Lib/petsc-3.2-p5_mumps_debug/lib
[0]PETSC ERROR: Configure run at Sun Nov 27 15:39:26 2011
[0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ 
--with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/ 
--with-debugging=1 --download-hypre=1 
--prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_debug COPTFLAGS=-O0 
FOPTFLAGS=-O0 --download-mumps=1 --download-parmetis=1 
--download-scalapack=1 --download-blacs=1
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

Yours sincerely,

TAY wee-beng


On 22/4/2012 9:06 PM, Jed Brown wrote:
>
> Run in a debugger and/or use --with-debugging=1 so that the error 
> trace has more information. You could also show us the exact code that 
> you used.
>
> On Apr 22, 2012 2:03 PM, "TAY wee-beng" <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I added "Vec x,b" after "KSP ksp"
>     and then "call KSPGetSolution(ksp, x, ierr)"
>
>     I wanted to see the output so I added "call VecView(x,ierr)" but I
>     got this error:
>
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
>     to find memory corruption errors
>     [0]PETSC ERROR: configure using --with-debugging=yes, recompile,
>     link, and run
>     [0]PETSC ERROR: to get more information on the crash.
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------------
>     [0]PETSC ERROR: Signal received!
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29
>     13:45:54 CDT 2011
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr
>     22 21:02:14 2012
>     [0]PETSC ERROR: Libraries linked from
>     /home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
>     [0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
>     [0]PETSC ERROR: Configure options
>     --with-mpi-dir=/opt/openmpi-1.5.3/
>     --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>     --with-debugging=0 --download-hypre=1
>     --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3
>     FOPTFLAGS=-O3 --download-mumps=1 --download-parmetis=1
>     --download-scalapack=1 --download-blacs=1
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: User provided function() line 0 in unknown
>     directory unknown file
>     --------------------------------------------------------------------------
>     MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>     with errorcode 59.
>
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 22/4/2012 2:53 PM, Matthew Knepley wrote:
>>     On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I am using petsc-dev 2012-04-20.
>>
>>         Btw, I'm referring to :
>>
>>         http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>>
>>         Part of the code is :
>>
>>         call KSPSetFromOptions(ksp,ierr)
>>                call KSPSetUp(ksp,ierr)
>>                call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>                call KSPDestroy(ksp,ierr)
>>                call DMDestroy(da,ierr)
>>                call PetscFinalize(ierr)
>>
>>
>>
>>         Unlike other codes like ex29c or ex45c, there isn't a "call
>>         KSPGetSolution(ksp,x,ierr)"
>>
>>
>>     You need to declare "Vec x", and then you can call
>>     KSPGetSolution(ksp, x, ierr)
>>
>>        Matt
>>
>>         Also I want to add "call VecView(x,ierr)" to print out the
>>         results, which is usally added after the above.
>>
>>         Thank you
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>>         On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             May I know if ex22f is complete? I can't find :
>>>
>>>             call KSPGetSolution(ksp,x,ierr)
>>>
>>>             If I entered it, it says x not found.
>>>
>>>
>>>         This is correct in petsc-dev. What version are you using?
>>>
>>>           Thanks,
>>>
>>>             Matt
>>>
>>>             Thank you!
>>>
>>>             -- 
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/e6cf1dee/attachment-0001.htm>
-------------- next part --------------
!
!   Laplacian in 3D. Modeled by the partial differential equation
!
!   Laplacian u = 1,0 < x,y,z < 1,
!
!   with boundary conditions
!
!   u = 1 for x = 0, x = 1, y = 0, y = 1, z = 0, z = 1.
!
!   This uses multigrid to solve the linear system

      program main
      implicit none

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                    Include files
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
!     petscsys.h  - base PETSc routines      petscvec.h - vectors
!     petscmat.h - matrices
!     petscksp.h    - Krylov subspace methods  petscpc.h  - preconditioners

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscpc.h>
#include <finclude/petscksp.h>
#include <finclude/petscdmda.h>
      PetscErrorCode   ierr
      DM               da
      KSP              ksp
      Vec              x,b
      external         ComputeRHS,ComputeMatrix
      PetscInt i1,i3

      call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)

      i3 = -3
      i1 = 1
      call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
      call DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,            &
     &    DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,                               &
     &    DMDA_STENCIL_STAR,i3,i3,i3,PETSC_DECIDE,PETSC_DECIDE,                        &
     &    PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                        &
     &                PETSC_NULL_INTEGER,da,ierr)  
      call DMSetFunction(da,ComputeRHS,ierr)
      call DMSetJacobian(da,ComputeMatrix,ierr)   
      call KSPSetDM(ksp,da,ierr)

      call KSPSetFromOptions(ksp,ierr)
      call KSPSetUp(ksp,ierr)
      call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
      call KSPGetSolution(ksp,x,ierr)
      call VecView(x,ierr)
      call KSPDestroy(ksp,ierr)
      call DMDestroy(da,ierr)
      call PetscFinalize(ierr)

      end

      subroutine ComputeRHS(da,x,b,ierr)
      implicit none

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscdmda.h>

      PetscErrorCode  ierr
      PetscInt mx,my,mz
      PetscScalar  h
      Vec          x,b
      DM           da

      call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,mz,                        &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                 &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                 &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                 &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                 &
     &               PETSC_NULL_INTEGER,ierr)
      h    = 1.d0/((mx-1)*(my-1)*(mz-1))

      call VecSet(b,h,ierr)
      return
      end

    
      subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
      implicit none

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscdmda.h>

      Mat          jac,JJ
      PetscErrorCode    ierr

      DM           da
      PetscInt    i,j,k,mx,my,mz,xm
      PetscInt    ym,zm,xs,ys,zs,i1,i7
      PetscScalar  v(7),Hx,Hy,Hz
      PetscScalar  HxHydHz,HyHzdHx
      PetscScalar  HxHzdHy
      MatStencil   row(4),col(4,7)
      Vec          x
      MatStructure str
      i1 = 1
      i7 = 7
      call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,mz,                       &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                &
     &               PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,                &
     &               PETSC_NULL_INTEGER,ierr)

      Hx = 1.d0 / (mx-1)
      Hy = 1.d0 / (my-1)
      Hz = 1.d0 / (mz-1)
      HxHydHz = Hx*Hy/Hz
      HxHzdHy = Hx*Hz/Hy
      HyHzdHx = Hy*Hz/Hx
      call DMDAGetCorners(da,xs,ys,zs,xm,ym,zm,ierr)        
  
      do 10,k=zs,zs+zm-1    
        do 20,j=ys,ys+ym-1
          do 30,i=xs,xs+xm-1
          row(MatStencil_i) = i  
          row(MatStencil_j) = j
          row(MatStencil_k) = k 
          if (i.eq.0 .or. j.eq.0 .or. k.eq.0 .or. i.eq.mx-1 .or.         &
     &         j.eq.my-1 .or. k.eq.mz-1) then
            v(1) = 2.d0*(HxHydHz + HxHzdHy + HyHzdHx)
            call MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,    &
     &                               ierr)
          else 
            v(1) = -HxHydHz
             col(MatStencil_i,1) = i
             col(MatStencil_j,1) = j
             col(MatStencil_k,1) = k-1
            v(2) = -HxHzdHy
             col(MatStencil_i,2) = i
             col(MatStencil_j,2) = j-1
             col(MatStencil_k,2) = k
            v(3) = -HyHzdHx
             col(MatStencil_i,3) = i-1 
             col(MatStencil_j,3) = j
             col(MatStencil_k,3) = k
            v(4) = 2.d0*(HxHydHz + HxHzdHy + HyHzdHx)
             col(MatStencil_i,4) = i 
             col(MatStencil_j,4) = j
             col(MatStencil_k,4) = k
            v(5) = -HyHzdHx
             col(MatStencil_i,5) = i+1 
             col(MatStencil_j,5) = j 
             col(MatStencil_k,5) = k
            v(6) = -HxHzdHy
             col(MatStencil_i,6) = i 
             col(MatStencil_j,6) = j+1 
             col(MatStencil_k,6) = k
            v(7) = -HxHydHz
             col(MatStencil_i,7) = i 
             col(MatStencil_j,7) = j 
             col(MatStencil_k,7) = k+1
      call MatSetValuesStencil(jac,i1,row,i7,col,v,INSERT_VALUES,               &
     &                               ierr)
          endif
 30       continue
 20     continue
 10   continue

      call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
      return 
      end

From jedbrown at mcs.anl.gov  Sun Apr 22 14:20:48 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 14:20:48 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F94599F.3040504@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
Message-ID: <CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>

Look at the man page for VecView, it needs a PetscViewer argument.
On Apr 22, 2012 2:19 PM, "TAY wee-beng" <zonexo at gmail.com> wrote:

>  Hi,
>
> I have attached the ex22f.F file. The changes I added are given in bold:
>
> ...
>
> PetscErrorCode   ierr
>       DM               da
>       KSP              ksp
>      * Vec              x,b*
>       external         ComputeRHS,ComputeMatrix
>
> ....
>
> call KSPSetUp(ksp,ierr)
>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>       *call KSPGetSolution(ksp,x,ierr)
>       call VecView(x,ierr)*
>       call KSPDestroy(ksp,ierr)
>       call DMDestroy(da,ierr)
>
> The error is:
>
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] VecView line 735
> /home/wtay/Codes/petsc-3.2-p5/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22
> 21:11:39 2012
> [0]PETSC ERROR: Libraries linked from
> /home/wtay/Lib/petsc-3.2-p5_mumps_debug/lib
> [0]PETSC ERROR: Configure run at Sun Nov 27 15:39:26 2011
> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
> --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
> --with-debugging=1 --download-hypre=1
> --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_debug COPTFLAGS=-O0
> FOPTFLAGS=-O0 --download-mumps=1 --download-parmetis=1
> --download-scalapack=1 --download-blacs=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 22/4/2012 9:06 PM, Jed Brown wrote:
>
> Run in a debugger and/or use --with-debugging=1 so that the error trace
> has more information. You could also show us the exact code that you used.
> On Apr 22, 2012 2:03 PM, "TAY wee-beng" <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I added "Vec x,b" after "KSP ksp"
>> and then "call KSPGetSolution(ksp, x, ierr)"
>>
>> I wanted to see the output so I added "call VecView(x,ierr)" but I got
>> this error:
>>
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>> and run
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
>> CDT 2011
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22
>> 21:02:14 2012
>> [0]PETSC ERROR: Libraries linked from
>> /home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
>> [0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>> --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>> --with-debugging=0 --download-hypre=1
>> --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3 FOPTFLAGS=-O3
>> --download-mumps=1 --download-parmetis=1 --download-scalapack=1
>> --download-blacs=1
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> with errorcode 59.
>>
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 22/4/2012 2:53 PM, Matthew Knepley wrote:
>>
>> On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I am using petsc-dev 2012-04-20.
>>>
>>> Btw, I'm referring to :
>>>
>>>
>>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>>>
>>> Part of the code is :
>>>
>>> call KSPSetFromOptions(ksp,ierr)
>>>       call KSPSetUp(ksp,ierr)
>>>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>       call KSPDestroy(ksp,ierr)
>>>       call DMDestroy(da,ierr)
>>>       call PetscFinalize(ierr)
>>>
>>>
>>>
>>> Unlike other codes like ex29c or ex45c, there isn't a "call
>>> KSPGetSolution(ksp,x,ierr)"
>>>
>>
>>  You need to declare "Vec x", and then you can call KSPGetSolution(ksp,
>> x, ierr)
>>
>>     Matt
>>
>>
>>>  Also I want to add "call VecView(x,ierr)" to print out the results,
>>> which is usally added after the above.
>>>
>>> Thank you
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>>
>>> On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> May I know if ex22f is complete? I can't find :
>>>>
>>>> call KSPGetSolution(ksp,x,ierr)
>>>>
>>>> If I entered it, it says x not found.
>>>>
>>>
>>>  This is correct in petsc-dev. What version are you using?
>>>
>>>    Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thank you!
>>>>
>>>> --
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/5f11b897/attachment.htm>

From knepley at gmail.com  Sun Apr 22 14:28:39 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 15:28:39 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
Message-ID: <CAMYG4GkQQwLifk51gnA_C10tWOqFJ2StFRO=NVFS4QPb9kuerg@mail.gmail.com>

On Sun, Apr 22, 2012 at 3:20 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Look at the man page for VecView, it needs a PetscViewer argument.
>
Works for me with VecView(x, PETSC_NULL_OBJECT, ierr), although it is 0

   Matt

> On Apr 22, 2012 2:19 PM, "TAY wee-beng" <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I have attached the ex22f.F file. The changes I added are given in bold:
>>
>> ...
>>
>> PetscErrorCode   ierr
>>       DM               da
>>       KSP              ksp
>>      * Vec              x,b*
>>       external         ComputeRHS,ComputeMatrix
>>
>> ....
>>
>> call KSPSetUp(ksp,ierr)
>>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>       *call KSPGetSolution(ksp,x,ierr)
>>       call VecView(x,ierr)*
>>       call KSPDestroy(ksp,ierr)
>>       call DMDestroy(da,ierr)
>>
>> The error is:
>>
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] VecView line 735
>> /home/wtay/Codes/petsc-3.2-p5/src/vec/vec/interface/vector.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
>> CDT 2011
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22
>> 21:11:39 2012
>> [0]PETSC ERROR: Libraries linked from
>> /home/wtay/Lib/petsc-3.2-p5_mumps_debug/lib
>> [0]PETSC ERROR: Configure run at Sun Nov 27 15:39:26 2011
>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>> --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>> --with-debugging=1 --download-hypre=1
>> --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_debug COPTFLAGS=-O0
>> FOPTFLAGS=-O0 --download-mumps=1 --download-parmetis=1
>> --download-scalapack=1 --download-blacs=1
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> with errorcode 59.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 22/4/2012 9:06 PM, Jed Brown wrote:
>>
>> Run in a debugger and/or use --with-debugging=1 so that the error trace
>> has more information. You could also show us the exact code that you used.
>> On Apr 22, 2012 2:03 PM, "TAY wee-beng" <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I added "Vec x,b" after "KSP ksp"
>>> and then "call KSPGetSolution(ksp, x, ierr)"
>>>
>>> I wanted to see the output so I added "call VecView(x,ierr)" but I got
>>> this error:
>>>
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>>> find memory corruption errors
>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
>>> and run
>>> [0]PETSC ERROR: to get more information on the crash.
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Signal received!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29
>>> 13:45:54 CDT 2011
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22
>>> 21:02:14 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
>>> [0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>>> --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>>> --with-debugging=0 --download-hypre=1
>>> --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3 FOPTFLAGS=-O3
>>> --download-mumps=1 --download-parmetis=1 --download-scalapack=1
>>> --download-blacs=1
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>>> unknown file
>>>
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 22/4/2012 2:53 PM, Matthew Knepley wrote:
>>>
>>> On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I am using petsc-dev 2012-04-20.
>>>>
>>>> Btw, I'm referring to :
>>>>
>>>>
>>>> http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>>>>
>>>> Part of the code is :
>>>>
>>>> call KSPSetFromOptions(ksp,ierr)
>>>>       call KSPSetUp(ksp,ierr)
>>>>       call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>>       call KSPDestroy(ksp,ierr)
>>>>       call DMDestroy(da,ierr)
>>>>       call PetscFinalize(ierr)
>>>>
>>>>
>>>>
>>>> Unlike other codes like ex29c or ex45c, there isn't a "call
>>>> KSPGetSolution(ksp,x,ierr)"
>>>>
>>>
>>>  You need to declare "Vec x", and then you can call KSPGetSolution(ksp,
>>> x, ierr)
>>>
>>>     Matt
>>>
>>>
>>>>  Also I want to add "call VecView(x,ierr)" to print out the results,
>>>> which is usally added after the above.
>>>>
>>>> Thank you
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>>>
>>>> On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> May I know if ex22f is complete? I can't find :
>>>>>
>>>>> call KSPGetSolution(ksp,x,ierr)
>>>>>
>>>>> If I entered it, it says x not found.
>>>>>
>>>>
>>>>  This is correct in petsc-dev. What version are you using?
>>>>
>>>>    Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thank you!
>>>>>
>>>>> --
>>>>> Yours sincerely,
>>>>>
>>>>> TAY wee-beng
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/7fad2d54/attachment-0001.htm>

From zonexo at gmail.com  Sun Apr 22 14:30:48 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 21:30:48 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
Message-ID: <4F945C68.5090201@gmail.com>

Hi,

Sorry for the mistake. I tried again, changing to "call VecView(x,0,ierr)"

but still got the same error:

[wtay at hpc12:tutorials]$ ./ex22f
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecView line 735 
/home/wtay/Codes/petsc-3.2-p5/src/vec/vec/interface/vector.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 
13:45:54 CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr 22 
21:25:01 2012
[0]PETSC ERROR: Libraries linked from 
/home/wtay/Lib/petsc-3.2-p5_mumps_debug/lib
[0]PETSC ERROR: Configure run at Sun Nov 27 15:39:26 2011
[0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ 
--with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/ 
--with-debugging=1 --download-hypre=1 
--prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_debug COPTFLAGS=-O0 
FOPTFLAGS=-O0 --download-mumps=1 --download-parmetis=1 
--download-scalapack=1 --download-blacs=1
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------

Yours sincerely,

TAY wee-beng


On 22/4/2012 9:20 PM, Jed Brown wrote:
>
> Look at the man page for VecView, it needs a PetscViewer argument.
>
> On Apr 22, 2012 2:19 PM, "TAY wee-beng" <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I have attached the ex22f.F file. The changes I added are given in
>     bold:
>
>     ...
>
>     PetscErrorCode   ierr
>           DM               da
>           KSP              ksp
>     *Vec              x,b*
>           external         ComputeRHS,ComputeMatrix
>
>     ....
>
>     call KSPSetUp(ksp,ierr)
>           call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>     *call KSPGetSolution(ksp,x,ierr)
>           call VecView(x,ierr)*
>           call KSPDestroy(ksp,ierr)
>           call DMDestroy(da,ierr)
>
>     The error is:
>
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
>     to find memory corruption errors
>     [0]PETSC ERROR: likely location of problem given in stack below
>     [0]PETSC ERROR: ---------------------  Stack Frames
>     ------------------------------------
>     [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>     available,
>     [0]PETSC ERROR:       INSTEAD the line number of the start of the
>     function
>     [0]PETSC ERROR:       is given.
>     [0]PETSC ERROR: [0] VecView line 735
>     /home/wtay/Codes/petsc-3.2-p5/src/vec/vec/interface/vector.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------------
>     [0]PETSC ERROR: Signal received!
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29
>     13:45:54 CDT 2011
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay Sun Apr
>     22 21:11:39 2012
>     [0]PETSC ERROR: Libraries linked from
>     /home/wtay/Lib/petsc-3.2-p5_mumps_debug/lib
>     [0]PETSC ERROR: Configure run at Sun Nov 27 15:39:26 2011
>     [0]PETSC ERROR: Configure options
>     --with-mpi-dir=/opt/openmpi-1.5.3/
>     --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>     --with-debugging=1 --download-hypre=1
>     --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_debug COPTFLAGS=-O0
>     FOPTFLAGS=-O0 --download-mumps=1 --download-parmetis=1
>     --download-scalapack=1 --download-blacs=1
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: User provided function() line 0 in unknown
>     directory unknown file
>     --------------------------------------------------------------------------
>     MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>     with errorcode 59.
>
>     NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>     You may or may not see output from other processes, depending on
>     exactly when Open MPI kills them.
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 22/4/2012 9:06 PM, Jed Brown wrote:
>>
>>     Run in a debugger and/or use --with-debugging=1 so that the error
>>     trace has more information. You could also show us the exact code
>>     that you used.
>>
>>     On Apr 22, 2012 2:03 PM, "TAY wee-beng" <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I added "Vec x,b" after "KSP ksp"
>>         and then "call KSPGetSolution(ksp, x, ierr)"
>>
>>         I wanted to see the output so I added "call VecView(x,ierr)"
>>         but I got this error:
>>
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>         Violation, probably memory access out of range
>>         [0]PETSC ERROR: Try option -start_in_debugger or
>>         -on_error_attach_debugger
>>         [0]PETSC ERROR: or see
>>         http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
>>         ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>         OS X to find memory corruption errors
>>         [0]PETSC ERROR: configure using --with-debugging=yes,
>>         recompile, link, and run
>>         [0]PETSC ERROR: to get more information on the crash.
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ------------------------------------
>>         [0]PETSC ERROR: Signal received!
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct
>>         29 13:45:54 CDT 2011
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: ./ex22f on a arch-linu named hpc12 by wtay
>>         Sun Apr 22 21:02:14 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /home/wtay/Lib/petsc-3.2-p5_mumps_rel/lib
>>         [0]PETSC ERROR: Configure run at Sun Nov 27 15:18:15 2011
>>         [0]PETSC ERROR: Configure options
>>         --with-mpi-dir=/opt/openmpi-1.5.3/
>>         --with-blas-lapack-dir=/opt/intel_xe_2011/mkl/lib/intel64/
>>         --with-debugging=0 --download-hypre=1
>>         --prefix=/home/wtay/Lib/petsc-3.2-p5_mumps_rel COPTFLAGS=-O3
>>         FOPTFLAGS=-O3 --download-mumps=1 --download-parmetis=1
>>         --download-scalapack=1 --download-blacs=1
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: User provided function() line 0 in unknown
>>         directory unknown file
>>         --------------------------------------------------------------------------
>>         MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>         with errorcode 59.
>>
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 22/4/2012 2:53 PM, Matthew Knepley wrote:
>>>         On Sun, Apr 22, 2012 at 3:31 AM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I am using petsc-dev 2012-04-20.
>>>
>>>             Btw, I'm referring to :
>>>
>>>             http://www.mcs.anl.gov/petsc/petsc-dev/src/ksp/ksp/examples/tutorials/ex22f.F.html
>>>
>>>             Part of the code is :
>>>
>>>             call KSPSetFromOptions(ksp,ierr)
>>>                    call KSPSetUp(ksp,ierr)
>>>                    call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>                    call KSPDestroy(ksp,ierr)
>>>                    call DMDestroy(da,ierr)
>>>                    call PetscFinalize(ierr)
>>>
>>>
>>>
>>>             Unlike other codes like ex29c or ex45c, there isn't a
>>>             "call KSPGetSolution(ksp,x,ierr)"
>>>
>>>
>>>         You need to declare "Vec x", and then you can call
>>>         KSPGetSolution(ksp, x, ierr)
>>>
>>>            Matt
>>>
>>>             Also I want to add "call VecView(x,ierr)" to print out
>>>             the results, which is usally added after the above.
>>>
>>>             Thank you
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>             On 22/4/2012 1:14 AM, Matthew Knepley wrote:
>>>>             On Sat, Apr 21, 2012 at 6:31 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 May I know if ex22f is complete? I can't find :
>>>>
>>>>                 call KSPGetSolution(ksp,x,ierr)
>>>>
>>>>                 If I entered it, it says x not found.
>>>>
>>>>
>>>>             This is correct in petsc-dev. What version are you using?
>>>>
>>>>               Thanks,
>>>>
>>>>                 Matt
>>>>
>>>>                 Thank you!
>>>>
>>>>                 -- 
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/591a0d2d/attachment.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 14:35:07 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 14:35:07 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F945C68.5090201@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
Message-ID: <CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>

On Sun, Apr 22, 2012 at 14:30, TAY wee-beng <zonexo at gmail.com> wrote:

> Sorry for the mistake. I tried again, changing to "call VecView(x,0,ierr)"
>
> but still got the same error:
>

0 has the wrong type. Use VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/649d3e6e/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 14:36:35 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 14:36:35 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAMYG4GkQQwLifk51gnA_C10tWOqFJ2StFRO=NVFS4QPb9kuerg@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<CAMYG4GkQQwLifk51gnA_C10tWOqFJ2StFRO=NVFS4QPb9kuerg@mail.gmail.com>
Message-ID: <CAM9tzSnd1hug9hdStk99k0=hVb35t_Qbm7fb9APd+Hy7ExKsGQ@mail.gmail.com>

On Sun, Apr 22, 2012 at 14:28, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Apr 22, 2012 at 3:20 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Look at the man page for VecView, it needs a PetscViewer argument.
>>
> Works for me with VecView(x, PETSC_NULL_OBJECT, ierr), although it is 0
>

Matt, are you trying to cause confusion? The posted code called VecView
with the wrong number of arguments. Also, telling any Fortran user that "0"
is even related to PETSC_NULL_OBJECT is a terrible idea because they are
totally different things on the Fortran side.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/401471b1/attachment.htm>

From knepley at gmail.com  Sun Apr 22 14:41:08 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 15:41:08 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSnd1hug9hdStk99k0=hVb35t_Qbm7fb9APd+Hy7ExKsGQ@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<CAMYG4GkQQwLifk51gnA_C10tWOqFJ2StFRO=NVFS4QPb9kuerg@mail.gmail.com>
	<CAM9tzSnd1hug9hdStk99k0=hVb35t_Qbm7fb9APd+Hy7ExKsGQ@mail.gmail.com>
Message-ID: <CAMYG4G=WD6sdE7D-b7S6piB1G1f6ajVU597cKms2LMxFGnJw3w@mail.gmail.com>

On Sun, Apr 22, 2012 at 3:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Sun, Apr 22, 2012 at 14:28, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sun, Apr 22, 2012 at 3:20 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>> Look at the man page for VecView, it needs a PetscViewer argument.
>>>
>> Works for me with VecView(x, PETSC_NULL_OBJECT, ierr), although it is 0
>>
>
> Matt, are you trying to cause confusion? The posted code called VecView
> with the wrong number of arguments. Also, telling any Fortran user that "0"
> is even related to PETSC_NULL_OBJECT is a terrible idea because they are
> totally different things on the Fortran side.
>

This is perfectly sensible:

 1) If you use PETSC_NULL_OBJECT, you get the default viewer

 2) I was talking about the solution, which is the 0 vector in my output

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/abbe80d5/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 14:47:34 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 21:47:34 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAMYG4G=WD6sdE7D-b7S6piB1G1f6ajVU597cKms2LMxFGnJw3w@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<CAMYG4GkQQwLifk51gnA_C10tWOqFJ2StFRO=NVFS4QPb9kuerg@mail.gmail.com>
	<CAM9tzSnd1hug9hdStk99k0=hVb35t_Qbm7fb9APd+Hy7ExKsGQ@mail.gmail.com>
	<CAMYG4G=WD6sdE7D-b7S6piB1G1f6ajVU597cKms2LMxFGnJw3w@mail.gmail.com>
Message-ID: <4F946056.2080009@gmail.com>

Hi,

I changed from "0" to PETSC_NULL_OBJECT. In linux, I managed to get the 
output now:

Vector Object:Vec_0x24c7f10_0 1 MPI processes
   type: mpi
Process [0]
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0833333
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667
0.0416667

Doing the same in VS2008 in windows 64bit, both "0" and 
"PETSC_NULL_OBJECT" give the results as:

Vector Object:Vec_0000000084000000_0 1 MPI
   type: mpi
Process [0]
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Why are the results different? The changes I made (as mentioned earlier) 
are the same in both.

Yours sincerely,

TAY wee-beng


On 22/4/2012 9:41 PM, Matthew Knepley wrote:
> On Sun, Apr 22, 2012 at 3:36 PM, Jed Brown <jedbrown at mcs.anl.gov 
> <mailto:jedbrown at mcs.anl.gov>> wrote:
>
>     On Sun, Apr 22, 2012 at 14:28, Matthew Knepley <knepley at gmail.com
>     <mailto:knepley at gmail.com>> wrote:
>
>         On Sun, Apr 22, 2012 at 3:20 PM, Jed Brown
>         <jedbrown at mcs.anl.gov <mailto:jedbrown at mcs.anl.gov>> wrote:
>
>             Look at the man page for VecView, it needs a PetscViewer
>             argument.
>
>         Works for me with VecView(x, PETSC_NULL_OBJECT, ierr),
>         although it is 0
>
>
>     Matt, are you trying to cause confusion? The posted code called
>     VecView with the wrong number of arguments. Also, telling any
>     Fortran user that "0" is even related to PETSC_NULL_OBJECT is a
>     terrible idea because they are totally different things on the
>     Fortran side.
>
>
> This is perfectly sensible:
>
>  1) If you use PETSC_NULL_OBJECT, you get the default viewer
>
>  2) I was talking about the solution, which is the 0 vector in my output
>
>    Matt
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/265de8e1/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 14:55:49 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 21:55:49 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
Message-ID: <4F946245.1000502@gmail.com>

Hi,

I changed the vs2008 and linux source code to 
VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) but the results for both 
outputs are 0 and 0.0416667 (except for one value which is 0.0833333) 
respectively.

Yours sincerely,

TAY wee-beng


On 22/4/2012 9:35 PM, Jed Brown wrote:
> On Sun, Apr 22, 2012 at 14:30, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Sorry for the mistake. I tried again, changing to "call
>     VecView(x,0,ierr)"
>
>     but still got the same error:
>
>
> 0 has the wrong type. Use VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/7d1d9209/attachment.htm>

From knepley at gmail.com  Sun Apr 22 15:02:58 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 16:02:58 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F946245.1000502@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
Message-ID: <CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>

On Sun, Apr 22, 2012 at 3:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I changed the vs2008 and linux source code to
> VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) but the results for both outputs
> are 0 and 0.0416667 (except for one value which is 0.0833333) respectively.
>

It looks to me like there is a bug in KSPSetUp(). The DMKSPGetContext() is
returning a null struct,
so the RHS is never formed, and the Mat is formed by the default call
to DMComputeJacobian. What
is intended here with KSPDM?

   Matt


> Yours sincerely,
>
> TAY wee-beng
>
>
> On 22/4/2012 9:35 PM, Jed Brown wrote:
>
> On Sun, Apr 22, 2012 at 14:30, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Sorry for the mistake. I tried again, changing to "call VecView(x,0,ierr)"
>>
>> but still got the same error:
>>
>
> 0 has the wrong type. Use VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/4adb5ff2/attachment.htm>

From andrew.spott at gmail.com  Sun Apr 22 15:44:43 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sun, 22 Apr 2012 14:44:43 -0600
Subject: [petsc-users] Errors in PETSc
Message-ID: <DE149237-FCDD-4E4F-AF6D-64FC0A1D2B0F@gmail.com>

How do I get line numbers to show up in the errors that petsc generates?  Instead of getting line numbers I get line=0 or just no information.

-Andrew

From jedbrown at mcs.anl.gov  Sun Apr 22 16:03:55 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 16:03:55 -0500
Subject: [petsc-users] Errors in PETSc
In-Reply-To: <DE149237-FCDD-4E4F-AF6D-64FC0A1D2B0F@gmail.com>
References: <DE149237-FCDD-4E4F-AF6D-64FC0A1D2B0F@gmail.com>
Message-ID: <CAM9tzS=+6mS9ej_moqPMorzfuMv=7vQrEXWZpfMMPjQxCwa=BQ@mail.gmail.com>

On Sun, Apr 22, 2012 at 15:44, Andrew Spott <andrew.spott at gmail.com> wrote:

> How do I get line numbers to show up in the errors that petsc generates?
>  Instead of getting line numbers I get line=0 or just no information.


In your code or in PETSc code? Usually you just configure PETSc
--with-debugging=1 (the default). In your code, you can use the CHKERRQ
macro and define __FUNCT__ and use
PetscFunctionBegin/PetscFunctionReturn(0). Most examples do this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/90f4446b/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 16:21:19 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 23:21:19 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
Message-ID: <4F94764F.80200@gmail.com>


On 22/4/2012 10:02 PM, Matthew Knepley wrote:
> On Sun, Apr 22, 2012 at 3:55 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I changed the vs2008 and linux source code to
>     VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) but the results for both
>     outputs are 0 and 0.0416667 (except for one value which is
>     0.0833333) respectively.
>
>
> It looks to me like there is a bug in KSPSetUp(). 
> The DMKSPGetContext() is returning a null struct,
> so the RHS is never formed, and the Mat is formed by the default call 
> to DMComputeJacobian. What
> is intended here with KSPDM?
Hi,

So is there a way to fix the bug? What can be done to get it working? Is 
this the only fortran example which solve the laplacian equation using 
multigrid ?

Thanks!
>
>    Matt
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 22/4/2012 9:35 PM, Jed Brown wrote:
>>     On Sun, Apr 22, 2012 at 14:30, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Sorry for the mistake. I tried again, changing to "call
>>         VecView(x,0,ierr)"
>>
>>         but still got the same error:
>>
>>
>>     0 has the wrong type. Use VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/112a285b/attachment.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 16:23:40 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 16:23:40 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F94764F.80200@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
Message-ID: <CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>

On Sun, Apr 22, 2012 at 16:21, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 22/4/2012 10:02 PM, Matthew Knepley wrote:
>
> On Sun, Apr 22, 2012 at 3:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I changed the vs2008 and linux source code to
>> VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) but the results for both outputs
>> are 0 and 0.0416667 (except for one value which is 0.0833333) respectively.
>>
>
>  It looks to me like there is a bug in KSPSetUp(). The DMKSPGetContext()
> is returning a null struct,
> so the RHS is never formed, and the Mat is formed by the default call
> to DMComputeJacobian. What
> is intended here with KSPDM?
>
> Hi?
>
> So is there a way to fix the bug? What can be done to get it working? Is
> this the only fortran example which solve the laplacian equation using
> multigrid ?
>

I'm working on it now, will report back when it's fixed. I normally
recommend src/snes/examples/tutorials/ex5f.F, even for linear problems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/0f3ec1ea/attachment.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 16:43:41 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 16:43:41 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
	<CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
Message-ID: <CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>

On Sun, Apr 22, 2012 at 16:23, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> I'm working on it now, will report back when it's fixed. I normally
> recommend src/snes/examples/tutorials/ex5f.F, even for linear problems.
>

Fixed here.

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/39d1753f550d
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/215c0b4a/attachment.htm>

From zonexo at gmail.com  Sun Apr 22 16:48:59 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 22 Apr 2012 23:48:59 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
	<CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
	<CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>
Message-ID: <4F947CCB.1020404@gmail.com>

Hi Jed,

Thanks for the quick fix. Btw, what's the easiest way to add your fix? 
Do I need to recompile?

Yours sincerely,

TAY wee-beng


On 22/4/2012 11:43 PM, Jed Brown wrote:
> On Sun, Apr 22, 2012 at 16:23, Jed Brown <jedbrown at mcs.anl.gov 
> <mailto:jedbrown at mcs.anl.gov>> wrote:
>
>     I'm working on it now, will report back when it's fixed. I
>     normally recommend src/snes/examples/tutorials/ex5f.F, even for
>     linear problems.
>
>
> Fixed here.
>
> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/39d1753f550d
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/1811317f/attachment.htm>

From knepley at gmail.com  Sun Apr 22 16:51:46 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 22 Apr 2012 17:51:46 -0400
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F947CCB.1020404@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
	<CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
	<CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>
	<4F947CCB.1020404@gmail.com>
Message-ID: <CAMYG4GnLTGUbEhKHxEngqjqREhG_Fa4jFMUkrAasOSB2K9raVQ@mail.gmail.com>

On Sun, Apr 22, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi Jed,
>
> Thanks for the quick fix. Btw, what's the easiest way to add your fix? Do
> I need to recompile?
>

Yes

   Matt


> Yours sincerely,
>
> TAY wee-beng
>
>
> On 22/4/2012 11:43 PM, Jed Brown wrote:
>
> On Sun, Apr 22, 2012 at 16:23, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> I'm working on it now, will report back when it's fixed. I normally
>> recommend src/snes/examples/tutorials/ex5f.F, even for linear problems.
>>
>
> Fixed here.
>
>  http://petsc.cs.iit.edu/petsc/petsc-dev/rev/39d1753f550d
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/0563f7e9/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Sun Apr 22 16:52:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 22 Apr 2012 16:52:02 -0500
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <4F947CCB.1020404@gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
	<CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
	<CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>
	<4F947CCB.1020404@gmail.com>
Message-ID: <CAM9tzSk_aF-yuYmNtsrYF=pGsWadoM1jQ54mdnm=9f5aENUUvg@mail.gmail.com>

On Sun, Apr 22, 2012 at 16:48, TAY wee-beng <zonexo at gmail.com> wrote:

> Thanks for the quick fix. Btw, what's the easiest way to add your fix? Do
> I need to recompile?


hg pull --update
$PETSC_ARCH/conf/reconfigure-*.py # if you haven't pulled petsc-dev
recently, not needed for this patch, but maybe for earlier patches
make
cd src/ksp/ksp/examples/tutorials
# uncomment VecView if you want
make ex22f
./ex22f
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120422/c6656d9e/attachment.htm>

From dominik at itis.ethz.ch  Mon Apr 23 07:17:33 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Mon, 23 Apr 2012 14:17:33 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
Message-ID: <CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>

>> Thanks Barry for the hint.
>> I am indeed solving my nonlinear system using the Picard scheme.
>> I was once advised to set the Jacobian to my coefficient matrix "A" in
>> the FormJacobian function passed to SNES to achieve this goal (I am
>> still trying to get it to work with limited success though).
>> Are you suggesting I should abandon this approach in favor of the new
>> functionality in petsc-dev?
>
> ?Yes if you want to do Picard then you should use SNESSetPicard() that is what it designed for.
>
> ? Barry

Great to hear there is a dedicated functionality for the Picard
scheme. I have 2 related questions:

1) What is 'func' on the SNESSetPicard manual page. It only says
"function evaluation routine". What function? Do you mean Ax-b?
2) Is it supposed to perform better / more optimally then using
FormFunction with FormJacobian and passing A as the Jacobian?

Regards,
Dominik

From jedbrown at mcs.anl.gov  Mon Apr 23 07:30:32 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 07:30:32 -0500
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
Message-ID: <CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>

On Mon, Apr 23, 2012 at 07:17, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> Great to hear there is a dedicated functionality for the Picard
> scheme. I have 2 related questions:
>
> 1) What is 'func' on the SNESSetPicard manual page. It only says
> "function evaluation routine". What function? Do you mean Ax-b?
> 2) Is it supposed to perform better / more optimally then using
> FormFunction with FormJacobian and passing A as the Jacobian?
>

All SNESSetPicard does is evaluate both A and b in "residual evaluation",
compute F(x) = A(x)x - b(x), and internally cache the matrix A(x). Using it
is completely optional. If you are willing to write residual evaluation for
the whole F(x), then you will benefit from having less expensive residual
evaluations, making line searches and matrix-free Newton
(-snes_mf_operator) affordable.

Personally, I dislike Picard. I have yet to find a problem in which I can't
make some variant of Newton beat it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/f6cc45c9/attachment.htm>

From dominik at itis.ethz.ch  Mon Apr 23 08:11:04 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Mon, 23 Apr 2012 15:11:04 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
	<CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
Message-ID: <CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>

>> 1) What is 'func' on the SNESSetPicard manual page. It only says
>> "function evaluation routine". What function? Do you mean Ax-b?
>> 2) Is it supposed to perform better / more optimally then using
>> FormFunction with FormJacobian and passing A as the Jacobian?
>
>

What is 'func' on the SNESSetPicard manual page. It only says
"function evaluation routine". What function? Do you mean Ax-b?

> All SNESSetPicard does is evaluate both A and b in "residual evaluation",
> compute F(x) = A(x)x - b(x), and internally cache the matrix A(x). Using it
> is completely optional. If you are willing to write residual evaluation for
> the whole F(x), then you will benefit from having less expensive residual
> evaluations, making line searches and matrix-free Newton (-snes_mf_operator)
> affordable.

I understood from the documentation sec. 5.1 and 5.6 that
-snes_mf_operator is not well suited for unstructured problems, rather
for structured problems with known stencil. Is it not so?

Dominik

From jedbrown at mcs.anl.gov  Mon Apr 23 08:13:03 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 08:13:03 -0500
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
	<CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
	<CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>
Message-ID: <CAM9tzSnWSnX2EybSM1fEaixibN=iTd5vGbc+V9L1amtuah9stg@mail.gmail.com>

On Mon, Apr 23, 2012 at 08:11, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> What is 'func' on the SNESSetPicard manual page. It only says
> "function evaluation routine". What function? Do you mean Ax-b?
>

Just b(x)


>
> > All SNESSetPicard does is evaluate both A and b in "residual evaluation",
> > compute F(x) = A(x)x - b(x), and internally cache the matrix A(x). Using
> it
> > is completely optional. If you are willing to write residual evaluation
> for
> > the whole F(x), then you will benefit from having less expensive residual
> > evaluations, making line searches and matrix-free Newton
> (-snes_mf_operator)
> > affordable.
>
> I understood from the documentation sec. 5.1 and 5.6 that
> -snes_mf_operator is not well suited for unstructured problems, rather
> for structured problems with known stencil. Is it not so?


You must be thinking of coloring.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/32594545/attachment.htm>

From dominik at itis.ethz.ch  Mon Apr 23 08:23:21 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Mon, 23 Apr 2012 15:23:21 +0200
Subject: [petsc-users] unexpected behavior of the snes solver
Message-ID: <CALm7otpj7VTBc2ks=tYh_nUBpAECqLGHqktx6oRNJ5EEpJ6fOw@mail.gmail.com>

Hi,

I am implementing both FormFunction and FormJacobian for the SNES
solver. It seems to start off correctly, with the 2 functions being
called one after another in a loop, and with ||Ax-b|| decreasing about
3 orders of magnitude. But then only the FormFunction is being called
several times, without subsequent calls to FormJacobian and obviously
the residue is not being reduced any further, because there are no
calls to the underlying ksp. If I inspect the solution it looks
qualitatively "on the way" to the expected one, but the values are not
correct, just as if the iterations were prematurely stopped. What sort
of a problem can such a behavior indicate?

Regards,
Dominik

From jedbrown at mcs.anl.gov  Mon Apr 23 08:25:44 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 08:25:44 -0500
Subject: [petsc-users] unexpected behavior of the snes solver
In-Reply-To: <CALm7otpj7VTBc2ks=tYh_nUBpAECqLGHqktx6oRNJ5EEpJ6fOw@mail.gmail.com>
References: <CALm7otpj7VTBc2ks=tYh_nUBpAECqLGHqktx6oRNJ5EEpJ6fOw@mail.gmail.com>
Message-ID: <CAM9tzS=RhRusJqCeQo1m8-60ftKZFAZQW-itp67CJBinWpTFwA@mail.gmail.com>

The linear solve computed a search direction that was not a descent
direction. It determined this by attempting a line search.
On Apr 23, 2012 8:23 AM, "Dominik Szczerba" <dominik at itis.ethz.ch> wrote:

> Hi,
>
> I am implementing both FormFunction and FormJacobian for the SNES
> solver. It seems to start off correctly, with the 2 functions being
> called one after another in a loop, and with ||Ax-b|| decreasing about
> 3 orders of magnitude. But then only the FormFunction is being called
> several times, without subsequent calls to FormJacobian and obviously
> the residue is not being reduced any further, because there are no
> calls to the underlying ksp. If I inspect the solution it looks
> qualitatively "on the way" to the expected one, but the values are not
> correct, just as if the iterations were prematurely stopped. What sort
> of a problem can such a behavior indicate?
>
> Regards,
> Dominik
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/a10d7ad6/attachment.htm>

From knepley at gmail.com  Mon Apr 23 08:26:06 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 23 Apr 2012 09:26:06 -0400
Subject: [petsc-users] unexpected behavior of the snes solver
In-Reply-To: <CALm7otpj7VTBc2ks=tYh_nUBpAECqLGHqktx6oRNJ5EEpJ6fOw@mail.gmail.com>
References: <CALm7otpj7VTBc2ks=tYh_nUBpAECqLGHqktx6oRNJ5EEpJ6fOw@mail.gmail.com>
Message-ID: <CAMYG4G=sOr0bJ3tLxXs2s-jRJxOhygrc9kZVJVcZpgARgpd13g@mail.gmail.com>

On Mon, Apr 23, 2012 at 9:23 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> Hi,
>
> I am implementing both FormFunction and FormJacobian for the SNES
> solver. It seems to start off correctly, with the 2 functions being
> called one after another in a loop, and with ||Ax-b|| decreasing about
> 3 orders of magnitude. But then only the FormFunction is being called
> several times, without subsequent calls to FormJacobian and obviously
> the residue is not being reduced any further, because there are no
> calls to the underlying ksp. If I inspect the solution it looks
>

This is a line search. If you upgrade to petsc-dev, line search has a lot
more diagnostics.

   Matt


> qualitatively "on the way" to the expected one, but the values are not
> correct, just as if the iterations were prematurely stopped. What sort
> of a problem can such a behavior indicate?
>
> Regards,
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/c6772a59/attachment-0001.htm>

From dominik at itis.ethz.ch  Mon Apr 23 08:28:30 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Mon, 23 Apr 2012 15:28:30 +0200
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CAM9tzSnWSnX2EybSM1fEaixibN=iTd5vGbc+V9L1amtuah9stg@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
	<CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
	<CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>
	<CAM9tzSnWSnX2EybSM1fEaixibN=iTd5vGbc+V9L1amtuah9stg@mail.gmail.com>
Message-ID: <CALm7otrxEpn8VA7MJvUSxcyvOXNseEMuYazhSq2JsspJi-pAoQ@mail.gmail.com>

On Mon, Apr 23, 2012 at 3:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Mon, Apr 23, 2012 at 08:11, Dominik Szczerba <dominik at itis.ethz.ch>
> wrote:
>>
>> What is 'func' on the SNESSetPicard manual page. It only says
>> "function evaluation routine". What function? Do you mean Ax-b?
>
>
> Just b(x)
>
>>
>>
>> > All SNESSetPicard does is evaluate both A and b in "residual
>> > evaluation",
>> > compute F(x) = A(x)x - b(x), and internally cache the matrix A(x). Using
>> > it
>> > is completely optional. If you are willing to write residual evaluation
>> > for
>> > the whole F(x), then you will benefit from having less expensive
>> > residual
>> > evaluations, making line searches and matrix-free Newton
>> > (-snes_mf_operator)
>> > affordable.
>>
>> I understood from the documentation sec. 5.1 and 5.6 that
>> -snes_mf_operator is not well suited for unstructured problems, rather
>> for structured problems with known stencil. Is it not so?
>
>
> You must be thinking of coloring.

5.1.2: "This causes PETSc to approximate the Jacobian using ?nite
differencing of the function evaluation (discussed in section 5.6),"
5.1.6: "PETSc provides some tools to help approximate the Jacobian
matrices ef?ciently via ?nite differences. (...) The approximation
requires several steps: (...) Determining the structure
is problem dependent, but fortunately, for most structured grid
problems (the class of problems for which
111PETSc is designed) if one knows the stencil used for the nonlinear
function one can usually fairly easily
obtain an estimate of the location of nonzeros in the matrix. This is
harder in the unstructured case, and has
not yet been implemented in general."

I am dealing with unstructured problems. Will I still be able to
benefit from -snes_mf_operator and will it approximate the Jacobian
correctly/efficiently in such cases?

Dominik

From knepley at gmail.com  Mon Apr 23 08:41:08 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 23 Apr 2012 09:41:08 -0400
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CALm7otrxEpn8VA7MJvUSxcyvOXNseEMuYazhSq2JsspJi-pAoQ@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
	<CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
	<CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>
	<CAM9tzSnWSnX2EybSM1fEaixibN=iTd5vGbc+V9L1amtuah9stg@mail.gmail.com>
	<CALm7otrxEpn8VA7MJvUSxcyvOXNseEMuYazhSq2JsspJi-pAoQ@mail.gmail.com>
Message-ID: <CAMYG4G=8kfqgmyq2=NjOvuodzwgQtu_uVWj3eoJJ28VyS8ak1g@mail.gmail.com>

On Mon, Apr 23, 2012 at 9:28 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> On Mon, Apr 23, 2012 at 3:13 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > On Mon, Apr 23, 2012 at 08:11, Dominik Szczerba <dominik at itis.ethz.ch>
> > wrote:
> >>
> >> What is 'func' on the SNESSetPicard manual page. It only says
> >> "function evaluation routine". What function? Do you mean Ax-b?
> >
> >
> > Just b(x)
> >
> >>
> >>
> >> > All SNESSetPicard does is evaluate both A and b in "residual
> >> > evaluation",
> >> > compute F(x) = A(x)x - b(x), and internally cache the matrix A(x).
> Using
> >> > it
> >> > is completely optional. If you are willing to write residual
> evaluation
> >> > for
> >> > the whole F(x), then you will benefit from having less expensive
> >> > residual
> >> > evaluations, making line searches and matrix-free Newton
> >> > (-snes_mf_operator)
> >> > affordable.
> >>
> >> I understood from the documentation sec. 5.1 and 5.6 that
> >> -snes_mf_operator is not well suited for unstructured problems, rather
> >> for structured problems with known stencil. Is it not so?
> >
> >
> > You must be thinking of coloring.
>
> 5.1.2: "This causes PETSc to approximate the Jacobian using ?nite
> differencing of the function evaluation (discussed in section 5.6),"
> 5.1.6: "PETSc provides some tools to help approximate the Jacobian
> matrices ef?ciently via ?nite differences. (...) The approximation
> requires several steps: (...) Determining the structure
> is problem dependent, but fortunately, for most structured grid
> problems (the class of problems for which
> 111PETSc is designed) if one knows the stencil used for the nonlinear
> function one can usually fairly easily
> obtain an estimate of the location of nonzeros in the matrix. This is
> harder in the unstructured case, and has
> not yet been implemented in general."
>

-snes_mf uses FD to approximate the ACTION of the operator. The above
quotes are talking about using FD to approximate the operator itself.


> I am dealing with unstructured problems. Will I still be able to
> benefit from -snes_mf_operator and will it approximate the Jacobian
> correctly/efficiently in such cases?


-snes_mf_operator allows the user to specify a preconditioner, which is
usually necessary for good convergence.

   Matt


>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/4cc01640/attachment.htm>

From jedbrown at mcs.anl.gov  Mon Apr 23 08:43:30 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 08:43:30 -0500
Subject: [petsc-users] SNES and ghosted vector of unknowns
In-Reply-To: <CAMYG4G=8kfqgmyq2=NjOvuodzwgQtu_uVWj3eoJJ28VyS8ak1g@mail.gmail.com>
References: <CALm7otoa7zoBZrO9GTCh_uV8ME5TEd0BQnW7u9OVHuGEEjqrSQ@mail.gmail.com>
	<CAMYG4Gkkyz2+jPNjrzDmWPy2-GjkYn8KLbOCy2o754OR5KXdbA@mail.gmail.com>
	<CAM9tzSmk9Oip5uvoTRUpvanW0qpHY026Jw7+WnHqp99CKDrVzg@mail.gmail.com>
	<CALm7otqsGquN4vJQ8Q0432HXM+7ECKghi=g0gs_Py2yVrpGO_w@mail.gmail.com>
	<148A3C66-B0A4-41A4-873B-47CFD4ED9673@mcs.anl.gov>
	<CALm7otqZxjvoWX8guuoPm3WS5Vep9grZup+uQ+g+2sSUZ5Su9g@mail.gmail.com>
	<43EC5C21-BAE8-4793-8EEE-04214BFD49D0@mcs.anl.gov>
	<CALm7otr36ua490Rm2ovNU9K34rx1qx3V1LoX1LYK4HO84S=zzg@mail.gmail.com>
	<CAM9tzSnYqpg3L9cGET4=src6+J+thi+N6EnVuD90kX18HSdXTA@mail.gmail.com>
	<CALm7otrVSSd66TC+qPOmXqH0O5YR+yP3+gsvQtENiaDKqBu84w@mail.gmail.com>
	<CAM9tzSnWSnX2EybSM1fEaixibN=iTd5vGbc+V9L1amtuah9stg@mail.gmail.com>
	<CALm7otrxEpn8VA7MJvUSxcyvOXNseEMuYazhSq2JsspJi-pAoQ@mail.gmail.com>
	<CAMYG4G=8kfqgmyq2=NjOvuodzwgQtu_uVWj3eoJJ28VyS8ak1g@mail.gmail.com>
Message-ID: <CAM9tzS=N+jegNY8UDdPbXrJOaVGWy-Rw3mO2aUNWC=povbdqEA@mail.gmail.com>

On Mon, Apr 23, 2012 at 08:41, Matthew Knepley <knepley at gmail.com> wrote:

> -snes_mf uses FD to approximate the ACTION of the operator. The above
> quotes are talking about using FD to approximate the operator itself.
>

I.e., to assemble as a sparse matrix, so that it can be used as input to a
preconditioning method.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/237debdf/attachment.htm>

From fuentesdt at gmail.com  Mon Apr 23 09:46:26 2012
From: fuentesdt at gmail.com (David Fuentes)
Date: Mon, 23 Apr 2012 09:46:26 -0500
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
Message-ID: <CAKnkmZwJXfiGNh+k-57wQrpQ5Dppm39a-6aV1gja-Wk64vkOwQ@mail.gmail.com>

I have some existing libMesh code that I would like to use with this ex52
GPU code example.

I think it may be reasonable to use the existing code for setup of initial
conditions and jacobian calc,
But then use the GPU residual calculation.

Where is the nodal degree of freedom stored in the "DM_Complex" data
structure  ?
I would like to setup the same dof ordering in the libMesh data structures.

I'm also a little unfamiliar with the terminology in the DM_Complex data
structure,
what is being referred to by the "cones", "maxConeSize", and
"coneOrientation" ?

Lastly,
Are there any data structures to pass spatially varying element-wise
material parameters to the inner loop of the assembly?

for example, something like:

diff -r 58bb1a4e8a0d src/snes/examples/tutorials/ex52_integrateElement.cu
--- a/src/snes/examples/tutorials/ex52_integrateElement.cu      Tue Mar 27
23:22:11 2012 -0500
+++ b/src/snes/examples/tutorials/ex52_integrateElement.cu      Mon Apr 23
09:36:30 2012 -0500
@@ -3,8 +3,8 @@

 #include "ex52_gpu.h"

-__device__ vecType f1_laplacian(float u[], vecType gradU[], int comp) {
-  return gradU[comp];
+__device__ vecType f1_laplacian(float thermalconductivity, float u[],
vecType gradU[], int comp) {
+  return thermalconductivity*gradU[comp];
 }




Thanks,
David






On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>
>>> Yes. This would work.
>>> I had trouble compiling in single precision using the some of the
>>> external package options I was using for double.
>>>
>>
>> Okay,  getting on it now.
>>
>
> Okay, I have tested using PetscReal=double with floats on the GPU. If you
> want doubles on the GPU, you can
> currently change PETSc/FEM.py:434. I am deciding what the best way to
> specify this type is.
>
>   Thanks,
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/9516d2bb/attachment.htm>

From knepley at gmail.com  Mon Apr 23 11:13:15 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 23 Apr 2012 12:13:15 -0400
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAKnkmZwJXfiGNh+k-57wQrpQ5Dppm39a-6aV1gja-Wk64vkOwQ@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
	<CAKnkmZwJXfiGNh+k-57wQrpQ5Dppm39a-6aV1gja-Wk64vkOwQ@mail.gmail.com>
Message-ID: <CAMYG4G=m3X3E85XzGjsr1Lf0W1CVWdwr5YZPXigakx27mOA8Qg@mail.gmail.com>

On Mon, Apr 23, 2012 at 10:46 AM, David Fuentes <fuentesdt at gmail.com> wrote:

> I have some existing libMesh code that I would like to use with this ex52
> GPU code example.
>
> I think it may be reasonable to use the existing code for setup of initial
> conditions and jacobian calc,
> But then use the GPU residual calculation.
>
> Where is the nodal degree of freedom stored in the "DM_Complex" data
> structure  ?
> I would like to setup the same dof ordering in the libMesh data structures.
>
> I'm also a little unfamiliar with the terminology in the DM_Complex data
> structure,
> what is being referred to by the "cones", "maxConeSize", and
> "coneOrientation" ?
>

Okay, the DMComplex only stores topological information. The "cone" and
"support" terminology is
from http://arxiv.org/abs/0908.4427, and in the language of DAGs means in
edges and out edges. In
the Hasse Diagram, from combinatorial topology, the in edges come from
lower dimensional mesh
pieces that "cover" the given piece of the mesh. For instance, a cell is
covered by its vertices, so
the cone of a cell can be its vertices. This can be made more involved by
including edges and faces.

All the information for data layout is not in the DMComplex, but in the
PetscSection object. This is just
a map between pieces of the mesh, like vertices, and degrees of freedom,
which are offsets into a
PETSc Vec. All values are stored in Vecs. There is a convenience function
for creating this layout with
FEM information


http://petsc.cs.iit.edu/petsc/petsc-dev/annotate/305aed156be4/src/snes/examples/tutorials/ex52.c#l188

but you can dot it yourself using the PetscSection functions to match any
LibMesh layout, and then call


http://petsc.cs.iit.edu/petsc/petsc-dev/annotate/305aed156be4/src/snes/examples/tutorials/ex52.c#l189

Let me know if this makes sense. Also, I have started the variable
coefficient stuff, but its not done yet.

     Matt


> Lastly,
> Are there any data structures to pass spatially varying element-wise
> material parameters to the inner loop of the assembly?
>
> for example, something like:
>
> diff -r 58bb1a4e8a0d src/snes/examples/tutorials/ex52_integrateElement.cu
> --- a/src/snes/examples/tutorials/ex52_integrateElement.cu      Tue Mar 27
> 23:22:11 2012 -0500
> +++ b/src/snes/examples/tutorials/ex52_integrateElement.cu      Mon Apr 23
> 09:36:30 2012 -0500
> @@ -3,8 +3,8 @@
>
>  #include "ex52_gpu.h"
>
> -__device__ vecType f1_laplacian(float u[], vecType gradU[], int comp) {
> -  return gradU[comp];
> +__device__ vecType f1_laplacian(float thermalconductivity, float u[],
> vecType gradU[], int comp) {
> +  return thermalconductivity*gradU[comp];
>  }
>
>
>
>
> Thanks,
> David
>
>
>
>
>
>
> On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>
>>>> Yes. This would work.
>>>> I had trouble compiling in single precision using the some of the
>>>> external package options I was using for double.
>>>>
>>>
>>> Okay,  getting on it now.
>>>
>>
>> Okay, I have tested using PetscReal=double with floats on the GPU. If you
>> want doubles on the GPU, you can
>> currently change PETSc/FEM.py:434. I am deciding what the best way to
>> specify this type is.
>>
>>   Thanks,
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/4676c467/attachment-0001.htm>

From zonexo at gmail.com  Mon Apr 23 15:56:08 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Mon, 23 Apr 2012 22:56:08 +0200
Subject: [petsc-users] Questions abt ex22f
In-Reply-To: <CAM9tzSk_aF-yuYmNtsrYF=pGsWadoM1jQ54mdnm=9f5aENUUvg@mail.gmail.com>
References: <4F933526.9070304@gmail.com>
	<CAMYG4Gm3iacc8SHcFZz5kSZqei8azk1QtqjFtE+14gEuAPOJPQ@mail.gmail.com>
	<4F93B3D1.2060304@gmail.com>
	<CAMYG4G=Gm4mgeAU3amGGjT87a7QVSkJQrYuLNqDsYs+HxmsgDA@mail.gmail.com>
	<4F9455F4.9050704@gmail.com>
	<CAM9tzSkD4Y6WO5vd1+BjPFswmmv384cdU659qnp0UBTX0swZvA@mail.gmail.com>
	<4F94599F.3040504@gmail.com>
	<CAM9tzSmN7r4LaQiViKV8phtY3_NwTmPAr+Q+GtmS2CKqF4-9Mw@mail.gmail.com>
	<4F945C68.5090201@gmail.com>
	<CAM9tzSnQ6oJaPtgguKAVjRbE18ioE6OcAnNTqeHx=aYpu58azQ@mail.gmail.com>
	<4F946245.1000502@gmail.com>
	<CAMYG4GnPrZTRyYUBghaEdrGGG9y0r1ArgTGx5RQBS1Yc09Pgew@mail.gmail.com>
	<4F94764F.80200@gmail.com>
	<CAM9tzS=HNXyceh2ZmBWXNHVyu6Nm8ceZ7e=t56b3fyZunrYX1w@mail.gmail.com>
	<CAM9tzS=SR-aVs_Y7vgtqixw63d=h+kPX6_0Yyfyco6iwf005Vg@mail.gmail.com>
	<4F947CCB.1020404@gmail.com>
	<CAM9tzSk_aF-yuYmNtsrYF=pGsWadoM1jQ54mdnm=9f5aENUUvg@mail.gmail.com>
Message-ID: <4F95C1E8.30408@gmail.com>

Hi,

I just tested the new ex22f on win7 vs2008. It's now giving the same 
results as the linux.

Thanks for the fix!

Yours sincerely,

TAY wee-beng


On 22/4/2012 11:52 PM, Jed Brown wrote:
> On Sun, Apr 22, 2012 at 16:48, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Thanks for the quick fix. Btw, what's the easiest way to add your
>     fix? Do I need to recompile?
>
>
> hg pull --update
> $PETSC_ARCH/conf/reconfigure-*.py # if you haven't pulled petsc-dev 
> recently, not needed for this patch, but maybe for earlier patches
> make
> cd src/ksp/ksp/examples/tutorials
> # uncomment VecView if you want
> make ex22f
> ./ex22f
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/1a26d30e/attachment.htm>

From burckhardt at itis.ethz.ch  Mon Apr 23 20:27:06 2012
From: burckhardt at itis.ethz.ch (tribur)
Date: Tue, 24 Apr 2012 03:27:06 +0200
Subject: [petsc-users] Gauss-Seidel
Message-ID: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>

Dear Petsc team,

I would like to do 10 iterations of a point-by-point Gauss-Seidel 
procedure for the purpose of testing.

Is it correct if I type "mpiexec -n 1 ... -ksp_type preonly -pc_type 
sor -pc_sor_its 10"?

How can I define the initial guess nonzero?

Best regards,

Kathrin

From jedbrown at mcs.anl.gov  Mon Apr 23 20:29:22 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 20:29:22 -0500
Subject: [petsc-users] Gauss-Seidel
In-Reply-To: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>
References: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>
Message-ID: <CAM9tzSkW6THMO3yYXsCkEN1VCKfbKioW9HB=hRvzJqY+o1S7Yw@mail.gmail.com>

On Mon, Apr 23, 2012 at 20:27, tribur <burckhardt at itis.ethz.ch> wrote:

> I would like to do 10 iterations of a point-by-point Gauss-Seidel
> procedure for the purpose of testing.
>
> Is it correct if I type "mpiexec -n 1 ... -ksp_type preonly -pc_type sor
> -pc_sor_its 10"?
>
> How can I define the initial guess nonzero?
>

With a nonzero initial guess, you are doing Richardson (not preonly). You
can set the nonzero initial guess as usual (KSPSetInitialGuessNonzero()).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/2a115d14/attachment.htm>

From burckhardt at itis.ethz.ch  Mon Apr 23 20:40:41 2012
From: burckhardt at itis.ethz.ch (tribur)
Date: Tue, 24 Apr 2012 03:40:41 +0200
Subject: [petsc-users] Gauss-Seidel
In-Reply-To: <CAM9tzSkW6THMO3yYXsCkEN1VCKfbKioW9HB=hRvzJqY+o1S7Yw@mail.gmail.com>
References: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>
	<CAM9tzSkW6THMO3yYXsCkEN1VCKfbKioW9HB=hRvzJqY+o1S7Yw@mail.gmail.com>
Message-ID: <c6fd659077446c854ccb9d9721251658@oetiker.ch>

Hi Jed,

Thanks for the ultra fast response.

Apart from the nonzero initial guess, is -ksp_type preonly -pc_type sor 
-pc_sor_its 10 right?

Am 24.04.2012 03:29, schrieb Jed Brown:
> On Mon, Apr 23, 2012 at 20:27, tribur <burckhardt at itis.ethz.ch [1]>
> wrote:
>
>> I would like to do 10 iterations of a point-by-point Gauss-Seidel
>> procedure for the purpose of testing.
>>
>> Is it correct if I type "mpiexec -n 1 ... -ksp_type preonly
>> -pc_type sor -pc_sor_its 10"?
>>
>> How can I define the initial guess nonzero?
>
> With a nonzero initial guess, you are doing Richardson (not preonly).
> You can set the nonzero initial guess as usual
> (KSPSetInitialGuessNonzero()).
>
> Links:
> ------
> [1] mailto:burckhardt at itis.ethz.ch


From jedbrown at mcs.anl.gov  Mon Apr 23 20:42:45 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 23 Apr 2012 20:42:45 -0500
Subject: [petsc-users] Gauss-Seidel
In-Reply-To: <c6fd659077446c854ccb9d9721251658@oetiker.ch>
References: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>
	<CAM9tzSkW6THMO3yYXsCkEN1VCKfbKioW9HB=hRvzJqY+o1S7Yw@mail.gmail.com>
	<c6fd659077446c854ccb9d9721251658@oetiker.ch>
Message-ID: <CAM9tzSkvx+swuSrjhby73D2eEnM=mvhbokDU=ZRCanh9qqpsfg@mail.gmail.com>

On Mon, Apr 23, 2012 at 20:40, tribur <burckhardt at itis.ethz.ch> wrote:

> Hi Jed,
>
> Thanks for the ultra fast response.
>
> Apart from the nonzero initial guess, is -ksp_type preonly -pc_type sor
> -pc_sor_its 10 right?
>

Yes.

You can use -ksp_initial_guess_nonzero -ksp_type richardson


>
> Am 24.04.2012 03:29, schrieb Jed Brown:
>
>> On Mon, Apr 23, 2012 at 20:27, tribur <burckhardt at itis.ethz.ch [1]>
>>
>> wrote:
>>
>>  I would like to do 10 iterations of a point-by-point Gauss-Seidel
>>> procedure for the purpose of testing.
>>>
>>> Is it correct if I type "mpiexec -n 1 ... -ksp_type preonly
>>> -pc_type sor -pc_sor_its 10"?
>>>
>>> How can I define the initial guess nonzero?
>>>
>>
>> With a nonzero initial guess, you are doing Richardson (not preonly).
>> You can set the nonzero initial guess as usual
>> (KSPSetInitialGuessNonzero()).
>>
>> Links:
>> ------
>> [1] mailto:burckhardt at itis.ethz.ch
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120423/32380d26/attachment.htm>

From burckhardt at itis.ethz.ch  Mon Apr 23 20:45:12 2012
From: burckhardt at itis.ethz.ch (tribur)
Date: Tue, 24 Apr 2012 03:45:12 +0200
Subject: [petsc-users] Gauss-Seidel
In-Reply-To: <CAM9tzSkvx+swuSrjhby73D2eEnM=mvhbokDU=ZRCanh9qqpsfg@mail.gmail.com>
References: <c6d0d215089372cf7104b4c56db8d64a@oetiker.ch>
	<CAM9tzSkW6THMO3yYXsCkEN1VCKfbKioW9HB=hRvzJqY+o1S7Yw@mail.gmail.com>
	<c6fd659077446c854ccb9d9721251658@oetiker.ch>
	<CAM9tzSkvx+swuSrjhby73D2eEnM=mvhbokDU=ZRCanh9qqpsfg@mail.gmail.com>
Message-ID: <97f75552c02793658e8a67c057b1fbb2@oetiker.ch>

Thanks a lot!

Am 24.04.2012 03:42, schrieb Jed Brown:
> On Mon, Apr 23, 2012 at 20:40, tribur <burckhardt at itis.ethz.ch [4]>
> wrote:
>
>> Hi Jed,
>>
>> Thanks for the ultra fast response.
>>
>> Apart from the nonzero initial guess, is -ksp_type preonly -pc_type
>> sor -pc_sor_its 10 right?
>
> Yes.
>
> You can use?-ksp_initial_guess_nonzero -ksp_type richardson
> ?
>
>> Am 24.04.2012 03 [3]:29, schrieb Jed Brown:
>>
>>> On Mon, Apr 23, 2012 at 20:27, tribur <burckhardt at itis.ethz.ch [1]
>>> [1]>
>>>
>>> wrote:
>>>
>>>> I would like to do 10 iterations of a point-by-point
>>>> Gauss-Seidel
>>>> procedure for the purpose of testing.
>>>>
>>>> Is it correct if I type "mpiexec -n 1 ... -ksp_type preonly
>>>> -pc_type sor -pc_sor_its 10"?
>>>>
>>>> How can I define the initial guess nonzero?
>>>
>>> With a nonzero initial guess, you are doing Richardson (not
>>> preonly).
>>> You can set the nonzero initial guess as usual
>>> (KSPSetInitialGuessNonzero()).
>>>
>>> Links:
>>> ------
>>> [1] mailto:burckhardt at itis.ethz.ch [2]
>
>
>
> Links:
> ------
> [1] mailto:burckhardt at itis.ethz.ch
> [2] mailto:burckhardt at itis.ethz.ch
> [3] http://webmail.oetiker.ch/tel:24.04.2012%2003
> [4] mailto:burckhardt at itis.ethz.ch


From jiangwen84 at gmail.com  Tue Apr 24 12:43:06 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Tue, 24 Apr 2012 13:43:06 -0400
Subject: [petsc-users] mumps get stuck with parmetis
Message-ID: <CAMJxm+Da_byqMzm-_j+hRBUJSBt25oQAfDmUnVqHgBS4VPy8_A@mail.gmail.com>

Hi,

My code will hang at the solving stage when I use mumps with the runtime
option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
options, my code works fine. I am using PETSc 3.2 and configure it
with --download-mumps=1
and --download-parmetis=1. Could anyone give me any hints? Thanks.

Regards,
Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/edd22bfe/attachment.htm>

From hzhang at mcs.anl.gov  Tue Apr 24 13:29:32 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 24 Apr 2012 13:29:32 -0500
Subject: [petsc-users] mumps get stuck with parmetis
In-Reply-To: <CAMJxm+Da_byqMzm-_j+hRBUJSBt25oQAfDmUnVqHgBS4VPy8_A@mail.gmail.com>
References: <CAMJxm+Da_byqMzm-_j+hRBUJSBt25oQAfDmUnVqHgBS4VPy8_A@mail.gmail.com>
Message-ID: <CAGCphBurn0+f--gOZ_4MSrLcntvYSVCQ2_cmSG93nvv3hHSEjA@mail.gmail.com>

Wen :
I cannot repeat your error with petsc-dev. Running
petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c:
mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps
-mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2
Norm of error 1.67272e-15 iterations 1

Can you run above with your petsc-3.2 installation?

Hong

Hi,
>
> My code will hang at the solving stage when I use mumps with the runtime
> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> options, my code works fine. I am using PETSc 3.2 and configure it with --download-mumps=1
> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>
> Regards,
> Wen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/196ff3c5/attachment.htm>

From agrayver at gfz-potsdam.de  Tue Apr 24 14:01:45 2012
From: agrayver at gfz-potsdam.de (=?utf-8?B?QWxleGFuZGVyIEdyYXl2ZXI=?=)
Date: Tue, 24 Apr 2012 21:01:45 +0200
Subject: [petsc-users] =?utf-8?q?mumps_get_stuck_with_parmetis?=
Message-ID: <auto-000008199816@cgp2.gfz-potsdam.de>

Can you set:

-mat_mumps_icntl_4 1

And send mumps output?
Also do you use lu or ilu? How large is your matrix?

Regards,
Alexander

----- Reply message -----
From: "Wen Jiang" <jiangwen84 at gmail.com>
To: <petsc-users at mcs.anl.gov>
Subject: [petsc-users] mumps get stuck with parmetis
Date: Tue, Apr 24, 2012 19:43
Hi,

My code will hang at the solving stage when I use mumps with the runtime option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two options, my code works fine. I am using PETSc 3.2 and configure it with --download-mumps=1 and --download-parmetis=1. Could anyone give me any hints? Thanks.


Regards,
Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2eb1767b/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Tue Apr 24 14:13:59 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 24 Apr 2012 22:13:59 +0300
Subject: [petsc-users] mumps get stuck with parmetis
In-Reply-To: <auto-000008199816@cgp2.gfz-potsdam.de>
References: <auto-000008199816@cgp2.gfz-potsdam.de>
Message-ID: <CADVoUZQDYqOpoZtxWY1DGE=WGgbZvxMVuRy8zRxYTtrSCMP3sg@mail.gmail.com>

I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on BlueGene/P
was causing similar behavior without even setting any options.  The only
way I was able to get a direct solver going was by switching over to
SuperLU.

A

On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <agrayver at gfz-potsdam.de
> wrote:

> Can you set:
>
> -mat_mumps_icntl_4 1
>
> And send mumps output?
> Also do you use lu or ilu? How large is your matrix?
>
> Regards,
> Alexander
>
>
> ----- Reply message -----
> From: "Wen Jiang" <jiangwen84 at gmail.com>
> To: <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] mumps get stuck with parmetis
> Date: Tue, Apr 24, 2012 19:43
>
>
> Hi,
>
> My code will hang at the solving stage when I use mumps with the runtime
> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> options, my code works fine. I am using PETSc 3.2 and configure it with --download-mumps=1
> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>
> Regards,
> Wen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/b0503b1c/attachment.htm>

From agrayver at gfz-potsdam.de  Tue Apr 24 14:22:51 2012
From: agrayver at gfz-potsdam.de (=?utf-8?B?QWxleGFuZGVyIEdyYXl2ZXI=?=)
Date: Tue, 24 Apr 2012 21:22:51 +0200
Subject: [petsc-users] =?utf-8?q?mumps_get_stuck_with_parmetis?=
Message-ID: <auto-000056660896@cgp1.gfz-potsdam.de>

Aron,

This parameter let's to see mumps output in console. The important this to understand where mumps hangs, during analysis, factorization or actual solution (substitutions)? I'm almost sure it's factorization step. I observe this pretty often with mumps compiled with petsc (whereas when mumps is used directly it's quite rare to come along with this problem).

Regards,
Alexander 

----- Reply message -----
From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Subject: [petsc-users] mumps get stuck with parmetis
Date: Tue, Apr 24, 2012 21:13
I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on BlueGene/P was causing similar behavior without even setting any options. ?The only way I was able to get a direct solver going was by switching over to SuperLU.

A

On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <agrayver at gfz-potsdam.de> wrote:

Can you set:

-mat_mumps_icntl_4 1

And send mumps output?
Also do you use lu or ilu? How large is your matrix?


Regards,
Alexander

----- Reply message -----
From: "Wen Jiang" <jiangwen84 at gmail.com>

To: <petsc-users at mcs.anl.gov>
Subject: [petsc-users] mumps get stuck with parmetis
Date: Tue, Apr 24, 2012 19:43


Hi,

My code will hang at the solving stage when I use mumps with the runtime option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two options, my code works fine. I am using PETSc 3.2 and configure it with --download-mumps=1 and --download-parmetis=1. Could anyone give me any hints? Thanks.



Regards,
Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2db311da/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Tue Apr 24 14:41:00 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 24 Apr 2012 22:41:00 +0300
Subject: [petsc-users] mumps get stuck with parmetis
In-Reply-To: <auto-000056660896@cgp1.gfz-potsdam.de>
References: <auto-000056660896@cgp1.gfz-potsdam.de>
Message-ID: <CADVoUZSL+UXMJ0V52ELpE2vLLXc6WbZe96+yL5tYPsyMYiLXeQ@mail.gmail.com>

/project/k121/sandbox/petsc
-dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_part9.F:4666
/project/k121/sandbox/petsc
-dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part5.F:465
/project/k121/sandbox/petsc
-dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part1.F:409
/project/k121/sandbox/petsc
-dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part3.F:6651
/project/k121/sandbox/petsc
-dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_c.c:422

I don't know the MUMPS source code very well so I couldn't tell you what
this set of routines are doing, but this is a snippet of the stack trace I
was seeing when the jobs died on BG/P.

If you set the "-info" flag on a PETSc run, it sends a lot of debugging
output to the screen, which is useful when you're in a situation where it
is hard to get access to a debugger or the stack trace.

A

On Tue, Apr 24, 2012 at 10:22 PM, Alexander Grayver <agrayver at gfz-potsdam.de
> wrote:

> Aron,
>
> This parameter let's to see mumps output in console. The important this to
> understand where mumps hangs, during analysis, factorization or actual
> solution (substitutions)? I'm almost sure it's factorization step. I
> observe this pretty often with mumps compiled with petsc (whereas when
> mumps is used directly it's quite rare to come along with this problem).
>
> Regards,
> Alexander
>
> ----- Reply message -----
> From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] mumps get stuck with parmetis
> Date: Tue, Apr 24, 2012 21:13
>
>
> I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
> BlueGene/P was causing similar behavior without even setting any options.
>  The only way I was able to get a direct solver going was by switching over
> to SuperLU.
>
> A
>
> On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
> agrayver at gfz-potsdam.de> wrote:
>
>> Can you set:
>>
>> -mat_mumps_icntl_4 1
>>
>> And send mumps output?
>> Also do you use lu or ilu? How large is your matrix?
>>
>> Regards,
>> Alexander
>>
>>
>> ----- Reply message -----
>> From: "Wen Jiang" <jiangwen84 at gmail.com>
>> To: <petsc-users at mcs.anl.gov>
>> Subject: [petsc-users] mumps get stuck with parmetis
>> Date: Tue, Apr 24, 2012 19:43
>>
>>
>> Hi,
>>
>> My code will hang at the solving stage when I use mumps with the runtime
>> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
>> options, my code works fine. I am using PETSc 3.2 and configure it with --download-mumps=1
>> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>>
>> Regards,
>> Wen
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/3efe54e8/attachment-0001.htm>

From zonexo at gmail.com  Tue Apr 24 15:32:13 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 24 Apr 2012 22:32:13 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
Message-ID: <4F970DCD.1060302@gmail.com>

Hi,

I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran 
version. The fortran version is based on the template of ex22f, which is 
in 3d.

I have attached the code below. I have 2 problems.

1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c 
version.

However, when I change i3 to -4, I got the following error:

[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:08:46 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in 
src/mat/impls/aij/seq/C:\temp\
PETSC-~1\src\mat\impls\aij\seq\aij.c
[0]PETSC ERROR: MatSetValues() line 1148 in 
src/mat/interface/C:\temp\PETSC-~1\s
rc\mat\INTERF~1\matrix.c
[0]PETSC ERROR: MatSetValuesLocal() line 2003 in 
src/mat/interface/C:\temp\PETSC
-~1\src\mat\INTERF~1\matrix.c
[0]PETSC ERROR: MatSetValuesStencil() line 1379 in 
src/mat/interface/C:\temp\PET
SC-~1\src\mat\INTERF~1\matrix.c
[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (10,2) caused a malloc!

...

What did I do wrong?


2. When I wanted to use the neumann BC, following ex29, I added :

In subroutine ComputeRHS

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)

call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)

call MatNullSpaceDestroy(nullspace,ierr)

and in subroutine ComputeMatrix

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)

call MatSetNullSpace(jac,nullspace,ierr)

call MatNullSpaceDestroy(nullspace,ierr)

When I compile, it says unresolved external symbol MatNullSpaceRemove. 
Removing MatNullSpaceRemove, I can compile but running gives the error:

[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Corrupt argument:
see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:30:31 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatNullSpaceCreate() line 250 in 
src/mat/interface/C:\temp\PETSC
-~1\src\mat\INTERF~1\matnull.c
[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:30:31 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatNullSpaceDestroy() line 305 in 
src/mat/interface/C:\temp\PETS
C-~1\src\mat\INTERF~1\matnull.c
[0]PETSC ERROR: ourdmfunction() line 30 in 
src/dm/interface/ftn-custom/C:\temp\P
ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
[0]PETSC ERROR: DMComputeFunction() line 1783 in 
src/dm/interface/C:\temp\PETSC-
~1\src\dm\INTERF~1\dm.c
[0]PETSC ERROR: KSPSetUp() line 218 in 
src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
c\ksp\ksp\INTERF~1\itfunc.c
[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Corrupt argument:
see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:30:31 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatNullSpaceCreate() line 250 in 
src/mat/interface/C:\temp\PETSC
-~1\src\mat\INTERF~1\matnull.c
[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:30:31 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatNullSpaceDestroy() line 305 in 
src/mat/interface/C:\temp\PETS
C-~1\src\mat\INTERF~1\matnull.c
[0]PETSC ERROR: ourdmfunction() line 30 in 
src/dm/interface/ftn-custom/C:\temp\P
ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
[0]PETSC ERROR: DMComputeFunction() line 1783 in 
src/dm/interface/C:\temp\PETSC-
~1\src\dm\INTERF~1\dm.c
[0]PETSC ERROR: KSPSetUp() line 218 in 
src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
c\ksp\ksp\INTERF~1\itfunc.c
[0]PETSC ERROR: KSPSolve() line 402 in 
src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
c\ksp\ksp\INTERF~1\itfunc.c
Vector Object:Vec_0000000084000000_0 1 MPI processes
   type: mpi
Process [0]

What's wrong?


*program ex29f*

implicit none

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                    Include files
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
!     petscsys.h  - base PETSc routines      petscvec.h - vectors
!     petscmat.h - matrices
!     petscksp.h    - Krylov subspace methods  petscpc.h  - preconditioners

#include "finclude/petsc.h90"

PetscErrorCode   ierr

DM               da

KSP              ksp

Vec              x

external         ComputeRHS,ComputeMatrix

PetscInt i1,i3

call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)

i3 = -4

i1 = 1

call KSPCreate(MPI_COMM_WORLD,ksp,ierr)

call 
DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
call DMSetFunction(da,ComputeRHS,ierr)
call DMSetJacobian(da,ComputeMatrix,ierr)
call KSPSetDM(ksp,da,ierr)

call KSPSetFromOptions(ksp,ierr)
call KSPSetUp(ksp,ierr)
call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
call KSPGetSolution(ksp,x,ierr)
call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
call KSPDestroy(ksp,ierr)
call DMDestroy(da,ierr)
call PetscFinalize(ierr)

end program ex29f

subroutine ComputeRHS(da,x,b,ierr)

implicit none

#include "finclude/petsc.h90"

PetscErrorCode  ierr
PetscInt i,j,mx,my,xm,ym,xs,ys
PetscScalar  h,nu,rho
PetscScalar Hx,Hy
PetscScalar,pointer :: array(:,:)
Vec          x,b
DM           da
MatNullSpace nullspace    !>neumann BC

nu = 0.1

call 
DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)

Hx = 1.d0 / (mx-1)

Hy = 1.d0 / (my-1)

call 
DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)

call DMDAVecGetArrayF90(da,b,array,ierr)

do j = ys,ys+ym-1

     do i = xs,xs+xm-1

         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy

     end do

end do

call DMDAVecRestoreArrayF90(da,b,array,ierr)

call VecAssemblyBegin(b,ierr)

call VecAssemblyEnd(b,ierr)

!call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)

!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)

!call MatNullSpaceDestroy(nullspace,ierr)

end subroutine ComputeRHS


subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)

implicit none

#include "finclude/petsc.h90"

Mat          jac,JJ

PetscErrorCode    ierr

DM           da

PetscInt    i,j,k,mx,my,xm

PetscInt    ym,xs,ys,i1,i5

PetscScalar  v(5),Hx,Hy,rho,centerRho

PetscScalar  HydHx

PetscScalar  HxdHy

MatStencil   row(4),col(4,5)

Vec          x

MatStructure str

MatNullSpace nullspace    !>neumann BC

rho = 1.0

i1 = 1

i5 = 5

centerRho = rho

call 
DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)

Hx = 1.d0 / (mx-1)

Hy = 1.d0 / (my-1)

HxdHy = Hx/Hy

HydHx = Hy/Hx

call 
DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)

do j=ys,ys+ym-1

     do i=xs,xs+xm-1

         row(MatStencil_i) = i

         row(MatStencil_j) = j

         call ComputeRho(i,j,mx,my,centerRho,rho)

         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then

             v(1) = 2.0*rho*(HxdHy + HydHx)

             call 
MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)

         else

             v(1) = -rho*HxdHy

             col(MatStencil_i,1) = i

             col(MatStencil_j,2) = j-1

             v(2) = -rho*HydHx

             col(MatStencil_i,2) = i-1

             col(MatStencil_j,2) = j

             v(3) = 2.0*rho*(HxdHy + HydHx)

             col(MatStencil_i,3) = i

             col(MatStencil_j,3) = j

             v(4) = -rho*HydHx

             col(MatStencil_i,4) = i+1

             col(MatStencil_j,4) = j

             v(5) = -rho*HxdHy

             col(MatStencil_i,5) = i

             col(MatStencil_j,5) = j+1

             call 
MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)

         end if

     end do

end do

call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)

call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)

!call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)

!call MatSetNullSpace(jac,nullspace,ierr)

!call MatNullSpaceDestroy(nullspace,ierr)

end subroutine ComputeMatrix

subroutine ComputeRho(i,j,mx,my,centerRho,rho)

PetscInt    i,j,mx,my

PetscScalar  rho,centerRho

if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j < 
2.0*my/3.0)) then

     rho = centerRho

else

     rho = 1.0

end if


end subroutine ComputeRho

-- 
Yours sincerely,

TAY wee-beng

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/c24ed305/attachment.htm>

From knepley at gmail.com  Tue Apr 24 15:37:32 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 24 Apr 2012 16:37:32 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F970DCD.1060302@gmail.com>
References: <4F970DCD.1060302@gmail.com>
Message-ID: <CAMYG4GkhVPK3QAA3vEuiWrH3hO8ZjFVFvT6Sx9C9iWsY22oUDw@mail.gmail.com>

On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran version.
> The fortran version is based on the template of ex22f, which is in 3d.
>
> I have attached the code below. I have 2 problems.
>
> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c
> version.
>
> However, when I change i3 to -4, I got the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:08:46 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
> src/mat/impls/aij/seq/C:\temp\
> PETSC-~1\src\mat\impls\aij\seq\aij.c
> [0]PETSC ERROR: MatSetValues() line 1148 in
> src/mat/interface/C:\temp\PETSC-~1\s
> rc\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
> src/mat/interface/C:\temp\PETSC
> -~1\src\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
> src/mat/interface/C:\temp\PET
> SC-~1\src\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>
> ...
>
> What did I do wrong?
>
>
> 2. When I wanted to use the neumann BC, following ex29, I added :
>
> In subroutine ComputeRHS
>
> call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
> call MatNullSpaceDestroy(nullspace,ierr)
>
> and in subroutine ComputeMatrix
>
> call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
> call MatSetNullSpace(jac,nullspace,ierr)
>
> call MatNullSpaceDestroy(nullspace,ierr)
>
> When I compile, it says unresolved external symbol MatNullSpaceRemove.
> Removing MatNullSpaceRemove, I can compile but running gives the error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Corrupt argument:
> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:30:31 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
> src/mat/interface/C:\temp\PETSC
> -~1\src\mat\INTERF~1\matnull.c
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:30:31 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
> src/mat/interface/C:\temp\PETS
> C-~1\src\mat\INTERF~1\matnull.c
> [0]PETSC ERROR: ourdmfunction() line 30 in
> src/dm/interface/ftn-custom/C:\temp\P
> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
> [0]PETSC ERROR: DMComputeFunction() line 1783 in
> src/dm/interface/C:\temp\PETSC-
> ~1\src\dm\INTERF~1\dm.c
> [0]PETSC ERROR: KSPSetUp() line 218 in
> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
> c\ksp\ksp\INTERF~1\itfunc.c
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Corrupt argument:
> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:30:31 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
> src/mat/interface/C:\temp\PETSC
> -~1\src\mat\INTERF~1\matnull.c
> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:30:31 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
> src/mat/interface/C:\temp\PETS
> C-~1\src\mat\INTERF~1\matnull.c
> [0]PETSC ERROR: ourdmfunction() line 30 in
> src/dm/interface/ftn-custom/C:\temp\P
> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
> [0]PETSC ERROR: DMComputeFunction() line 1783 in
> src/dm/interface/C:\temp\PETSC-
> ~1\src\dm\INTERF~1\dm.c
> [0]PETSC ERROR: KSPSetUp() line 218 in
> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
> c\ksp\ksp\INTERF~1\itfunc.c
> [0]PETSC ERROR: KSPSolve() line 402 in
> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
> c\ksp\ksp\INTERF~1\itfunc.c
> Vector Object:Vec_0000000084000000_0 1 MPI processes
>   type: mpi
> Process [0]
>
> What's wrong?
>
>
> *program ex29f*
>
> implicit none
>
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                    Include files
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !
> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
> !     petscmat.h - matrices
> !     petscksp.h    - Krylov subspace methods  petscpc.h  - preconditioners
>
> #include "finclude/petsc.h90"
>
> PetscErrorCode   ierr
>
> DM               da
>
> KSP              ksp
>
> Vec              x
>
> external         ComputeRHS,ComputeMatrix
>
> PetscInt i1,i3
>
> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>
> i3 = -4
>
> i1 = 1
>
> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>
> call
> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> call DMSetFunction(da,ComputeRHS,ierr)
> call DMSetJacobian(da,ComputeMatrix,ierr)
> call KSPSetDM(ksp,da,ierr)
>
> call KSPSetFromOptions(ksp,ierr)
> call KSPSetUp(ksp,ierr)
> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
> call KSPGetSolution(ksp,x,ierr)
> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
> call KSPDestroy(ksp,ierr)
> call DMDestroy(da,ierr)
> call PetscFinalize(ierr)
>
> end program ex29f
>
> subroutine ComputeRHS(da,x,b,ierr)
>
> implicit none
>
> #include "finclude/petsc.h90"
>
> PetscErrorCode  ierr
> PetscInt i,j,mx,my,xm,ym,xs,ys
> PetscScalar  h,nu,rho
> PetscScalar Hx,Hy
> PetscScalar,pointer :: array(:,:)
> Vec          x,b
> DM           da
> MatNullSpace nullspace    !>neumann BC
>
> nu = 0.1
>
> call
> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>
> Hx = 1.d0 / (mx-1)
>
> Hy = 1.d0 / (my-1)
>
> call
> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
> call DMDAVecGetArrayF90(da,b,array,ierr)
>
> do j = ys,ys+ym-1
>
>     do i = xs,xs+xm-1
>
>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>
>     end do
>
> end do
>
> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>
> call VecAssemblyBegin(b,ierr)
>
> call VecAssemblyEnd(b,ierr)
>
> !call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>

4th argument should be PETSC_NULL_OBJECT.

   Matt


> !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
> !call MatNullSpaceDestroy(nullspace,ierr)
>
> end subroutine ComputeRHS
>
>
> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>
> implicit none
>
> #include "finclude/petsc.h90"
>
> Mat          jac,JJ
>
> PetscErrorCode    ierr
>
> DM           da
>
> PetscInt    i,j,k,mx,my,xm
>
> PetscInt    ym,xs,ys,i1,i5
>
> PetscScalar  v(5),Hx,Hy,rho,centerRho
>
> PetscScalar  HydHx
>
> PetscScalar  HxdHy
>
> MatStencil   row(4),col(4,5)
>
> Vec          x
>
> MatStructure str
>
> MatNullSpace nullspace    !>neumann BC
>
> rho = 1.0
>
> i1 = 1
>
> i5 = 5
>
> centerRho = rho
>
> call
> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>
> Hx = 1.d0 / (mx-1)
>
> Hy = 1.d0 / (my-1)
>
> HxdHy = Hx/Hy
>
> HydHx = Hy/Hx
>
> call
> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
>
> do j=ys,ys+ym-1
>
>     do i=xs,xs+xm-1
>
>         row(MatStencil_i) = i
>
>         row(MatStencil_j) = j
>
>         call ComputeRho(i,j,mx,my,centerRho,rho)
>
>         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>
>             v(1) = 2.0*rho*(HxdHy + HydHx)
>
>             call
> MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>
>         else
>
>             v(1) = -rho*HxdHy
>
>             col(MatStencil_i,1) = i
>
>             col(MatStencil_j,2) = j-1
>

Cut and paste error above.

  Matt


>             v(2) = -rho*HydHx
>
>             col(MatStencil_i,2) = i-1
>
>             col(MatStencil_j,2) = j
>
>             v(3) = 2.0*rho*(HxdHy + HydHx)
>
>             col(MatStencil_i,3) = i
>
>             col(MatStencil_j,3) = j
>
>             v(4) = -rho*HydHx
>
>             col(MatStencil_i,4) = i+1
>
>             col(MatStencil_j,4) = j
>
>             v(5) = -rho*HxdHy
>
>             col(MatStencil_i,5) = i
>
>             col(MatStencil_j,5) = j+1
>
>             call
> MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>
>         end if
>
>     end do
>
> end do
>
> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>
> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>
> !call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
> !call MatSetNullSpace(jac,nullspace,ierr)
>
> !call MatNullSpaceDestroy(nullspace,ierr)
>
> end subroutine ComputeMatrix
>
> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>
> PetscInt    i,j,mx,my
>
> PetscScalar  rho,centerRho
>
> if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j <
> 2.0*my/3.0)) then
>
>     rho = centerRho
>
> else
>
>     rho = 1.0
>
> end if
>
>
> end subroutine ComputeRho
>
> --
> Yours sincerely,
>
> TAY wee-beng
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/4795406c/attachment-0001.htm>

From jiangwen84 at gmail.com  Tue Apr 24 16:01:41 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Tue, 24 Apr 2012 17:01:41 -0400
Subject: [petsc-users] mumps get stuck with parmetis
Message-ID: <CAMJxm+A5nka320vT2soLHv8uWmu-ucANqQ==ON5YPwPFoQSb+g@mail.gmail.com>

Thanks for all your replies.

Firstly, I repeat the ex2 for PETSc 3.2 with the options
-mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 and it is running without any
problem.

For my problem(3D FEM code), the matrix size is around 0.2 million and
solved by preonly lu. The code is running on 32 processors. I add
-mat_mumps_icntl_4 1 and -info. However, the mumps did not output any
information. The PETSc information output gets stuck at "[0]
VecScatterCreate(): Special case: processor zero gets entire parallel
vector, rest get none". Some more information are attached at end of this
email.

I still cannot locate the problem. Any suggestions? Thanks.

Regards,
Wen

**************************************************************************************
[0] PCSetUp(): Setting up new PC
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374780
[14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
-2080374777
[0] VecScatterCreate(): Special case: processor zero gets entire parallel
vector, rest get none
**************************************************************************************************************

On Tue, Apr 24, 2012 at 3:41 PM, <petsc-users-request at mcs.anl.gov> wrote:

> Send petsc-users mailing list submissions to
>        petsc-users at mcs.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> or, via email, send a message with subject or body 'help' to
>        petsc-users-request at mcs.anl.gov
>
> You can reach the person managing the list at
>        petsc-users-owner at mcs.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of petsc-users digest..."
>
>
> Today's Topics:
>
>   1.  mumps get stuck with parmetis (Wen Jiang)
>   2. Re:  mumps get stuck with parmetis (Hong Zhang)
>   3. Re:  mumps get stuck with parmetis ( Alexander Grayver )
>   4. Re:  mumps get stuck with parmetis (Aron Ahmadia)
>   5. Re:  mumps get stuck with parmetis ( Alexander Grayver )
>   6. Re:  mumps get stuck with parmetis (Aron Ahmadia)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 24 Apr 2012 13:43:06 -0400
> From: Wen Jiang <jiangwen84 at gmail.com>
> Subject: [petsc-users] mumps get stuck with parmetis
> To: petsc-users at mcs.anl.gov
> Message-ID:
>        <CAMJxm+Da_byqMzm-_j+hRBUJSBt25oQAfDmUnVqHgBS4VPy8_A at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi,
>
> My code will hang at the solving stage when I use mumps with the runtime
> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> options, my code works fine. I am using PETSc 3.2 and configure it
> with --download-mumps=1
> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>
> Regards,
> Wen
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/edd22bfe/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Tue, 24 Apr 2012 13:29:32 -0500
> From: Hong Zhang <hzhang at mcs.anl.gov>
> Subject: Re: [petsc-users] mumps get stuck with parmetis
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAGCphBurn0+f--gOZ_4MSrLcntvYSVCQ2_cmSG93nvv3hHSEjA at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> Wen :
> I cannot repeat your error with petsc-dev. Running
> petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c:
> mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps
> -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2
> Norm of error 1.67272e-15 iterations 1
>
> Can you run above with your petsc-3.2 installation?
>
> Hong
>
> Hi,
> >
> > My code will hang at the solving stage when I use mumps with the runtime
> > option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> > options, my code works fine. I am using PETSc 3.2 and configure it with
> --download-mumps=1
> > and --download-parmetis=1. Could anyone give me any hints? Thanks.
> >
> > Regards,
> > Wen
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/196ff3c5/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 3
> Date: Tue, 24 Apr 2012 21:01:45 +0200
> From: " Alexander Grayver " <agrayver at gfz-potsdam.de>
> Subject: Re: [petsc-users] mumps get stuck with parmetis
> To: " PETSc users list " <petsc-users at mcs.anl.gov>
> Message-ID: <auto-000008199816 at cgp2.gfz-potsdam.de>
> Content-Type: text/plain; charset="utf-8"
>
> Can you set:
>
> -mat_mumps_icntl_4 1
>
> And send mumps output?
> Also do you use lu or ilu? How large is your matrix?
>
> Regards,
> Alexander
>
> ----- Reply message -----
> From: "Wen Jiang" <jiangwen84 at gmail.com>
> To: <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] mumps get stuck with parmetis
> Date: Tue, Apr 24, 2012 19:43
> Hi,
>
> My code will hang at the solving stage when I use mumps with the runtime
> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> options, my code works fine. I am using PETSc 3.2 and configure it with
> --download-mumps=1 and --download-parmetis=1. Could anyone give me any
> hints? Thanks.
>
>
> Regards,
> Wen
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2eb1767b/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 4
> Date: Tue, 24 Apr 2012 22:13:59 +0300
> From: Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
> Subject: Re: [petsc-users] mumps get stuck with parmetis
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CADVoUZQDYqOpoZtxWY1DGE=WGgbZvxMVuRy8zRxYTtrSCMP3sg at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on BlueGene/P
> was causing similar behavior without even setting any options.  The only
> way I was able to get a direct solver going was by switching over to
> SuperLU.
>
> A
>
> On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
> agrayver at gfz-potsdam.de
> > wrote:
>
> > Can you set:
> >
> > -mat_mumps_icntl_4 1
> >
> > And send mumps output?
> > Also do you use lu or ilu? How large is your matrix?
> >
> > Regards,
> > Alexander
> >
> >
> > ----- Reply message -----
> > From: "Wen Jiang" <jiangwen84 at gmail.com>
> > To: <petsc-users at mcs.anl.gov>
> > Subject: [petsc-users] mumps get stuck with parmetis
> > Date: Tue, Apr 24, 2012 19:43
> >
> >
> > Hi,
> >
> > My code will hang at the solving stage when I use mumps with the runtime
> > option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> > options, my code works fine. I am using PETSc 3.2 and configure it with
> --download-mumps=1
> > and --download-parmetis=1. Could anyone give me any hints? Thanks.
> >
> > Regards,
> > Wen
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/b0503b1c/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 5
> Date: Tue, 24 Apr 2012 21:22:51 +0200
> From: " Alexander Grayver " <agrayver at gfz-potsdam.de>
> Subject: Re: [petsc-users] mumps get stuck with parmetis
> To: " PETSc users list " <petsc-users at mcs.anl.gov>
> Message-ID: <auto-000056660896 at cgp1.gfz-potsdam.de>
> Content-Type: text/plain; charset="utf-8"
>
> Aron,
>
> This parameter let's to see mumps output in console. The important this to
> understand where mumps hangs, during analysis, factorization or actual
> solution (substitutions)? I'm almost sure it's factorization step. I
> observe this pretty often with mumps compiled with petsc (whereas when
> mumps is used directly it's quite rare to come along with this problem).
>
> Regards,
> Alexander
>
> ----- Reply message -----
> From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] mumps get stuck with parmetis
> Date: Tue, Apr 24, 2012 21:13
> I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
> BlueGene/P was causing similar behavior without even setting any options.
> ?The only way I was able to get a direct solver going was by switching over
> to SuperLU.
>
> A
>
> On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
> agrayver at gfz-potsdam.de> wrote:
>
> Can you set:
>
> -mat_mumps_icntl_4 1
>
> And send mumps output?
> Also do you use lu or ilu? How large is your matrix?
>
>
> Regards,
> Alexander
>
> ----- Reply message -----
> From: "Wen Jiang" <jiangwen84 at gmail.com>
>
> To: <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] mumps get stuck with parmetis
> Date: Tue, Apr 24, 2012 19:43
>
>
> Hi,
>
> My code will hang at the solving stage when I use mumps with the runtime
> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
> options, my code works fine. I am using PETSc 3.2 and configure it with
> --download-mumps=1 and --download-parmetis=1. Could anyone give me any
> hints? Thanks.
>
>
>
> Regards,
> Wen
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2db311da/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 6
> Date: Tue, 24 Apr 2012 22:41:00 +0300
> From: Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
> Subject: Re: [petsc-users] mumps get stuck with parmetis
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CADVoUZSL+UXMJ0V52ELpE2vLLXc6WbZe96+yL5tYPsyMYiLXeQ at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> /project/k121/sandbox/petsc
> -dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_part9.F:4666
> /project/k121/sandbox/petsc
> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part5.F:465
> /project/k121/sandbox/petsc
> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part1.F:409
> /project/k121/sandbox/petsc
> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part3.F:6651
> /project/k121/sandbox/petsc
> -dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_c.c:422
>
> I don't know the MUMPS source code very well so I couldn't tell you what
> this set of routines are doing, but this is a snippet of the stack trace I
> was seeing when the jobs died on BG/P.
>
> If you set the "-info" flag on a PETSc run, it sends a lot of debugging
> output to the screen, which is useful when you're in a situation where it
> is hard to get access to a debugger or the stack trace.
>
> A
>
> On Tue, Apr 24, 2012 at 10:22 PM, Alexander Grayver <
> agrayver at gfz-potsdam.de
> > wrote:
>
> > Aron,
> >
> > This parameter let's to see mumps output in console. The important this
> to
> > understand where mumps hangs, during analysis, factorization or actual
> > solution (substitutions)? I'm almost sure it's factorization step. I
> > observe this pretty often with mumps compiled with petsc (whereas when
> > mumps is used directly it's quite rare to come along with this problem).
> >
> > Regards,
> > Alexander
> >
> > ----- Reply message -----
> > From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
> > To: "PETSc users list" <petsc-users at mcs.anl.gov>
> > Subject: [petsc-users] mumps get stuck with parmetis
> > Date: Tue, Apr 24, 2012 21:13
> >
> >
> > I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
> > BlueGene/P was causing similar behavior without even setting any options.
> >  The only way I was able to get a direct solver going was by switching
> over
> > to SuperLU.
> >
> > A
> >
> > On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
> > agrayver at gfz-potsdam.de> wrote:
> >
> >> Can you set:
> >>
> >> -mat_mumps_icntl_4 1
> >>
> >> And send mumps output?
> >> Also do you use lu or ilu? How large is your matrix?
> >>
> >> Regards,
> >> Alexander
> >>
> >>
> >> ----- Reply message -----
> >> From: "Wen Jiang" <jiangwen84 at gmail.com>
> >> To: <petsc-users at mcs.anl.gov>
> >> Subject: [petsc-users] mumps get stuck with parmetis
> >> Date: Tue, Apr 24, 2012 19:43
> >>
> >>
> >> Hi,
> >>
> >> My code will hang at the solving stage when I use mumps with the runtime
> >> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these
> two
> >> options, my code works fine. I am using PETSc 3.2 and configure it with
> --download-mumps=1
> >> and --download-parmetis=1. Could anyone give me any hints? Thanks.
> >>
> >> Regards,
> >> Wen
> >>
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/3efe54e8/attachment.htm
> >
>
> ------------------------------
>
> _______________________________________________
> petsc-users mailing list
> petsc-users at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>
>
> End of petsc-users Digest, Vol 40, Issue 76
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/b30f9ae1/attachment-0001.htm>

From zonexo at gmail.com  Tue Apr 24 15:57:03 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 24 Apr 2012 22:57:03 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4GkhVPK3QAA3vEuiWrH3hO8ZjFVFvT6Sx9C9iWsY22oUDw@mail.gmail.com>
References: <4F970DCD.1060302@gmail.com>
	<CAMYG4GkhVPK3QAA3vEuiWrH3hO8ZjFVFvT6Sx9C9iWsY22oUDw@mail.gmail.com>
Message-ID: <4F97139F.2030205@gmail.com>

Hi,

I still got the same error the moment this subroutine is called, after 
changing to :

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)


[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 4!
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: Petsc Development HG revision: 
fc934c1d84bc6ba8e2686702a8a99539d
50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named 
USER-PC by
  User Tue Apr 24 22:54:50 2012
[0]PETSC ERROR: Libraries linked from 
/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
/lib
[0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" 
--with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ 
--download-f-b
las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 
--with-debuggin
g=1 --useThreads=0
[0]PETSC ERROR: 
----------------------------------------------------------------
--------
[0]PETSC ERROR: MatNullSpaceCreate() line 250 in 
src/mat/interface/C:\temp\PETSC
-~1\src\mat\INTERF~1\matnull.c

Yours sincerely,

TAY wee-beng


On 24/4/2012 10:37 PM, Matthew Knepley wrote:
> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran
>     version. The fortran version is based on the template of ex22f,
>     which is in 3d.
>
>     I have attached the code below. I have 2 problems.
>
>     1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in
>     c version.
>
>     However, when I change i3 to -4, I got the following error:
>
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Argument out of range!
>     [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:08:46 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>     src/mat/impls/aij/seq/C:\temp\
>     PETSC-~1\src\mat\impls\aij\seq\aij.c
>     [0]PETSC ERROR: MatSetValues() line 1148 in
>     src/mat/interface/C:\temp\PETSC-~1\s
>     rc\mat\INTERF~1\matrix.c
>     [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>     src/mat/interface/C:\temp\PETSC
>     -~1\src\mat\INTERF~1\matrix.c
>     [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>     src/mat/interface/C:\temp\PET
>     SC-~1\src\mat\INTERF~1\matrix.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Argument out of range!
>     [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>
>     ...
>
>     What did I do wrong?
>
>
>     2. When I wanted to use the neumann BC, following ex29, I added :
>
>     In subroutine ComputeRHS
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
>     call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>     call MatNullSpaceDestroy(nullspace,ierr)
>
>     and in subroutine ComputeMatrix
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
>     call MatSetNullSpace(jac,nullspace,ierr)
>
>     call MatNullSpaceDestroy(nullspace,ierr)
>
>     When I compile, it says unresolved external symbol
>     MatNullSpaceRemove. Removing MatNullSpaceRemove, I can compile but
>     running gives the error:
>
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Corrupt argument:
>     see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>     [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:30:31 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>     src/mat/interface/C:\temp\PETSC
>     -~1\src\mat\INTERF~1\matnull.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Invalid argument!
>     [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:30:31 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>     src/mat/interface/C:\temp\PETS
>     C-~1\src\mat\INTERF~1\matnull.c
>     [0]PETSC ERROR: ourdmfunction() line 30 in
>     src/dm/interface/ftn-custom/C:\temp\P
>     ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>     [0]PETSC ERROR: DMComputeFunction() line 1783 in
>     src/dm/interface/C:\temp\PETSC-
>     ~1\src\dm\INTERF~1\dm.c
>     [0]PETSC ERROR: KSPSetUp() line 218 in
>     src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>     c\ksp\ksp\INTERF~1\itfunc.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Corrupt argument:
>     see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>     [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:30:31 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>     src/mat/interface/C:\temp\PETSC
>     -~1\src\mat\INTERF~1\matnull.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Invalid argument!
>     [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:30:31 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>     src/mat/interface/C:\temp\PETS
>     C-~1\src\mat\INTERF~1\matnull.c
>     [0]PETSC ERROR: ourdmfunction() line 30 in
>     src/dm/interface/ftn-custom/C:\temp\P
>     ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>     [0]PETSC ERROR: DMComputeFunction() line 1783 in
>     src/dm/interface/C:\temp\PETSC-
>     ~1\src\dm\INTERF~1\dm.c
>     [0]PETSC ERROR: KSPSetUp() line 218 in
>     src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>     c\ksp\ksp\INTERF~1\itfunc.c
>     [0]PETSC ERROR: KSPSolve() line 402 in
>     src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>     c\ksp\ksp\INTERF~1\itfunc.c
>     Vector Object:Vec_0000000084000000_0 1 MPI processes
>       type: mpi
>     Process [0]
>
>     What's wrong?
>
>
>     *program ex29f*
>
>     implicit none
>
>     ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     - - - -
>     !                    Include files
>     ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     - - - -
>     !
>     !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>     !     petscmat.h - matrices
>     !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>     preconditioners
>
>     #include "finclude/petsc.h90"
>
>     PetscErrorCode   ierr
>
>     DM               da
>
>     KSP              ksp
>
>     Vec              x
>
>     external         ComputeRHS,ComputeMatrix
>
>     PetscInt i1,i3
>
>     call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>
>     i3 = -4
>
>     i1 = 1
>
>     call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>
>     call
>     DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>     call DMSetFunction(da,ComputeRHS,ierr)
>     call DMSetJacobian(da,ComputeMatrix,ierr)
>     call KSPSetDM(ksp,da,ierr)
>
>     call KSPSetFromOptions(ksp,ierr)
>     call KSPSetUp(ksp,ierr)
>     call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>     call KSPGetSolution(ksp,x,ierr)
>     call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>     call KSPDestroy(ksp,ierr)
>     call DMDestroy(da,ierr)
>     call PetscFinalize(ierr)
>
>     end program ex29f
>
>     subroutine ComputeRHS(da,x,b,ierr)
>
>     implicit none
>
>     #include "finclude/petsc.h90"
>
>     PetscErrorCode  ierr
>     PetscInt i,j,mx,my,xm,ym,xs,ys
>     PetscScalar  h,nu,rho
>     PetscScalar Hx,Hy
>     PetscScalar,pointer :: array(:,:)
>     Vec          x,b
>     DM           da
>     MatNullSpace nullspace    !>neumann BC
>
>     nu = 0.1
>
>     call
>     DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>
>     Hx = 1.d0 / (mx-1)
>
>     Hy = 1.d0 / (my-1)
>
>     call
>     DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
>     call DMDAVecGetArrayF90(da,b,array,ierr)
>
>     do j = ys,ys+ym-1
>
>         do i = xs,xs+xm-1
>
>             array(i,j) =
>     exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>
>         end do
>
>     end do
>
>     call DMDAVecRestoreArrayF90(da,b,array,ierr)
>
>     call VecAssemblyBegin(b,ierr)
>
>     call VecAssemblyEnd(b,ierr)
>
>     !call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
>
> 4th argument should be PETSC_NULL_OBJECT.
>
>    Matt
>
>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>     !call MatNullSpaceDestroy(nullspace,ierr)
>
>     end subroutine ComputeRHS
>
>
>     subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>
>     implicit none
>
>     #include "finclude/petsc.h90"
>
>     Mat          jac,JJ
>
>     PetscErrorCode    ierr
>
>     DM           da
>
>     PetscInt    i,j,k,mx,my,xm
>
>     PetscInt    ym,xs,ys,i1,i5
>
>     PetscScalar  v(5),Hx,Hy,rho,centerRho
>
>     PetscScalar  HydHx
>
>     PetscScalar  HxdHy
>
>     MatStencil   row(4),col(4,5)
>
>     Vec          x
>
>     MatStructure str
>
>     MatNullSpace nullspace    !>neumann BC
>
>     rho = 1.0
>
>     i1 = 1
>
>     i5 = 5
>
>     centerRho = rho
>
>     call
>     DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>
>     Hx = 1.d0 / (mx-1)
>
>     Hy = 1.d0 / (my-1)
>
>     HxdHy = Hx/Hy
>
>     HydHx = Hy/Hx
>
>     call
>     DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
>
>     do j=ys,ys+ym-1
>
>         do i=xs,xs+xm-1
>
>             row(MatStencil_i) = i
>
>             row(MatStencil_j) = j
>
>             call ComputeRho(i,j,mx,my,centerRho,rho)
>
>             if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>
>                 v(1) = 2.0*rho*(HxdHy + HydHx)
>
>                 call
>     MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>
>             else
>
>                 v(1) = -rho*HxdHy
>
>                 col(MatStencil_i,1) = i
>
>                 col(MatStencil_j,2) = j-1
>
>
> Cut and paste error above.
>
>   Matt
>
>                 v(2) = -rho*HydHx
>
>                 col(MatStencil_i,2) = i-1
>
>                 col(MatStencil_j,2) = j
>
>                 v(3) = 2.0*rho*(HxdHy + HydHx)
>
>                 col(MatStencil_i,3) = i
>
>                 col(MatStencil_j,3) = j
>
>                 v(4) = -rho*HydHx
>
>                 col(MatStencil_i,4) = i+1
>
>                 col(MatStencil_j,4) = j
>
>                 v(5) = -rho*HxdHy
>
>                 col(MatStencil_i,5) = i
>
>                 col(MatStencil_j,5) = j+1
>
>                 call
>     MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>
>             end if
>
>         end do
>
>     end do
>
>     call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>
>     call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>
>     !call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>
>     !call MatSetNullSpace(jac,nullspace,ierr)
>
>     !call MatNullSpaceDestroy(nullspace,ierr)
>
>     end subroutine ComputeMatrix
>
>     subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>
>     PetscInt    i,j,mx,my
>
>     PetscScalar  rho,centerRho
>
>     if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and.
>     (j < 2.0*my/3.0)) then
>
>         rho = centerRho
>
>     else
>
>         rho = 1.0
>
>     end if
>
>
>     end subroutine ComputeRho
>
>     -- 
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/070c3e2f/attachment-0001.htm>

From knepley at gmail.com  Tue Apr 24 17:06:43 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 24 Apr 2012 18:06:43 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F97139F.2030205@gmail.com>
References: <4F970DCD.1060302@gmail.com>
	<CAMYG4GkhVPK3QAA3vEuiWrH3hO8ZjFVFvT6Sx9C9iWsY22oUDw@mail.gmail.com>
	<4F97139F.2030205@gmail.com>
Message-ID: <CAMYG4Gk-fXEXyj5L__QMgnGc4f9fuoEEj4W_Cm4Nmcdy72TV0A@mail.gmail.com>

On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I still got the same error the moment this subroutine is called, after
> changing to :
>
> call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>

Argument 3 should be 0.

  Matt


> [0]PETSC ERROR: Null argument, when expecting valid pointer!
> [0]PETSC ERROR: Null Object: Parameter # 4!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: Petsc Development HG revision:
> fc934c1d84bc6ba8e2686702a8a99539d
> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
> USER-PC by
>  User Tue Apr 24 22:54:50 2012
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> /lib
> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
> --with-fc="win32fe ifor
> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
> --download-f-b
> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
> --with-debuggin
> g=1 --useThreads=0
> [0]PETSC ERROR:
> ----------------------------------------------------------------
> --------
> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
> src/mat/interface/C:\temp\PETSC
> -~1\src\mat\INTERF~1\matnull.c
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>
> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran version.
>> The fortran version is based on the template of ex22f, which is in 3d.
>>
>> I have attached the code below. I have 2 problems.
>>
>> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c
>> version.
>>
>> However, when I change i3 to -4, I got the following error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:08:46 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>> src/mat/impls/aij/seq/C:\temp\
>> PETSC-~1\src\mat\impls\aij\seq\aij.c
>> [0]PETSC ERROR: MatSetValues() line 1148 in
>> src/mat/interface/C:\temp\PETSC-~1\s
>> rc\mat\INTERF~1\matrix.c
>> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>> src/mat/interface/C:\temp\PETSC
>> -~1\src\mat\INTERF~1\matrix.c
>> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>> src/mat/interface/C:\temp\PET
>> SC-~1\src\mat\INTERF~1\matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>
>> ...
>>
>> What did I do wrong?
>>
>>
>> 2. When I wanted to use the neumann BC, following ex29, I added :
>>
>> In subroutine ComputeRHS
>>
>> call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>> call MatNullSpaceDestroy(nullspace,ierr)
>>
>> and in subroutine ComputeMatrix
>>
>> call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>> call MatSetNullSpace(jac,nullspace,ierr)
>>
>> call MatNullSpaceDestroy(nullspace,ierr)
>>
>> When I compile, it says unresolved external symbol MatNullSpaceRemove.
>> Removing MatNullSpaceRemove, I can compile but running gives the error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Corrupt argument:
>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:30:31 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>> src/mat/interface/C:\temp\PETSC
>> -~1\src\mat\INTERF~1\matnull.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Invalid argument!
>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:30:31 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>> src/mat/interface/C:\temp\PETS
>> C-~1\src\mat\INTERF~1\matnull.c
>> [0]PETSC ERROR: ourdmfunction() line 30 in
>> src/dm/interface/ftn-custom/C:\temp\P
>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>> src/dm/interface/C:\temp\PETSC-
>> ~1\src\dm\INTERF~1\dm.c
>> [0]PETSC ERROR: KSPSetUp() line 218 in
>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>> c\ksp\ksp\INTERF~1\itfunc.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Corrupt argument:
>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:30:31 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>> src/mat/interface/C:\temp\PETSC
>> -~1\src\mat\INTERF~1\matnull.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ----------------------------
>> --------
>> [0]PETSC ERROR: Invalid argument!
>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:30:31 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>> src/mat/interface/C:\temp\PETS
>> C-~1\src\mat\INTERF~1\matnull.c
>> [0]PETSC ERROR: ourdmfunction() line 30 in
>> src/dm/interface/ftn-custom/C:\temp\P
>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>> src/dm/interface/C:\temp\PETSC-
>> ~1\src\dm\INTERF~1\dm.c
>> [0]PETSC ERROR: KSPSetUp() line 218 in
>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>> c\ksp\ksp\INTERF~1\itfunc.c
>> [0]PETSC ERROR: KSPSolve() line 402 in
>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>> c\ksp\ksp\INTERF~1\itfunc.c
>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>   type: mpi
>> Process [0]
>>
>> What's wrong?
>>
>>
>> *program ex29f*
>>
>> implicit none
>>
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> !                    Include files
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> !
>> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>> !     petscmat.h - matrices
>> !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>> preconditioners
>>
>> #include "finclude/petsc.h90"
>>
>> PetscErrorCode   ierr
>>
>> DM               da
>>
>> KSP              ksp
>>
>> Vec              x
>>
>> external         ComputeRHS,ComputeMatrix
>>
>> PetscInt i1,i3
>>
>> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>
>> i3 = -4
>>
>> i1 = 1
>>
>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>
>> call
>> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>> call DMSetFunction(da,ComputeRHS,ierr)
>> call DMSetJacobian(da,ComputeMatrix,ierr)
>> call KSPSetDM(ksp,da,ierr)
>>
>> call KSPSetFromOptions(ksp,ierr)
>> call KSPSetUp(ksp,ierr)
>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>> call KSPGetSolution(ksp,x,ierr)
>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>> call KSPDestroy(ksp,ierr)
>> call DMDestroy(da,ierr)
>> call PetscFinalize(ierr)
>>
>> end program ex29f
>>
>> subroutine ComputeRHS(da,x,b,ierr)
>>
>> implicit none
>>
>> #include "finclude/petsc.h90"
>>
>> PetscErrorCode  ierr
>> PetscInt i,j,mx,my,xm,ym,xs,ys
>> PetscScalar  h,nu,rho
>> PetscScalar Hx,Hy
>> PetscScalar,pointer :: array(:,:)
>> Vec          x,b
>> DM           da
>> MatNullSpace nullspace    !>neumann BC
>>
>> nu = 0.1
>>
>> call
>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>> Hx = 1.d0 / (mx-1)
>>
>> Hy = 1.d0 / (my-1)
>>
>> call
>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>> do j = ys,ys+ym-1
>>
>>     do i = xs,xs+xm-1
>>
>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>     end do
>>
>> end do
>>
>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>> call VecAssemblyBegin(b,ierr)
>>
>> call VecAssemblyEnd(b,ierr)
>>
>> !call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>
>  4th argument should be PETSC_NULL_OBJECT.
>
>     Matt
>
>
>>  !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>> !call MatNullSpaceDestroy(nullspace,ierr)
>>
>> end subroutine ComputeRHS
>>
>>
>> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>
>> implicit none
>>
>> #include "finclude/petsc.h90"
>>
>> Mat          jac,JJ
>>
>> PetscErrorCode    ierr
>>
>> DM           da
>>
>> PetscInt    i,j,k,mx,my,xm
>>
>> PetscInt    ym,xs,ys,i1,i5
>>
>> PetscScalar  v(5),Hx,Hy,rho,centerRho
>>
>> PetscScalar  HydHx
>>
>> PetscScalar  HxdHy
>>
>> MatStencil   row(4),col(4,5)
>>
>> Vec          x
>>
>> MatStructure str
>>
>> MatNullSpace nullspace    !>neumann BC
>>
>> rho = 1.0
>>
>> i1 = 1
>>
>> i5 = 5
>>
>> centerRho = rho
>>
>> call
>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>> Hx = 1.d0 / (mx-1)
>>
>> Hy = 1.d0 / (my-1)
>>
>> HxdHy = Hx/Hy
>>
>> HydHx = Hy/Hx
>>
>> call
>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>
>> do j=ys,ys+ym-1
>>
>>     do i=xs,xs+xm-1
>>
>>         row(MatStencil_i) = i
>>
>>         row(MatStencil_j) = j
>>
>>         call ComputeRho(i,j,mx,my,centerRho,rho)
>>
>>         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>>
>>             v(1) = 2.0*rho*(HxdHy + HydHx)
>>
>>             call
>> MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>
>>         else
>>
>>             v(1) = -rho*HxdHy
>>
>>             col(MatStencil_i,1) = i
>>
>>             col(MatStencil_j,2) = j-1
>>
>
>  Cut and paste error above.
>
>    Matt
>
>
>>              v(2) = -rho*HydHx
>>
>>             col(MatStencil_i,2) = i-1
>>
>>             col(MatStencil_j,2) = j
>>
>>             v(3) = 2.0*rho*(HxdHy + HydHx)
>>
>>             col(MatStencil_i,3) = i
>>
>>             col(MatStencil_j,3) = j
>>
>>             v(4) = -rho*HydHx
>>
>>             col(MatStencil_i,4) = i+1
>>
>>             col(MatStencil_j,4) = j
>>
>>             v(5) = -rho*HxdHy
>>
>>             col(MatStencil_i,5) = i
>>
>>             col(MatStencil_j,5) = j+1
>>
>>             call
>> MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>
>>         end if
>>
>>     end do
>>
>> end do
>>
>> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>> !call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>> !call MatSetNullSpace(jac,nullspace,ierr)
>>
>> !call MatNullSpaceDestroy(nullspace,ierr)
>>
>> end subroutine ComputeMatrix
>>
>> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>
>> PetscInt    i,j,mx,my
>>
>> PetscScalar  rho,centerRho
>>
>> if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j <
>> 2.0*my/3.0)) then
>>
>>     rho = centerRho
>>
>> else
>>
>>     rho = 1.0
>>
>> end if
>>
>>
>> end subroutine ComputeRho
>>
>> --
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/4048f55b/attachment-0001.htm>

From mirzadeh at gmail.com  Tue Apr 24 18:01:04 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 24 Apr 2012 16:01:04 -0700
Subject: [petsc-users] error in petsc-dev
Message-ID: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>

Hi,

While trying to figure out a problem, I came across the following
situation. Consider the following code:

int main (int argc, char **argv){

    PetscInitialize(&argc, &argv, (char*)0, help);


    Mat m;

    MatCreate(PETSC_COMM_WORLD, &m);

    MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);

    MatSetFromOptions(m);

    MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);

    MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);

    MatView(m, PETSC_VIEWER_STDOUT_WORLD);

    MatDestroy(&m);


    PetscFinalize();

    return 0;

}


This runs without any problem under 3.2-p6 but fails with petsc-dev:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
argument 1 "mat" before MatAssemblyBegin()!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 15:55:40 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatAssemblyBegin() line 4810 in
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatAssemblyEnd_SeqAIJ line 800
/home/mohammad/soft/petsc-dev/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: [0] MatAssemblyEnd line 4984
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] MatAssemblyBegin line 4807
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 15:55:40 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0


Eventually I could fix this by adding MatSetUp(m) after setting the
options. Why do I need this in petsc-dev? Does this somehow preallocate the
matrix?

Thanks,
Mohammad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/4e5a494a/attachment.htm>

From hzhang at mcs.anl.gov  Tue Apr 24 18:22:21 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 24 Apr 2012 18:22:21 -0500
Subject: [petsc-users] mumps get stuck with parmetis
In-Reply-To: <CAMJxm+A5nka320vT2soLHv8uWmu-ucANqQ==ON5YPwPFoQSb+g@mail.gmail.com>
References: <CAMJxm+A5nka320vT2soLHv8uWmu-ucANqQ==ON5YPwPFoQSb+g@mail.gmail.com>
Message-ID: <CAGCphBt3eXjSEfXrK+hO+FRRvQ1WC19Pp+2BZauBo+CB6=Ctxw@mail.gmail.com>

Wen :

> Thanks for all your replies.
>
> Firstly, I repeat the ex2 for PETSc 3.2 with the options
> -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 and it is running without any
> problem.
>
> For my problem(3D FEM code), the matrix size is around 0.2 million and
> solved by preonly lu. The code is running on 32 processors. I add
> -mat_mumps_icntl_4 1 and -info. However, the mumps did not output any
> information. The PETSc information output gets stuck at "[0]
> VecScatterCreate(): Special case: processor zero gets entire parallel
> vector, rest get none". Some more information are attached at end of this
> email.
>

0.2 million with 32 processors, i.e., each processor holds approx. 6k
equations is reasonable size for mumps
unless lu factorization causes large amount of fill-ins.
' -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2' activates parallel symbolic
factorization and is relatively new
from mumps. Unless symbolic factorization (called analysis phase in mumps)
takes large portion of your run,
which I've never seen from my experiments,why not use default options of
petsc interface (i.e., sequential analysis)
and check the output of '-log_summary' to see if you really need optimize
this phase by parallelization.

>From the errors reported, hang occurs inside mumps (seems
in MPI_ALLREDUCE()).
Suggest report the problem to mumps developers. They are very supportive.

Hong




>
> I still cannot locate the problem. Any suggestions? Thanks.
>
> Regards,
> Wen
>
>
> **************************************************************************************
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [5] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [4] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [3] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [2] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [31] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [11] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [10] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [7] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [25] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [24] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [29] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [15] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [16] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [18] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [30] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [6] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [8] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [28] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [13] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [26] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [9] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [12] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [27] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [21] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [20] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [23] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [19] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [17] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [22] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [14] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374777
> [0] VecScatterCreate(): Special case: processor zero gets entire parallel
> vector, rest get none
>
> **************************************************************************************************************
>
> On Tue, Apr 24, 2012 at 3:41 PM, <petsc-users-request at mcs.anl.gov> wrote:
>
>> Send petsc-users mailing list submissions to
>>        petsc-users at mcs.anl.gov
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>        https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>> or, via email, send a message with subject or body 'help' to
>>        petsc-users-request at mcs.anl.gov
>>
>> You can reach the person managing the list at
>>        petsc-users-owner at mcs.anl.gov
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of petsc-users digest..."
>>
>>
>> Today's Topics:
>>
>>   1.  mumps get stuck with parmetis (Wen Jiang)
>>   2. Re:  mumps get stuck with parmetis (Hong Zhang)
>>   3. Re:  mumps get stuck with parmetis ( Alexander Grayver )
>>   4. Re:  mumps get stuck with parmetis (Aron Ahmadia)
>>   5. Re:  mumps get stuck with parmetis ( Alexander Grayver )
>>   6. Re:  mumps get stuck with parmetis (Aron Ahmadia)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 24 Apr 2012 13:43:06 -0400
>> From: Wen Jiang <jiangwen84 at gmail.com>
>>
>> Subject: [petsc-users] mumps get stuck with parmetis
>> To: petsc-users at mcs.anl.gov
>> Message-ID:
>>        <
>> CAMJxm+Da_byqMzm-_j+hRBUJSBt25oQAfDmUnVqHgBS4VPy8_A at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> Hi,
>>
>> My code will hang at the solving stage when I use mumps with the runtime
>> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
>> options, my code works fine. I am using PETSc 3.2 and configure it
>> with --download-mumps=1
>> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>>
>> Regards,
>> Wen
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/edd22bfe/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 24 Apr 2012 13:29:32 -0500
>> From: Hong Zhang <hzhang at mcs.anl.gov>
>> Subject: Re: [petsc-users] mumps get stuck with parmetis
>>
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Message-ID:
>>        <
>> CAGCphBurn0+f--gOZ_4MSrLcntvYSVCQ2_cmSG93nvv3hHSEjA at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> Wen :
>> I cannot repeat your error with petsc-dev. Running
>> petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c:
>> mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps
>> -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2
>> Norm of error 1.67272e-15 iterations 1
>>
>> Can you run above with your petsc-3.2 installation?
>>
>> Hong
>>
>> Hi,
>> >
>> > My code will hang at the solving stage when I use mumps with the runtime
>> > option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these
>> two
>> > options, my code works fine. I am using PETSc 3.2 and configure it with
>> --download-mumps=1
>> > and --download-parmetis=1. Could anyone give me any hints? Thanks.
>> >
>> > Regards,
>> > Wen
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/196ff3c5/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 24 Apr 2012 21:01:45 +0200
>> From: " Alexander Grayver " <agrayver at gfz-potsdam.de>
>> Subject: Re: [petsc-users] mumps get stuck with parmetis
>>
>> To: " PETSc users list " <petsc-users at mcs.anl.gov>
>> Message-ID: <auto-000008199816 at cgp2.gfz-potsdam.de>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Can you set:
>>
>> -mat_mumps_icntl_4 1
>>
>> And send mumps output?
>> Also do you use lu or ilu? How large is your matrix?
>>
>> Regards,
>> Alexander
>>
>> ----- Reply message -----
>> From: "Wen Jiang" <jiangwen84 at gmail.com>
>> To: <petsc-users at mcs.anl.gov>
>> Subject: [petsc-users] mumps get stuck with parmetis
>> Date: Tue, Apr 24, 2012 19:43
>> Hi,
>>
>> My code will hang at the solving stage when I use mumps with the runtime
>> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
>> options, my code works fine. I am using PETSc 3.2 and configure it with
>> --download-mumps=1 and --download-parmetis=1. Could anyone give me any
>> hints? Thanks.
>>
>>
>> Regards,
>> Wen
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2eb1767b/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Tue, 24 Apr 2012 22:13:59 +0300
>> From: Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
>> Subject: Re: [petsc-users] mumps get stuck with parmetis
>>
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Message-ID:
>>        <CADVoUZQDYqOpoZtxWY1DGE=
>> WGgbZvxMVuRy8zRxYTtrSCMP3sg at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
>> BlueGene/P
>> was causing similar behavior without even setting any options.  The only
>> way I was able to get a direct solver going was by switching over to
>> SuperLU.
>>
>> A
>>
>> On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
>> agrayver at gfz-potsdam.de
>> > wrote:
>>
>> > Can you set:
>> >
>> > -mat_mumps_icntl_4 1
>> >
>> > And send mumps output?
>> > Also do you use lu or ilu? How large is your matrix?
>> >
>> > Regards,
>> > Alexander
>> >
>> >
>> > ----- Reply message -----
>> > From: "Wen Jiang" <jiangwen84 at gmail.com>
>> > To: <petsc-users at mcs.anl.gov>
>> > Subject: [petsc-users] mumps get stuck with parmetis
>> > Date: Tue, Apr 24, 2012 19:43
>> >
>> >
>> > Hi,
>> >
>> > My code will hang at the solving stage when I use mumps with the runtime
>> > option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these
>> two
>> > options, my code works fine. I am using PETSc 3.2 and configure it with
>> --download-mumps=1
>> > and --download-parmetis=1. Could anyone give me any hints? Thanks.
>> >
>> > Regards,
>> > Wen
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/b0503b1c/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Tue, 24 Apr 2012 21:22:51 +0200
>> From: " Alexander Grayver " <agrayver at gfz-potsdam.de>
>> Subject: Re: [petsc-users] mumps get stuck with parmetis
>>
>> To: " PETSc users list " <petsc-users at mcs.anl.gov>
>> Message-ID: <auto-000056660896 at cgp1.gfz-potsdam.de>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Aron,
>>
>> This parameter let's to see mumps output in console. The important this
>> to understand where mumps hangs, during analysis, factorization or actual
>> solution (substitutions)? I'm almost sure it's factorization step. I
>> observe this pretty often with mumps compiled with petsc (whereas when
>> mumps is used directly it's quite rare to come along with this problem).
>>
>> Regards,
>> Alexander
>>
>> ----- Reply message -----
>> From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Subject: [petsc-users] mumps get stuck with parmetis
>> Date: Tue, Apr 24, 2012 21:13
>> I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
>> BlueGene/P was causing similar behavior without even setting any options.
>> ?The only way I was able to get a direct solver going was by switching over
>> to SuperLU.
>>
>> A
>>
>> On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
>> agrayver at gfz-potsdam.de> wrote:
>>
>> Can you set:
>>
>> -mat_mumps_icntl_4 1
>>
>> And send mumps output?
>> Also do you use lu or ilu? How large is your matrix?
>>
>>
>> Regards,
>> Alexander
>>
>> ----- Reply message -----
>> From: "Wen Jiang" <jiangwen84 at gmail.com>
>>
>> To: <petsc-users at mcs.anl.gov>
>> Subject: [petsc-users] mumps get stuck with parmetis
>> Date: Tue, Apr 24, 2012 19:43
>>
>>
>> Hi,
>>
>> My code will hang at the solving stage when I use mumps with the runtime
>> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these two
>> options, my code works fine. I am using PETSc 3.2 and configure it with
>> --download-mumps=1 and --download-parmetis=1. Could anyone give me any
>> hints? Thanks.
>>
>>
>>
>> Regards,
>> Wen
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/2db311da/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Tue, 24 Apr 2012 22:41:00 +0300
>> From: Aron Ahmadia <aron.ahmadia at kaust.edu.sa>
>> Subject: Re: [petsc-users] mumps get stuck with parmetis
>>
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Message-ID:
>>        <
>> CADVoUZSL+UXMJ0V52ELpE2vLLXc6WbZe96+yL5tYPsyMYiLXeQ at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> /project/k121/sandbox/petsc
>> -dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_part9.F:4666
>> /project/k121/sandbox/petsc
>> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part5.F:465
>> /project/k121/sandbox/petsc
>> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part1.F:409
>> /project/k121/sandbox/petsc
>> -dev/externalpackages/MUMPS_4.10.0-p3/src/dmumps_part3.F:6651
>> /project/k121/sandbox/petsc
>>
>> -dev/externalpackages/MUMPS_4.10.0-p3/src/mumps_c.c:422
>>
>> I don't know the MUMPS source code very well so I couldn't tell you what
>> this set of routines are doing, but this is a snippet of the stack trace I
>> was seeing when the jobs died on BG/P.
>>
>> If you set the "-info" flag on a PETSc run, it sends a lot of debugging
>> output to the screen, which is useful when you're in a situation where it
>> is hard to get access to a debugger or the stack trace.
>>
>> A
>>
>> On Tue, Apr 24, 2012 at 10:22 PM, Alexander Grayver <
>> agrayver at gfz-potsdam.de
>> > wrote:
>>
>> > Aron,
>> >
>> > This parameter let's to see mumps output in console. The important this
>> to
>> > understand where mumps hangs, during analysis, factorization or actual
>> > solution (substitutions)? I'm almost sure it's factorization step. I
>> > observe this pretty often with mumps compiled with petsc (whereas when
>> > mumps is used directly it's quite rare to come along with this problem).
>> >
>> > Regards,
>> > Alexander
>> >
>> > ----- Reply message -----
>> > From: "Aron Ahmadia" <aron.ahmadia at kaust.edu.sa>
>> > To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> > Subject: [petsc-users] mumps get stuck with parmetis
>> > Date: Tue, Apr 24, 2012 21:13
>> >
>> >
>> > I'm not sure if this is related, but Parmetis+Mumps+PETSc 3.2 on
>> > BlueGene/P was causing similar behavior without even setting any
>> options.
>> >  The only way I was able to get a direct solver going was by switching
>> over
>> > to SuperLU.
>> >
>> > A
>> >
>> > On Tue, Apr 24, 2012 at 10:01 PM, Alexander Grayver <
>> > agrayver at gfz-potsdam.de> wrote:
>> >
>> >> Can you set:
>> >>
>> >> -mat_mumps_icntl_4 1
>> >>
>> >> And send mumps output?
>> >> Also do you use lu or ilu? How large is your matrix?
>> >>
>> >> Regards,
>> >> Alexander
>> >>
>> >>
>> >> ----- Reply message -----
>> >> From: "Wen Jiang" <jiangwen84 at gmail.com>
>> >> To: <petsc-users at mcs.anl.gov>
>> >> Subject: [petsc-users] mumps get stuck with parmetis
>> >> Date: Tue, Apr 24, 2012 19:43
>> >>
>> >>
>> >> Hi,
>> >>
>> >> My code will hang at the solving stage when I use mumps with the
>> runtime
>> >> option -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2. If I remove these
>> two
>> >> options, my code works fine. I am using PETSc 3.2 and configure it
>> with --download-mumps=1
>> >> and --download-parmetis=1. Could anyone give me any hints? Thanks.
>> >>
>> >> Regards,
>> >> Wen
>> >>
>> >
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/3efe54e8/attachment.htm
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> petsc-users mailing list
>> petsc-users at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>>
>>
>> End of petsc-users Digest, Vol 40, Issue 76
>> *******************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/3db0ea6c/attachment-0001.htm>

From hzhang at mcs.anl.gov  Tue Apr 24 18:25:14 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 24 Apr 2012 18:25:14 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
Message-ID: <CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>

See
http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:

You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix you
create directly (not using DMCreateMatrix()) before calling MatSetValues(),
MatSetValuesBlocked() etc.


On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi,
>
> While trying to figure out a problem, I came across the following
> situation. Consider the following code:
>
> int main (int argc, char **argv){
>
>     PetscInitialize(&argc, &argv, (char*)0, help);
>
>
>     Mat m;
>
>     MatCreate(PETSC_COMM_WORLD, &m);
>
>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>
>     MatSetFromOptions(m);
>
>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>
>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>
>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>
>     MatDestroy(&m);
>
>
>     PetscFinalize();
>
>     return 0;
>
> }
>
>
> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
> argument 1 "mat" before MatAssemblyBegin()!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
>
> Eventually I could fix this by adding MatSetUp(m) after setting the
> options. Why do I need this in petsc-dev? Does this somehow preallocate the
> matrix?
>
Yes.

Hong

>
> Thanks,
> Mohammad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/5fe51ea2/attachment.htm>

From mirzadeh at gmail.com  Tue Apr 24 18:27:25 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 24 Apr 2012 16:27:25 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
Message-ID: <CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>

Thanks Hong.

On Tue, Apr 24, 2012 at 4:25 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:

> See
> http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:
>
> You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix you
> create directly (not using DMCreateMatrix()) before calling MatSetValues(),
> MatSetValuesBlocked() etc.
>
>
> On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Hi,
>>
>> While trying to figure out a problem, I came across the following
>> situation. Consider the following code:
>>
>> int main (int argc, char **argv){
>>
>>     PetscInitialize(&argc, &argv, (char*)0, help);
>>
>>
>>     Mat m;
>>
>>     MatCreate(PETSC_COMM_WORLD, &m);
>>
>>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>>
>>     MatSetFromOptions(m);
>>
>>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>>
>>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>>
>>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>>
>>     MatDestroy(&m);
>>
>>
>>     PetscFinalize();
>>
>>     return 0;
>>
>> }
>>
>>
>> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Object is in wrong state!
>> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
>> argument 1 "mat" before MatAssemblyBegin()!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>>
>> Eventually I could fix this by adding MatSetUp(m) after setting the
>> options. Why do I need this in petsc-dev? Does this somehow preallocate the
>> matrix?
>>
> Yes.
>
> Hong
>
>>
>> Thanks,
>> Mohammad
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/9a1919e2/attachment.htm>

From mirzadeh at gmail.com  Tue Apr 24 18:53:51 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 24 Apr 2012 16:53:51 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
Message-ID: <CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>

Hong,

Is this also true with MatSetType()? petsc-dev seems to need it (3.2-p6
does not) but it's not listed in the changes link ...

On Tue, Apr 24, 2012 at 4:27 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Thanks Hong.
>
>
> On Tue, Apr 24, 2012 at 4:25 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>
>> See
>> http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:
>>
>> You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix
>> you create directly (not using DMCreateMatrix()) before calling
>> MatSetValues(), MatSetValuesBlocked() etc.
>>
>>
>> On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Hi,
>>>
>>> While trying to figure out a problem, I came across the following
>>> situation. Consider the following code:
>>>
>>> int main (int argc, char **argv){
>>>
>>>     PetscInitialize(&argc, &argv, (char*)0, help);
>>>
>>>
>>>     Mat m;
>>>
>>>     MatCreate(PETSC_COMM_WORLD, &m);
>>>
>>>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>>>
>>>     MatSetFromOptions(m);
>>>
>>>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>>>
>>>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>>>
>>>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>>>
>>>     MatDestroy(&m);
>>>
>>>
>>>     PetscFinalize();
>>>
>>>     return 0;
>>>
>>> }
>>>
>>>
>>> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Object is in wrong state!
>>> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
>>> argument 1 "mat" before MatAssemblyBegin()!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>>
>>> Eventually I could fix this by adding MatSetUp(m) after setting the
>>> options. Why do I need this in petsc-dev? Does this somehow preallocate the
>>> matrix?
>>>
>> Yes.
>>
>> Hong
>>
>>>
>>> Thanks,
>>> Mohammad
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/72daf2ac/attachment.htm>

From hzhang at mcs.anl.gov  Tue Apr 24 19:58:02 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 24 Apr 2012 19:58:02 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
Message-ID: <CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>

Mohammad :

>
> Is this also true with MatSetType()? petsc-dev seems to need it (3.2-p6
> does not) but it's not listed in the changes link ...
>
There is no change on  MatSetType(). The default type is aij.
You can check petsc examples,
e.g. petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c

Hong

>
>
> On Tue, Apr 24, 2012 at 4:27 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Thanks Hong.
>>
>>
>> On Tue, Apr 24, 2012 at 4:25 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>>
>>> See
>>> http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:
>>>
>>> You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix
>>> you create directly (not using DMCreateMatrix()) before calling
>>> MatSetValues(), MatSetValuesBlocked() etc.
>>>
>>>
>>> On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> While trying to figure out a problem, I came across the following
>>>> situation. Consider the following code:
>>>>
>>>> int main (int argc, char **argv){
>>>>
>>>>     PetscInitialize(&argc, &argv, (char*)0, help);
>>>>
>>>>
>>>>     Mat m;
>>>>
>>>>     MatCreate(PETSC_COMM_WORLD, &m);
>>>>
>>>>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>>>>
>>>>     MatSetFromOptions(m);
>>>>
>>>>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>>>>
>>>>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>>>>
>>>>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>>>>
>>>>     MatDestroy(&m);
>>>>
>>>>
>>>>     PetscFinalize();
>>>>
>>>>     return 0;
>>>>
>>>> }
>>>>
>>>>
>>>> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [0]PETSC ERROR: Object is in wrong state!
>>>> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
>>>> argument 1 "mat" before MatAssemblyBegin()!
>>>> [0]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>>
>>>> Eventually I could fix this by adding MatSetUp(m) after setting the
>>>> options. Why do I need this in petsc-dev? Does this somehow preallocate the
>>>> matrix?
>>>>
>>> Yes.
>>>
>>> Hong
>>>
>>>>
>>>> Thanks,
>>>> Mohammad
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/90844ed2/attachment-0001.htm>

From mirzadeh at gmail.com  Wed Apr 25 00:53:44 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 24 Apr 2012 22:53:44 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
Message-ID: <CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>

When I do not explicitly set the type in petsc-dev I get errors:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Mat object's type is not set: Argument # 1!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 22:49:38 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatMPIAIJSetPreallocation() line 3883 in
/home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
argument 1 "mat" before MatGetOwnershipRange()!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 22:49:38 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatGetOwnershipRange() line 6106 in
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
argument 1 "mat" before MatAssemblyBegin()!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 22:49:38 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatAssemblyBegin() line 4810 in
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatAssemblyEnd_SeqAIJ line 800
/home/mohammad/soft/petsc-dev/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: [0] MatAssemblyEnd line 4984
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] MatAssemblyBegin line 4807
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] MatGetOwnershipRange line 6101
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] MatMPIAIJSetPreallocation line 3881
/home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 22:49:38 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

Even more than that, I get lots of errors apparently complaining that I
have not preallocated the matrix:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (6,209) caused a malloc!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
Tue Apr 24 22:44:09 2012
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
[0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
--with-cxx=g++ --with-fc=gfortran --download-mpich=1
--download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
--download-metis=1 --download-parmetis=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatSetValues_MPIAIJ() line 507 in
/home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: MatSetValues() line 1148 in
/home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c


I do not see these with 3.2-p6. Does 3.2-p6 has any mechanism for such
alerts?

On Tue, Apr 24, 2012 at 5:58 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:

> Mohammad :
>
>>
>> Is this also true with MatSetType()? petsc-dev seems to need it (3.2-p6
>> does not) but it's not listed in the changes link ...
>>
> There is no change on  MatSetType(). The default type is aij.
> You can check petsc examples,
> e.g. petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c
>
> Hong
>
>>
>>
>> On Tue, Apr 24, 2012 at 4:27 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Thanks Hong.
>>>
>>>
>>> On Tue, Apr 24, 2012 at 4:25 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>>>
>>>> See
>>>> http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:
>>>>
>>>> You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix
>>>> you create directly (not using DMCreateMatrix()) before calling
>>>> MatSetValues(), MatSetValuesBlocked() etc.
>>>>
>>>>
>>>> On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> While trying to figure out a problem, I came across the following
>>>>> situation. Consider the following code:
>>>>>
>>>>> int main (int argc, char **argv){
>>>>>
>>>>>     PetscInitialize(&argc, &argv, (char*)0, help);
>>>>>
>>>>>
>>>>>     Mat m;
>>>>>
>>>>>     MatCreate(PETSC_COMM_WORLD, &m);
>>>>>
>>>>>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>>>>>
>>>>>     MatSetFromOptions(m);
>>>>>
>>>>>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>>>>>
>>>>>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>>>>>
>>>>>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>>>>>
>>>>>     MatDestroy(&m);
>>>>>
>>>>>
>>>>>     PetscFinalize();
>>>>>
>>>>>     return 0;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>>>>>
>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>> [0]PETSC ERROR: Object is in wrong state!
>>>>> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
>>>>> argument 1 "mat" before MatAssemblyBegin()!
>>>>> [0]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> Eventually I could fix this by adding MatSetUp(m) after setting the
>>>>> options. Why do I need this in petsc-dev? Does this somehow preallocate the
>>>>> matrix?
>>>>>
>>>> Yes.
>>>>
>>>> Hong
>>>>
>>>>>
>>>>> Thanks,
>>>>> Mohammad
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120424/0fb392a6/attachment.htm>

From thomas.witkowski at tu-dresden.de  Wed Apr 25 02:21:08 2012
From: thomas.witkowski at tu-dresden.de (Thomas Witkowski)
Date: Wed, 25 Apr 2012 09:21:08 +0200
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
Message-ID: <4F97A5E4.3090404@tu-dresden.de>

Is it possible to create a parallel matrix (MPIAIJ) with empty size on 
some ranks? Even if this should work, does it make sense (with respect 
to runtime and efficiency) to collect all ranks having a nonempty 
contribution the the global matrix in an mpi group and to use this 
communicator for matrix create/computations?

Thomas

From paeanball at gmail.com  Wed Apr 25 04:52:57 2012
From: paeanball at gmail.com (Bao Kai)
Date: Wed, 25 Apr 2012 12:52:57 +0300
Subject: [petsc-users] On pure boundary conditions with SNES
Message-ID: <CAGCCrwHJUcppjdNip6rW3q_YrKcCM94bPgOBWpJgicUCHJiEwA@mail.gmail.com>

Dear All,

I am solving a problem with pure boundary conditions with SNES with
PETSc. I got the correct result with the following code, and it seems
that Petsc detected the singularity and managed to solve that ( by
adding another constraint like \sigma p_i=0?). However, it was much
slower compared to other simulations with well defined boundary
conditions.

I know how to tell the solver to remove the null space for linear
solver ksp.  I am wondering if somebody  can tell me how to tell the
snes solver to remove the null space so that the solution may be much
faster.

Thank you very much.

Best Regards,
Kai

       call SNESSetFunction(solv%snes, solv%resi, FormFunction, equ, ierr)

       call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr)
       call DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr)
       call MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr)
       call MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr)
       call MatFDColoringSetFromOptions(matfdcoloring,ierr)
       call SNESSetJacobian(solv%snes, solv%jac,
solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr)
       call ISColoringDestroy(iscoloring,ierr)

       call SNESSetFromOptions(solv%snes,ierr)
       call SNESSolve(solv%snes,PETSC_NULL_OBJECT, solv%soln,ierr)
       call DMGetLocalVector(solv%da, soln_work, ierr)

From dominik at itis.ethz.ch  Wed Apr 25 05:18:21 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 12:18:21 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
Message-ID: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>

I am successfully solving a couple (unstructured FEM) benchmark
problems with SNES, providing both FormFunction and FormJacobian. I am
using a simple (polynomial) form for the nonlinear part so I am fairly
sure I compute the Jacobian correctly and exactly. I am also getting
the expected results.

Now I want to reproduce the same results without providing the
Jacobian, preparing for future real life cases where it will not be
known. I am under the impression that Petsc can approximate the
Jacobian itself using some sort of secant method. How exactly am I
supposed to use it? If I change nothing in the code, just pass
-snes_mf_operator to the options, the solver stagnates. The same if I
fill the Jacobian with zeros. The same if only with A. What am I
supposed to do without the known Jacobian to trigger its
approximation?

Thanks a lot
Dominik

From jedbrown at mcs.anl.gov  Wed Apr 25 06:21:12 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 06:21:12 -0500
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
In-Reply-To: <4F97A5E4.3090404@tu-dresden.de>
References: <4F97A5E4.3090404@tu-dresden.de>
Message-ID: <CAM9tzSnJNwcB4s0kdtkj5Q0GC6Eif1+i9C_+Qwq42HeBwMW5SA@mail.gmail.com>

On Wed, Apr 25, 2012 at 02:21, Thomas Witkowski <
thomas.witkowski at tu-dresden.de> wrote:

> Is it possible to create a parallel matrix (MPIAIJ) with empty size on
> some ranks? Even if this should work, does it make sense (with respect to
> runtime and efficiency) to collect all ranks having a nonempty contribution
> the the global matrix in an mpi group and to use this communicator for
> matrix create/computations?


It depends how many ranks are empty. For coarse levels of multigrid (where
this situation arises frequently), it's sometimes worthwhile to also move
the data to a part of the machine that is topologically "close".
Identifying "closeness" is not always easy because MPI does not necessarily
expose this information in a useful fashion.

What is the circumstance where this occurs for you?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/187cb266/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 06:25:05 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 06:25:05 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
Message-ID: <CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>

On Wed, Apr 25, 2012 at 05:18, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I am successfully solving a couple (unstructured FEM) benchmark
> problems with SNES, providing both FormFunction and FormJacobian. I am
> using a simple (polynomial) form for the nonlinear part so I am fairly
> sure I compute the Jacobian correctly and exactly. I am also getting
> the expected results.
>
> Now I want to reproduce the same results without providing the
> Jacobian, preparing for future real life cases where it will not be
> known. I am under the impression that Petsc can approximate the
> Jacobian itself using some sort of secant method. How exactly am I
> supposed to use it? If I change nothing in the code, just pass
> -snes_mf_operator to the options, the solver stagnates. The same if I
> fill the Jacobian with zeros. The same if only with A. What am I
> supposed to do without the known Jacobian to trigger its
> approximation?
>

Run with -ksp_monitor_true_residual -ksp_converged_reason -snes_monitor
-snes_converged_reason -snes_ls_monitor -snes_view and send the output.

Your SNES callback gets two Mat arguments, into which are you assembling
your approximate Jacobian? Are you still calling MatAssemblyBegin/End on
the first argument, as in the examples?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/72983ed2/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 06:28:42 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 06:28:42 -0500
Subject: [petsc-users] On pure boundary conditions with SNES
In-Reply-To: <CAGCCrwHJUcppjdNip6rW3q_YrKcCM94bPgOBWpJgicUCHJiEwA@mail.gmail.com>
References: <CAGCCrwHJUcppjdNip6rW3q_YrKcCM94bPgOBWpJgicUCHJiEwA@mail.gmail.com>
Message-ID: <CAM9tzSkeaVsWwfLv77ifzBdjLKfwn6eAbxQ5xkMs0VgSYBpfYA@mail.gmail.com>

On Wed, Apr 25, 2012 at 04:52, Bao Kai <paeanball at gmail.com> wrote:

> Dear All,
>
> I am solving a problem with pure boundary conditions with SNES with
> PETSc. I got the correct result with the following code, and it seems
> that Petsc detected the singularity and managed to solve that ( by
> adding another constraint like \sigma p_i=0?). However, it was much
> slower compared to other simulations with well defined boundary
> conditions.
>
> I know how to tell the solver to remove the null space for linear
> solver ksp.  I am wondering if somebody  can tell me how to tell the
> snes solver to remove the null space so that the solution may be much
> faster.
>

If it's just the constant, you can use -ksp_constant_null_space, but if it
converged (did the residual actually go to zero?), the problem might have
already been consistent. To set the null space more directly, use
SNESGetKSP, then KSPSetNullSpace; with petsc-dev, you can just call
MatSetNullSpace() on the operator (and preconditioning matrix, if
applicable).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/671eeebe/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 06:33:20 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 06:33:20 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
Message-ID: <CAM9tzSmYTHjxO5BtcQQJOKkpKaOHxSXdEp8gxMHhiP3of1hFkw@mail.gmail.com>

On Wed, Apr 25, 2012 at 00:53, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:

> Even more than that, I get lots of errors apparently complaining that I
> have not preallocated the matrix:
>

You are not checking error codes, otherwise you wouldn't see all the
subsequent errors.


> I do not see these with 3.2-p6. Does 3.2-p6 has any mechanism for such
> alerts?
>

The default behavior was changed in petsc-dev because forgetting to
preallocate was too common of a mistake.
http://www.mcs.anl.gov/petsc/documentation/changes/dev.html

* Preallocation routines now automatically set
MAT_NEW_NONZERO_ALLOCATION_ERR, if you intentionally preallocate less than
necessary then use
MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) to disable the
error generation.
* You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix
you create directly (not using DMCreateMatrix()) before calling
MatSetValues(), MatSetValuesBlocked() etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/cf557b1e/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 06:34:11 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 06:34:11 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
Message-ID: <CAM9tzSm90bhGZ0hgFJwGSO5CudZy6yc=5k9DmzWHotP=3kWoKg@mail.gmail.com>

On Wed, Apr 25, 2012 at 06:25, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Run with -ksp_monitor_true_residual -ksp_converged_reason -snes_monitor
> -snes_converged_reason -snes_ls_monitor -snes_view and send the output.
>
> Your SNES callback gets two Mat arguments, into which are you assembling
> your approximate Jacobian? Are you still calling MatAssemblyBegin/End on
> the first argument, as in the examples?
>

There are plenty of additional suggestions here

http://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/b22494f1/attachment-0001.htm>

From knepley at gmail.com  Wed Apr 25 06:38:11 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 07:38:11 -0400
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
In-Reply-To: <4F97A5E4.3090404@tu-dresden.de>
References: <4F97A5E4.3090404@tu-dresden.de>
Message-ID: <CAMYG4GkC-KkuFUhYUOYAnYyTJLmiKs+_karwdz2v1Zg_nAf_=g@mail.gmail.com>

On Wed, Apr 25, 2012 at 3:21 AM, Thomas Witkowski <
thomas.witkowski at tu-dresden.de> wrote:

> Is it possible to create a parallel matrix (MPIAIJ) with empty size on
> some ranks? Even if this should work, does it make sense (with respect to
> runtime and efficiency) to collect all ranks having a nonempty contribution
> the the global matrix in an mpi group and to use this communicator for
> matrix create/computations?


1) Yes

2) It depends. If there are a bunch of empty procs and you do reductions,
it may make sense to leave them out. But
    you would have to be on a large number of processes.

     Matt


>
> Thomas
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/6ee0760f/attachment-0001.htm>

From thomas.witkowski at tu-dresden.de  Wed Apr 25 06:42:25 2012
From: thomas.witkowski at tu-dresden.de (Thomas Witkowski)
Date: Wed, 25 Apr 2012 13:42:25 +0200
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
In-Reply-To: <CAM9tzSnJNwcB4s0kdtkj5Q0GC6Eif1+i9C_+Qwq42HeBwMW5SA@mail.gmail.com>
References: <4F97A5E4.3090404@tu-dresden.de>
	<CAM9tzSnJNwcB4s0kdtkj5Q0GC6Eif1+i9C_+Qwq42HeBwMW5SA@mail.gmail.com>
Message-ID: <4F97E321.2070400@tu-dresden.de>

Am 25.04.2012 13:21, schrieb Jed Brown:
> On Wed, Apr 25, 2012 at 02:21, Thomas Witkowski 
> <thomas.witkowski at tu-dresden.de 
> <mailto:thomas.witkowski at tu-dresden.de>> wrote:
>
>     Is it possible to create a parallel matrix (MPIAIJ) with empty
>     size on some ranks? Even if this should work, does it make sense
>     (with respect to runtime and efficiency) to collect all ranks
>     having a nonempty contribution the the global matrix in an mpi
>     group and to use this communicator for matrix create/computations?
>
>
> It depends how many ranks are empty. For coarse levels of multigrid 
> (where this situation arises frequently), it's sometimes worthwhile to 
> also move the data to a part of the machine that is topologically 
> "close". Identifying "closeness" is not always easy because MPI does 
> not necessarily expose this information in a useful fashion.
>
> What is the circumstance where this occurs for you?
Also some coarse space method. When not all ranks contribute to the 
coarse space, a matrix defined on the coarse space leads to local empty 
matrices on these ranks. So the question is if this leads to technical 
troubles in PETSc. The share of nodes participating in the coarse space 
may vary.

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/4a5fd0e5/attachment.htm>

From knepley at gmail.com  Wed Apr 25 06:51:33 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 07:51:33 -0400
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
In-Reply-To: <4F97E321.2070400@tu-dresden.de>
References: <4F97A5E4.3090404@tu-dresden.de>
	<CAM9tzSnJNwcB4s0kdtkj5Q0GC6Eif1+i9C_+Qwq42HeBwMW5SA@mail.gmail.com>
	<4F97E321.2070400@tu-dresden.de>
Message-ID: <CAMYG4G=sxTrXFtT1QjL4wFhXVBMmRHvxepAXcQHc9Qv47NMORw@mail.gmail.com>

On Wed, Apr 25, 2012 at 7:42 AM, Thomas Witkowski <
thomas.witkowski at tu-dresden.de> wrote:

>  Am 25.04.2012 13:21, schrieb Jed Brown:
>
> On Wed, Apr 25, 2012 at 02:21, Thomas Witkowski <
> thomas.witkowski at tu-dresden.de> wrote:
>
>> Is it possible to create a parallel matrix (MPIAIJ) with empty size on
>> some ranks? Even if this should work, does it make sense (with respect to
>> runtime and efficiency) to collect all ranks having a nonempty contribution
>> the the global matrix in an mpi group and to use this communicator for
>> matrix create/computations?
>
>
> It depends how many ranks are empty. For coarse levels of multigrid (where
> this situation arises frequently), it's sometimes worthwhile to also move
> the data to a part of the machine that is topologically "close".
> Identifying "closeness" is not always easy because MPI does not necessarily
> expose this information in a useful fashion.
>
>  What is the circumstance where this occurs for you?
>
> Also some coarse space method. When not all ranks contribute to the coarse
> space, a matrix defined on the coarse space leads to local empty matrices
> on these ranks. So the question is if this leads to technical troubles in
> PETSc. The share of nodes participating in the coarse space may vary.
>

No

  Matt


>
> Thomas
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/ef392618/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 07:01:24 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 07:01:24 -0500
Subject: [petsc-users] MatCreateMPIAIJ with zero size on some ranks
In-Reply-To: <4F97E321.2070400@tu-dresden.de>
References: <4F97A5E4.3090404@tu-dresden.de>
	<CAM9tzSnJNwcB4s0kdtkj5Q0GC6Eif1+i9C_+Qwq42HeBwMW5SA@mail.gmail.com>
	<4F97E321.2070400@tu-dresden.de>
Message-ID: <CAM9tzSkmL4XePrTY+mGw-zwdNhqatpJUTrHU_q_=c-W+OqanSw@mail.gmail.com>

On Wed, Apr 25, 2012 at 06:42, Thomas Witkowski <
thomas.witkowski at tu-dresden.de> wrote:

> Also some coarse space method. When not all ranks contribute to the coarse
> space, a matrix defined on the coarse space leads to local empty matrices
> on these ranks. So the question is if this leads to technical troubles in
> PETSc. The share of nodes participating in the coarse space may vary.


There is no "technical trouble".

Communicator contraction doesn't really affect the MatMult. Depending on
the hardware, it may affect the VecDot and VecNorm (contracting the
communicator is likely worse on Blue Gene/P because certain reduction
algorithms become unavailable---unless the contraction factor is massive
and the remaining participants are nearby on the torus, it's probably good
on many fat tree networks).

Redistributing the data for some locality (independent of whether you also
contract the communicator) is more useful for MatMult, but you have to
weight the benefits relative to the cost of moving the data, and it may not
pay off.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/9b44257b/attachment.htm>

From zonexo at gmail.com  Wed Apr 25 07:07:14 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 25 Apr 2012 14:07:14 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4Gk-fXEXyj5L__QMgnGc4f9fuoEEj4W_Cm4Nmcdy72TV0A@mail.gmail.com>
References: <4F970DCD.1060302@gmail.com>
	<CAMYG4GkhVPK3QAA3vEuiWrH3hO8ZjFVFvT6Sx9C9iWsY22oUDw@mail.gmail.com>
	<4F97139F.2030205@gmail.com>
	<CAMYG4Gk-fXEXyj5L__QMgnGc4f9fuoEEj4W_Cm4Nmcdy72TV0A@mail.gmail.com>
Message-ID: <4F97E8F2.5050803@gmail.com>

Hi,

There's no error now but I didn't get the same ans as ex29. I believe 
the error lies in the boundary evaluation of v.

In c, the code is :

else if (user->bcType == NEUMANN) {
           num = 0;
           if (j!=0) {
             v[num] = -rho*HxdHy;              col[num].i = i;   
col[num].j = j-1;
             num++;
           }
           if (i!=0) {
             v[num] = -rho*HydHx;              col[num].i = i-1; 
col[num].j = j;
             num++;
           }
           if (i!=mx-1) {
             v[num] = -rho*HydHx;              col[num].i = i+1; 
col[num].j = j;
             num++;
           }
           if (j!=my-1) {
             v[num] = -rho*HxdHy;              col[num].i = i;   
col[num].j = j+1;
             num++;
           }
           v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;   
col[num].j = j;
           num++;
           ierr = 
MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);

In fortran, I have changed to :

else if (bc_cond == 2) then

                 num = 1

                 if (j/=0) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,1) = i

                     col(MatStencil_j,2) = j-1

                     num = num + 1

                 end if

                 if (i/=0) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,1) = i-1

                     col(MatStencil_j,2) = j

                     num = num + 1

                 end if

                 if (i/=mx-1) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,1) = i+1

                     col(MatStencil_j,2) = j

                     num = num + 1

                 end if

                 if (j/=my-1) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,1) = i

                     col(MatStencil_j,2) = j+1

                     num = num + 1

                 end if

                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)

                 col(MatStencil_i,1) = i

                 col(MatStencil_j,2) = j+1

                 num = num + 1

                 call 
MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)

However, I got the error:

[0]PETSC ERROR: --------------------- Error Message 
---------------------------
--------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
[0]PETSC ERROR: 
---------------------------------------------------------------

How such is be defined in Fortran?

Thank you

Yours sincerely,

TAY wee-beng


On 25/4/2012 12:06 AM, Matthew Knepley wrote:
> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I still got the same error the moment this subroutine is called,
>     after changing to :
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>
>
> Argument 3 should be 0.
>
>   Matt
>
>     [0]PETSC ERROR: Null argument, when expecting valid pointer!
>     [0]PETSC ERROR: Null Object: Parameter # 4!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:54:50 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>     src/mat/interface/C:\temp\PETSC
>     -~1\src\mat\INTERF~1\matnull.c
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>     On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to
>>         fortran version. The fortran version is based on the template
>>         of ex22f, which is in 3d.
>>
>>         I have attached the code below. I have 2 problems.
>>
>>         1. When i3 = -3, in dirichlet BC, I got the same answer as
>>         ex29 in c version.
>>
>>         However, when I change i3 to -4, I got the following error:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Argument out of range!
>>         [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:08:46 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>         src/mat/impls/aij/seq/C:\temp\
>>         PETSC-~1\src\mat\impls\aij\seq\aij.c
>>         [0]PETSC ERROR: MatSetValues() line 1148 in
>>         src/mat/interface/C:\temp\PETSC-~1\s
>>         rc\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>         src/mat/interface/C:\temp\PET
>>         SC-~1\src\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Argument out of range!
>>         [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>
>>         ...
>>
>>         What did I do wrong?
>>
>>
>>         2. When I wanted to use the neumann BC, following ex29, I added :
>>
>>         In subroutine ComputeRHS
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>         call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         and in subroutine ComputeMatrix
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         call MatSetNullSpace(jac,nullspace,ierr)
>>
>>         call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         When I compile, it says unresolved external symbol
>>         MatNullSpaceRemove. Removing MatNullSpaceRemove, I can
>>         compile but running gives the error:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Corrupt argument:
>>         see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>         [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Invalid argument!
>>         [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>         src/mat/interface/C:\temp\PETS
>>         C-~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: ourdmfunction() line 30 in
>>         src/dm/interface/ftn-custom/C:\temp\P
>>         ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>         [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>         src/dm/interface/C:\temp\PETSC-
>>         ~1\src\dm\INTERF~1\dm.c
>>         [0]PETSC ERROR: KSPSetUp() line 218 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Corrupt argument:
>>         see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>         [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Invalid argument!
>>         [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>         src/mat/interface/C:\temp\PETS
>>         C-~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: ourdmfunction() line 30 in
>>         src/dm/interface/ftn-custom/C:\temp\P
>>         ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>         [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>         src/dm/interface/C:\temp\PETSC-
>>         ~1\src\dm\INTERF~1\dm.c
>>         [0]PETSC ERROR: KSPSetUp() line 218 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         [0]PETSC ERROR: KSPSolve() line 402 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         Vector Object:Vec_0000000084000000_0 1 MPI processes
>>           type: mpi
>>         Process [0]
>>
>>         What's wrong?
>>
>>
>>         *program ex29f*
>>
>>         implicit none
>>
>>         ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>         - - - - - -
>>         !                    Include files
>>         ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>         - - - - - -
>>         !
>>         !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>         !     petscmat.h - matrices
>>         !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>>         preconditioners
>>
>>         #include "finclude/petsc.h90"
>>
>>         PetscErrorCode   ierr
>>
>>         DM               da
>>
>>         KSP              ksp
>>
>>         Vec              x
>>
>>         external         ComputeRHS,ComputeMatrix
>>
>>         PetscInt i1,i3
>>
>>         call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>
>>         i3 = -4
>>
>>         i1 = 1
>>
>>         call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>
>>         call
>>         DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>         call DMSetFunction(da,ComputeRHS,ierr)
>>         call DMSetJacobian(da,ComputeMatrix,ierr)
>>         call KSPSetDM(ksp,da,ierr)
>>
>>         call KSPSetFromOptions(ksp,ierr)
>>         call KSPSetUp(ksp,ierr)
>>         call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>         call KSPGetSolution(ksp,x,ierr)
>>         call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>         call KSPDestroy(ksp,ierr)
>>         call DMDestroy(da,ierr)
>>         call PetscFinalize(ierr)
>>
>>         end program ex29f
>>
>>         subroutine ComputeRHS(da,x,b,ierr)
>>
>>         implicit none
>>
>>         #include "finclude/petsc.h90"
>>
>>         PetscErrorCode  ierr
>>         PetscInt i,j,mx,my,xm,ym,xs,ys
>>         PetscScalar  h,nu,rho
>>         PetscScalar Hx,Hy
>>         PetscScalar,pointer :: array(:,:)
>>         Vec          x,b
>>         DM           da
>>         MatNullSpace nullspace    !>neumann BC
>>
>>         nu = 0.1
>>
>>         call
>>         DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>>         Hx = 1.d0 / (mx-1)
>>
>>         Hy = 1.d0 / (my-1)
>>
>>         call
>>         DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>         call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>>         do j = ys,ys+ym-1
>>
>>             do i = xs,xs+xm-1
>>
>>                 array(i,j) =
>>         exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>             end do
>>
>>         end do
>>
>>         call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>>         call VecAssemblyBegin(b,ierr)
>>
>>         call VecAssemblyEnd(b,ierr)
>>
>>         !call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>
>>     4th argument should be PETSC_NULL_OBJECT.
>>
>>        Matt
>>
>>         !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>         !call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         end subroutine ComputeRHS
>>
>>
>>         subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>
>>         implicit none
>>
>>         #include "finclude/petsc.h90"
>>
>>         Mat          jac,JJ
>>
>>         PetscErrorCode    ierr
>>
>>         DM           da
>>
>>         PetscInt    i,j,k,mx,my,xm
>>
>>         PetscInt    ym,xs,ys,i1,i5
>>
>>         PetscScalar  v(5),Hx,Hy,rho,centerRho
>>
>>         PetscScalar  HydHx
>>
>>         PetscScalar  HxdHy
>>
>>         MatStencil   row(4),col(4,5)
>>
>>         Vec          x
>>
>>         MatStructure str
>>
>>         MatNullSpace nullspace    !>neumann BC
>>
>>         rho = 1.0
>>
>>         i1 = 1
>>
>>         i5 = 5
>>
>>         centerRho = rho
>>
>>         call
>>         DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>>         Hx = 1.d0 / (mx-1)
>>
>>         Hy = 1.d0 / (my-1)
>>
>>         HxdHy = Hx/Hy
>>
>>         HydHx = Hy/Hx
>>
>>         call
>>         DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>
>>         do j=ys,ys+ym-1
>>
>>             do i=xs,xs+xm-1
>>
>>                 row(MatStencil_i) = i
>>
>>                 row(MatStencil_j) = j
>>
>>                 call ComputeRho(i,j,mx,my,centerRho,rho)
>>
>>                 if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1)
>>         then
>>
>>                     v(1) = 2.0*rho*(HxdHy + HydHx)
>>
>>                     call
>>         MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>
>>                 else
>>
>>                     v(1) = -rho*HxdHy
>>
>>                     col(MatStencil_i,1) = i
>>
>>                     col(MatStencil_j,2) = j-1
>>
>>
>>     Cut and paste error above.
>>
>>       Matt
>>
>>                     v(2) = -rho*HydHx
>>
>>                     col(MatStencil_i,2) = i-1
>>
>>                     col(MatStencil_j,2) = j
>>
>>                     v(3) = 2.0*rho*(HxdHy + HydHx)
>>
>>                     col(MatStencil_i,3) = i
>>
>>                     col(MatStencil_j,3) = j
>>
>>                     v(4) = -rho*HydHx
>>
>>                     col(MatStencil_i,4) = i+1
>>
>>                     col(MatStencil_j,4) = j
>>
>>                     v(5) = -rho*HxdHy
>>
>>                     col(MatStencil_i,5) = i
>>
>>                     col(MatStencil_j,5) = j+1
>>
>>                     call
>>         MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>
>>                 end if
>>
>>             end do
>>
>>         end do
>>
>>         call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>>         call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>>         !call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         !call MatSetNullSpace(jac,nullspace,ierr)
>>
>>         !call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         end subroutine ComputeMatrix
>>
>>         subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>
>>         PetscInt    i,j,mx,my
>>
>>         PetscScalar  rho,centerRho
>>
>>         if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0)
>>         .and. (j < 2.0*my/3.0)) then
>>
>>             rho = centerRho
>>
>>         else
>>
>>             rho = 1.0
>>
>>         end if
>>
>>
>>         end subroutine ComputeRho
>>
>>         -- 
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/49c3c2fd/attachment-0001.htm>

From hzhang at mcs.anl.gov  Wed Apr 25 09:47:56 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Wed, 25 Apr 2012 09:47:56 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
Message-ID: <CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>

Mohammad :

> When I do not explicitly set the type in petsc-dev I get errors:
>

Did you follow same procedural call as petsc examples, e.g.,
petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c:
   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
  ierr =
MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr);
  ierr = MatSeqAIJSetPreallocation(A,5,PETSC_NULL);CHKERRQ(ierr);
...
Hong

>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Mat object's type is not set: Argument # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
> Tue Apr 24 22:49:38 2012
> [0]PETSC ERROR: Libraries linked from
> /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
> [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich=1
> --download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
> --download-metis=1 --download-parmetis=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatMPIAIJSetPreallocation() line 3883 in
> /home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
> argument 1 "mat" before MatGetOwnershipRange()!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
> Tue Apr 24 22:49:38 2012
> [0]PETSC ERROR: Libraries linked from
> /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
> [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich=1
> --download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
> --download-metis=1 --download-parmetis=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatGetOwnershipRange() line 6106 in
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
> argument 1 "mat" before MatAssemblyBegin()!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
> Tue Apr 24 22:49:38 2012
> [0]PETSC ERROR: Libraries linked from
> /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
> [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich=1
> --download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
> --download-metis=1 --download-parmetis=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatAssemblyBegin() line 4810 in
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatAssemblyEnd_SeqAIJ line 800
> /home/mohammad/soft/petsc-dev/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: [0] MatAssemblyEnd line 4984
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] MatAssemblyBegin line 4807
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] MatGetOwnershipRange line 6101
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] MatMPIAIJSetPreallocation line 3881
> /home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
> Tue Apr 24 22:49:38 2012
> [0]PETSC ERROR: Libraries linked from
> /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
> [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich=1
> --download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
> --download-metis=1 --download-parmetis=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> Even more than that, I get lots of errors apparently complaining that I
> have not preallocated the matrix:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (6,209) caused a malloc!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 5c943f0d8cbf252873fd4abeffa38c8d3c15987e  HG Date: Mon Apr 09 22:04:11 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad
> Tue Apr 24 22:44:09 2012
> [0]PETSC ERROR: Libraries linked from
> /home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Mon Apr  9 23:17:27 2012
> [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-cc=gcc
> --with-cxx=g++ --with-fc=gfortran --download-mpich=1
> --download-f-blas-lapack=1 --download-hypre=1 --download-superlu_dist=1
> --download-metis=1 --download-parmetis=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 507 in
> /home/mohammad/soft/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c
>  [0]PETSC ERROR: MatSetValues() line 1148 in
> /home/mohammad/soft/petsc-dev/src/mat/interface/matrix.c
>
>
> I do not see these with 3.2-p6. Does 3.2-p6 has any mechanism for such
> alerts?
>
> On Tue, Apr 24, 2012 at 5:58 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>
>> Mohammad :
>>
>>>
>>> Is this also true with MatSetType()? petsc-dev seems to need it (3.2-p6
>>> does not) but it's not listed in the changes link ...
>>>
>> There is no change on  MatSetType(). The default type is aij.
>> You can check petsc examples,
>> e.g. petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c
>>
>> Hong
>>
>>>
>>>
>>> On Tue, Apr 24, 2012 at 4:27 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Thanks Hong.
>>>>
>>>>
>>>> On Tue, Apr 24, 2012 at 4:25 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>>>>
>>>>> See
>>>>> http://www.mcs.anl.gov/petsc/documentation/changes/dev.html:
>>>>>
>>>>> You MUST now call MatXXXSetPreallocation() or MatSetUp() on any matrix
>>>>> you create directly (not using DMCreateMatrix()) before calling
>>>>> MatSetValues(), MatSetValuesBlocked() etc.
>>>>>
>>>>>
>>>>> On Tue, Apr 24, 2012 at 6:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> While trying to figure out a problem, I came across the following
>>>>>> situation. Consider the following code:
>>>>>>
>>>>>> int main (int argc, char **argv){
>>>>>>
>>>>>>     PetscInitialize(&argc, &argv, (char*)0, help);
>>>>>>
>>>>>>
>>>>>>     Mat m;
>>>>>>
>>>>>>     MatCreate(PETSC_COMM_WORLD, &m);
>>>>>>
>>>>>>     MatSetSizes(m, 10, 10, PETSC_DECIDE, PETSC_DECIDE);
>>>>>>
>>>>>>     MatSetFromOptions(m);
>>>>>>
>>>>>>     MatAssemblyBegin(m, MAT_FINAL_ASSEMBLY);
>>>>>>
>>>>>>     MatAssemblyEnd(m, MAT_FINAL_ASSEMBLY);
>>>>>>
>>>>>>     MatView(m, PETSC_VIEWER_STDOUT_WORLD);
>>>>>>
>>>>>>     MatDestroy(&m);
>>>>>>
>>>>>>
>>>>>>     PetscFinalize();
>>>>>>
>>>>>>     return 0;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> This runs without any problem under 3.2-p6 but fails with petsc-dev:
>>>>>>
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>> ------------------------------------
>>>>>> [0]PETSC ERROR: Object is in wrong state!
>>>>>> [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
>>>>>> argument 1 "mat" before MatAssemblyBegin()!
>>>>>> [0]PETSC ERROR:
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> Eventually I could fix this by adding MatSetUp(m) after setting the
>>>>>> options. Why do I need this in petsc-dev? Does this somehow preallocate the
>>>>>> matrix?
>>>>>>
>>>>> Yes.
>>>>>
>>>>> Hong
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Mohammad
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/ba5ddf9c/attachment.htm>

From dominik at itis.ethz.ch  Wed Apr 25 12:23:31 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 19:23:31 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
Message-ID: <CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>

>> Now I want to reproduce the same results without providing the
>> Jacobian, preparing for future real life cases where it will not be
>> known. I am under the impression that Petsc can approximate the
>> Jacobian itself using some sort of secant method. How exactly am I
>> supposed to use it? If I change nothing in the code, just pass
>> -snes_mf_operator to the options, the solver stagnates. The same if I
>> fill the Jacobian with zeros. The same if only with A. What am I
>> supposed to do without the known Jacobian to trigger its
>> approximation?
>
>
> Run with -ksp_monitor_true_residual -ksp_converged_reason -snes_monitor
> -snes_converged_reason -snes_ls_monitor -snes_view and send the output.

I see the following stagnated output:

0 SNES Function norm 1.747536282885e+01
  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 6.112827145626e-01 true resid norm
1.751036369876e+01 ||r(i)||/||b|| 1.002002869425e+00
  2 KSP unpreconditioned resid norm 6.046520497681e-01 true resid norm
2.300357730584e+01 ||r(i)||/||b|| 1.316343330386e+00
  3 KSP unpreconditioned resid norm 5.994471388765e-01 true resid norm
5.655832090079e+01 ||r(i)||/||b|| 3.236460464638e+00
  4 KSP unpreconditioned resid norm 5.974054975332e-01 true resid norm
5.997203971117e+01 ||r(i)||/||b|| 3.431805124649e+00
  5 KSP unpreconditioned resid norm 5.956696306552e-01 true resid norm
9.280866979439e+01 ||r(i)||/||b|| 5.310829348914e+00
  6 KSP unpreconditioned resid norm 5.944323800377e-01 true resid norm
9.573083746155e+01 ||r(i)||/||b|| 5.478045772159e+00
  7 KSP unpreconditioned resid norm 5.933318368437e-01 true resid norm
1.283420019927e+02 ||r(i)||/||b|| 7.344168086790e+00
  8 KSP unpreconditioned resid norm 5.924396174526e-01 true resid norm
1.311457478032e+02 ||r(i)||/||b|| 7.504608006575e+00
  9 KSP unpreconditioned resid norm 5.916244555561e-01 true resid norm
1.636933991914e+02 ||r(i)||/||b|| 9.367095882044e+00
 10 KSP unpreconditioned resid norm 5.909266031948e-01 true resid norm
1.664685701658e+02 ||r(i)||/||b|| 9.525900652028e+00
 11 KSP unpreconditioned resid norm 5.902780068683e-01 true resid norm
1.989974529969e+02 ||r(i)||/||b|| 1.138731452650e+01
 12 KSP unpreconditioned resid norm 5.897052848025e-01 true resid norm
2.017662138832e+02 ||r(i)||/||b|| 1.154575248933e+01
 13 KSP unpreconditioned resid norm 5.891666923831e-01 true resid norm
2.342886435057e+02 ||r(i)||/||b|| 1.340679708915e+01

and so on, excessively long

> Your SNES callback gets two Mat arguments, into which are you assembling
> your approximate Jacobian?

I am assembling my (exact) Jacobian in the second (preconditioner) matrix.

> Are you still calling MatAssemblyBegin/End on the
> first argument, as in the examples?

Yes, I do.

The moment I remove -snes_mf_operator from the options I get the
expected results in the expected time.

Dominik

From knepley at gmail.com  Wed Apr 25 12:29:59 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 13:29:59 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
Message-ID: <CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>

On Wed, Apr 25, 2012 at 1:23 PM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> >> Now I want to reproduce the same results without providing the
> >> Jacobian, preparing for future real life cases where it will not be
> >> known. I am under the impression that Petsc can approximate the
> >> Jacobian itself using some sort of secant method. How exactly am I
> >> supposed to use it? If I change nothing in the code, just pass
> >> -snes_mf_operator to the options, the solver stagnates. The same if I
> >> fill the Jacobian with zeros. The same if only with A. What am I
> >> supposed to do without the known Jacobian to trigger its
> >> approximation?
> >
> >
> > Run with -ksp_monitor_true_residual -ksp_converged_reason -snes_monitor
> > -snes_converged_reason -snes_ls_monitor -snes_view and send the output.
>
> I see the following stagnated output:
>

Start by comparing residuals, with an without. Is SNES 0 the same? Is KSP 0
the same? KSP 1?
Notice that the true residual norm is monotonically increasing, leading me
to believe that the
preconditioner is rank deficient since GMRES should never allow increase.

    Matt


> 0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.112827145626e-01 true resid norm
> 1.751036369876e+01 ||r(i)||/||b|| 1.002002869425e+00
>  2 KSP unpreconditioned resid norm 6.046520497681e-01 true resid norm
> 2.300357730584e+01 ||r(i)||/||b|| 1.316343330386e+00
>  3 KSP unpreconditioned resid norm 5.994471388765e-01 true resid norm
> 5.655832090079e+01 ||r(i)||/||b|| 3.236460464638e+00
>  4 KSP unpreconditioned resid norm 5.974054975332e-01 true resid norm
> 5.997203971117e+01 ||r(i)||/||b|| 3.431805124649e+00
>  5 KSP unpreconditioned resid norm 5.956696306552e-01 true resid norm
> 9.280866979439e+01 ||r(i)||/||b|| 5.310829348914e+00
>  6 KSP unpreconditioned resid norm 5.944323800377e-01 true resid norm
> 9.573083746155e+01 ||r(i)||/||b|| 5.478045772159e+00
>  7 KSP unpreconditioned resid norm 5.933318368437e-01 true resid norm
> 1.283420019927e+02 ||r(i)||/||b|| 7.344168086790e+00
>  8 KSP unpreconditioned resid norm 5.924396174526e-01 true resid norm
> 1.311457478032e+02 ||r(i)||/||b|| 7.504608006575e+00
>  9 KSP unpreconditioned resid norm 5.916244555561e-01 true resid norm
> 1.636933991914e+02 ||r(i)||/||b|| 9.367095882044e+00
>  10 KSP unpreconditioned resid norm 5.909266031948e-01 true resid norm
> 1.664685701658e+02 ||r(i)||/||b|| 9.525900652028e+00
>  11 KSP unpreconditioned resid norm 5.902780068683e-01 true resid norm
> 1.989974529969e+02 ||r(i)||/||b|| 1.138731452650e+01
>  12 KSP unpreconditioned resid norm 5.897052848025e-01 true resid norm
> 2.017662138832e+02 ||r(i)||/||b|| 1.154575248933e+01
>  13 KSP unpreconditioned resid norm 5.891666923831e-01 true resid norm
> 2.342886435057e+02 ||r(i)||/||b|| 1.340679708915e+01
>
> and so on, excessively long
>
> > Your SNES callback gets two Mat arguments, into which are you assembling
> > your approximate Jacobian?
>
> I am assembling my (exact) Jacobian in the second (preconditioner) matrix.
>
> > Are you still calling MatAssemblyBegin/End on the
> > first argument, as in the examples?
>
> Yes, I do.
>
> The moment I remove -snes_mf_operator from the options I get the
> expected results in the expected time.
>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/cba71eef/attachment.htm>

From dominik at itis.ethz.ch  Wed Apr 25 12:42:16 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 19:42:16 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
Message-ID: <CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>

> Start by comparing residuals, with an without. Is SNES 0 the same? Is KSP 0
> the same? KSP 1?

BCGS, with the option:

  0 SNES Function norm 1.747536282885e+01
  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 6.112827145626e-01 true resid norm
1.751036369876e+01 ||r(i)||/||b|| 1.002002869425e+00

BCGS, without:

  0 SNES Function norm 1.747536282885e+01
  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 2.930616102426e-01 true resid norm
2.930616102426e-01 ||r(i)||/||b|| 1.676998715922e-02

GMRES, with the option:

  0 SNES Function norm 1.747536282885e+01
  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 6.122286256494e-01 true resid norm
1.747536251439e+01 ||r(i)||/||b|| 9.999999820055e-01

GMRES, without:

  0 SNES Function norm 1.747536282885e+01
  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 6.122286294217e-01 true resid norm
6.122286294218e-01 ||r(i)||/||b|| 3.503381505824e-02

Dominik

From fuentesdt at gmail.com  Wed Apr 25 12:50:09 2012
From: fuentesdt at gmail.com (David Fuentes)
Date: Wed, 25 Apr 2012 12:50:09 -0500
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
Message-ID: <CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>

I'm pretty sure I pulled the latest on this petsc-dev and Generator, but
when i change GPU type to PetscReal or double, i'm getting key errors ?

hg diff
diff -r 4a642f9b7b70 config/PETSc/FEM.py
--- a/config/PETSc/FEM.py       Wed Apr 25 02:13:29 2012 -0500
+++ b/config/PETSc/FEM.py       Wed Apr 25 12:39:03 2012 -0500
@@ -434,7 +434,7 @@
     from Cxx import Define, Declarator
     dim  = element.get_reference_element().get_spatial_dimension()
     ext  = '_'+str(num)
-    real = 'float'
+    real = 'PetscReal'


SCRGP2$ python $PETSC_DIR//bin/pythonscripts/PetscGenerateFEMQuadrature.py
3 1 1 1 laplace ex52.h
[{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
{(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
{0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
[], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
Perm: [0, 1, 2, 3]
Creating /home/fuentes/tutorials/ex52.h
Traceback (most recent call last):
  File
"/opt/apps/PETSC/petsc-dev//bin/pythonscripts/PetscGenerateFEMQuadrature.py",
line 31, in <module>
    generator.run(elements, numBlocks, operator, filename)
  File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 977, in run
    self.outputElementSource(self.getElementSource(elements, numBlocks,
operator, sourceType = 'GPU'),
os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1])
  File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 936, in
getElementSource
    defns.extend(self.getComputationTypes(element, n))
  File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 450, in
getComputationTypes
    vecType.type       = self.Cxx.typeMap[real+str(dim)]
KeyError: 'PetscReal3'


do I need to add something like below ?


cd externalpackages/Generator/; hg diff
diff -r 361a2eec9c01 CxxHelper.py
--- a/CxxHelper.py      Tue Mar 27 09:01:27 2012 -0500
+++ b/CxxHelper.py      Wed Apr 25 12:39:12 2012 -0500
@@ -98,6 +98,11 @@
     cxxType.identifier = 'float'
     cxxType.baseType = True
     cxxType.const = True
+    typeMap['double3'] = cxxType
+    cxxType = Type()
+    cxxType.identifier = 'double'
+    cxxType.baseType = True
+    cxxType.const = True
     typeMap['const float1'] = cxxType
     cxxType = Type()
     cxxType.identifier = 'float2'






On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>
>>> Yes. This would work.
>>> I had trouble compiling in single precision using the some of the
>>> external package options I was using for double.
>>>
>>
>> Okay,  getting on it now.
>>
>
> Okay, I have tested using PetscReal=double with floats on the GPU. If you
> want doubles on the GPU, you can
> currently change PETSc/FEM.py:434. I am deciding what the best way to
> specify this type is.
>
>   Thanks,
>
>      Matt
>
>
>>    Matt
>>
>>
>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>
>>>>> works!
>>>>>
>>>>
>>>> Excellent. Now, my thinking was that GPUs are most useful doing single
>>>> work, but
>>>> I can see the utility of double accuracy for a residual.
>>>>
>>>> My inclination is to define another type, say GPUReal, and use it for
>>>> all kernels.
>>>> Would that do what you want?
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> SCRGP2$ make ex52
>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>> declared but never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>> never referenced
>>>>>
>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>> declared but never referenced
>>>>>
>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>  -lpetsc
>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>
>>>>>
>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>  N_t: 3, N_cb: 1
>>>>> Residual:
>>>>> Vector Object: 1 MPI processes
>>>>>   type: seq
>>>>> -0.25
>>>>> -0.5
>>>>> 0.25
>>>>> -0.5
>>>>> -1
>>>>> 0.5
>>>>> 0.25
>>>>> 0.5
>>>>> 0.75
>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>> Residual:
>>>>> Vector Object: 1 MPI processes
>>>>>   type: seq
>>>>> -0.25
>>>>> -0.5
>>>>> 0.25
>>>>> -0.5
>>>>> -1
>>>>> 0.5
>>>>> 0.25
>>>>> 0.5
>>>>> 0.75
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>
>>>>>> sure. will do.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>>
>>>>>>>> thanks! its running, but I seem to be getting different answer for
>>>>>>>> cpu/gpu ?
>>>>>>>> i had some floating point problems on this Tesla M2070 gpu before,
>>>>>>>> but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>
>>>>>>>>
>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>
>>>>>>>
>>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>>> have not decided exactly how to handle it. Can you
>>>>>>> make another ARCH --with-precision=single and make sure it works,
>>>>>>> and then we can fix the double issue?
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> 0
>>>>>>>> 755712
>>>>>>>> 0
>>>>>>>> -58720
>>>>>>>> -2953.13
>>>>>>>> 0.375
>>>>>>>> 1.50323e+07
>>>>>>>> 0.875
>>>>>>>> 0
>>>>>>>> SCRGP2$
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> -0.25
>>>>>>>> -0.5
>>>>>>>> 0.25
>>>>>>>> -0.5
>>>>>>>> -1
>>>>>>>> 0.5
>>>>>>>> 0.25
>>>>>>>> 0.5
>>>>>>>> 0.75
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>
>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>
>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>> sizeof(float));CHKERRQ(ierr);
>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ python
>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>> laplacian ex52.h
>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>> []}}
>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>> []}}
>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>>> declared but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>> but never referenced
>>>>>>>>>>
>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>>> declared but never referenced
>>>>>>>>>>
>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>> Residual:
>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>   type: seq
>>>>>>>>>> -0.25
>>>>>>>>>> -0.5
>>>>>>>>>> 0.25
>>>>>>>>>> -0.5
>>>>>>>>>> -1
>>>>>>>>>> 0.5
>>>>>>>>>> 0.25
>>>>>>>>>> 0.5
>>>>>>>>>> 0.75
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>> If so, applications like Preview can lock up the GPU. I know of no
>>>>>>>>> way to test this in CUDA while running. I just close
>>>>>>>>> apps until it runs.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <bourdin at lsu.edu
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>>> using the Python build system
>>>>>>>>>>>>
>>>>>>>>>>>>   ./config/builder2.py check src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>
>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Blaise
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>
>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>      Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> David
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  --
>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>> Technology
>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/3614b519/attachment-0001.htm>

From mirzadeh at gmail.com  Wed Apr 25 12:55:44 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 25 Apr 2012 10:55:44 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAM9tzSmYTHjxO5BtcQQJOKkpKaOHxSXdEp8gxMHhiP3of1hFkw@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAM9tzSmYTHjxO5BtcQQJOKkpKaOHxSXdEp8gxMHhiP3of1hFkw@mail.gmail.com>
Message-ID: <CAKsPKwnZ2N17n-0mMF0JWvRd4YaO2dbX7U-m0e2dg+SVD5w1pA@mail.gmail.com>

> You are not checking error codes, otherwise you wouldn't see all the
subsequent errors.

That's correct. I don't quite remember, but I think at some point I had
trouble with function returns and CHKERRQ( and also CHKERRV? -- not sure
though) and so I dropped it; is this terrible or just bad? PETSc still
produces errors when I'm not using it so I though I'm ok!

> The default behavior was changed in petsc-dev because forgetting to
preallocate was too common of a mistake

Thanks. In fact I do preallocate(exactly), it may be that I'm not doing it
right; gotta check.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/d393ef7b/attachment.htm>

From knepley at gmail.com  Wed Apr 25 13:00:01 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 14:00:01 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
Message-ID: <CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>

On Wed, Apr 25, 2012 at 1:42 PM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> > Start by comparing residuals, with an without. Is SNES 0 the same? Is
> KSP 0
> > the same? KSP 1?
>
> BCGS, with the option:
>
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.112827145626e-01 true resid norm
> 1.751036369876e+01 ||r(i)||/||b|| 1.002002869425e+00
>
> BCGS, without:
>
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 2.930616102426e-01 true resid norm
> 2.930616102426e-01 ||r(i)||/||b|| 1.676998715922e-02
>
> GMRES, with the option:
>
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.122286256494e-01 true resid norm
> 1.747536251439e+01 ||r(i)||/||b|| 9.999999820055e-01
>

Notice that there is no increase in the residual as is required. Notice
also that
there is no decrease. This must mean that the preconditioner is rank
deficient,
and I think it probably means that it is 0 since I find it hard to believe
that P x
produces no component in the b direction.

   Matt


> GMRES, without:
>
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.122286294217e-01 true resid norm
> 6.122286294218e-01 ||r(i)||/||b|| 3.503381505824e-02
>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/3e1361f4/attachment.htm>

From mirzadeh at gmail.com  Wed Apr 25 13:03:05 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 25 Apr 2012 11:03:05 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
Message-ID: <CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>

Hong,

> Did you follow same procedural call as petsc examples,
e.g., petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c:
>   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
>  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>  ierr =
MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr);
>  ierr = MatSeqAIJSetPreallocation(A,5,PETSC_NULL);CHKERRQ(ierr);

Well what I do is *similar*

MatCreate(comm, &A);
MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
globalColumnSize);
MatSetType(A, MATMPIAIJ);
MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
MatSetFromOptions(A);
MatGetOwnershipRange(A, &rStart, &rEnd);


This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
dev. The only difference I can see is 1) the order of MatSetFromOptions and
2) I do not call MatSeqAIJSetPreallocation which I think I do not need
anyway. Is there something I'm doing wrong?

Mohammd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/0ccae1d1/attachment-0001.htm>

From knepley at gmail.com  Wed Apr 25 13:06:09 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 14:06:09 -0400
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
	<CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>
Message-ID: <CAMYG4GnH2WSJdExhFsQ=BPp9p2MfnTMGn0b1p+2g6vB83zxfXQ@mail.gmail.com>

On Wed, Apr 25, 2012 at 1:50 PM, David Fuentes <fuentesdt at gmail.com> wrote:

> I'm pretty sure I pulled the latest on this petsc-dev and Generator, but
> when i change GPU type to PetscReal or double, i'm getting key errors ?
>

Ah well, I was getting around to fixing this. The problem is a little
deeper.  The C version uses pointers everywhere to be
general, but I was using the CUDA types since I thought the compiler would
be more aggressive even though it is
limiting. However, despite having float1, float2, float3, and float4, CUDA
provides only double2. I do all my tests in float.
I will fix this after the baby goes down and mail back.

  Thanks,

     Matt

hg diff
> diff -r 4a642f9b7b70 config/PETSc/FEM.py
> --- a/config/PETSc/FEM.py       Wed Apr 25 02:13:29 2012 -0500
> +++ b/config/PETSc/FEM.py       Wed Apr 25 12:39:03 2012 -0500
> @@ -434,7 +434,7 @@
>      from Cxx import Define, Declarator
>      dim  = element.get_reference_element().get_spatial_dimension()
>      ext  = '_'+str(num)
> -    real = 'float'
> +    real = 'PetscReal'
>
>
> SCRGP2$ python $PETSC_DIR//bin/pythonscripts/PetscGenerateFEMQuadrature.py
> 3 1 1 1 laplace ex52.h
> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
> [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
> Perm: [0, 1, 2, 3]
> Creating /home/fuentes/tutorials/ex52.h
> Traceback (most recent call last):
>   File
> "/opt/apps/PETSC/petsc-dev//bin/pythonscripts/PetscGenerateFEMQuadrature.py",
> line 31, in <module>
>     generator.run(elements, numBlocks, operator, filename)
>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 977, in run
>     self.outputElementSource(self.getElementSource(elements, numBlocks,
> operator, sourceType = 'GPU'),
> os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1])
>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 936, in
> getElementSource
>     defns.extend(self.getComputationTypes(element, n))
>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 450, in
> getComputationTypes
>     vecType.type       = self.Cxx.typeMap[real+str(dim)]
> KeyError: 'PetscReal3'
>
>
> do I need to add something like below ?
>
>
> cd externalpackages/Generator/; hg diff
> diff -r 361a2eec9c01 CxxHelper.py
> --- a/CxxHelper.py      Tue Mar 27 09:01:27 2012 -0500
> +++ b/CxxHelper.py      Wed Apr 25 12:39:12 2012 -0500
> @@ -98,6 +98,11 @@
>      cxxType.identifier = 'float'
>      cxxType.baseType = True
>      cxxType.const = True
> +    typeMap['double3'] = cxxType
> +    cxxType = Type()
> +    cxxType.identifier = 'double'
> +    cxxType.baseType = True
> +    cxxType.const = True
>      typeMap['const float1'] = cxxType
>      cxxType = Type()
>      cxxType.identifier = 'float2'
>
>
>
>
>
>
> On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>
>>>> Yes. This would work.
>>>> I had trouble compiling in single precision using the some of the
>>>> external package options I was using for double.
>>>>
>>>
>>> Okay,  getting on it now.
>>>
>>
>> Okay, I have tested using PetscReal=double with floats on the GPU. If you
>> want doubles on the GPU, you can
>> currently change PETSc/FEM.py:434. I am deciding what the best way to
>> specify this type is.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>    Matt
>>>
>>>
>>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>
>>>>>> works!
>>>>>>
>>>>>
>>>>> Excellent. Now, my thinking was that GPUs are most useful doing single
>>>>> work, but
>>>>> I can see the utility of double accuracy for a residual.
>>>>>
>>>>> My inclination is to define another type, say GPUReal, and use it for
>>>>> all kernels.
>>>>> Would that do what you want?
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> SCRGP2$ make ex52
>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>> declared but never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>> never referenced
>>>>>>
>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>> declared but never referenced
>>>>>>
>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>  -lpetsc
>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>
>>>>>>
>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>  N_t: 3, N_cb: 1
>>>>>> Residual:
>>>>>> Vector Object: 1 MPI processes
>>>>>>   type: seq
>>>>>> -0.25
>>>>>> -0.5
>>>>>> 0.25
>>>>>> -0.5
>>>>>> -1
>>>>>> 0.5
>>>>>> 0.25
>>>>>> 0.5
>>>>>> 0.75
>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>> Residual:
>>>>>> Vector Object: 1 MPI processes
>>>>>>   type: seq
>>>>>> -0.25
>>>>>> -0.5
>>>>>> 0.25
>>>>>> -0.5
>>>>>> -1
>>>>>> 0.5
>>>>>> 0.25
>>>>>> 0.5
>>>>>> 0.75
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>
>>>>>>> sure. will do.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <fuentesdt at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> thanks! its running, but I seem to be getting different answer for
>>>>>>>>> cpu/gpu ?
>>>>>>>>> i had some floating point problems on this Tesla M2070 gpu before,
>>>>>>>>> but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>>>> have not decided exactly how to handle it. Can you
>>>>>>>> make another ARCH --with-precision=single and make sure it works,
>>>>>>>> and then we can fix the double issue?
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>   type: seq
>>>>>>>>> 0
>>>>>>>>> 755712
>>>>>>>>> 0
>>>>>>>>> -58720
>>>>>>>>> -2953.13
>>>>>>>>> 0.375
>>>>>>>>> 1.50323e+07
>>>>>>>>> 0.875
>>>>>>>>> 0
>>>>>>>>> SCRGP2$
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>   type: seq
>>>>>>>>> -0.25
>>>>>>>>> -0.5
>>>>>>>>> 0.25
>>>>>>>>> -0.5
>>>>>>>>> -1
>>>>>>>>> 0.5
>>>>>>>>> 0.25
>>>>>>>>> 0.5
>>>>>>>>> 0.75
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>>
>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>
>>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> No.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>>> sizeof(float));CHKERRQ(ierr);
>>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt *
>>>>>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> SCRGP2$ python
>>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>>> laplacian ex52.h
>>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>> []}}
>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>> []}}
>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>>
>>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>> but never referenced
>>>>>>>>>>>
>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>
>>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>> Residual:
>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>   type: seq
>>>>>>>>>>> -0.25
>>>>>>>>>>> -0.5
>>>>>>>>>>> 0.25
>>>>>>>>>>> -0.5
>>>>>>>>>>> -1
>>>>>>>>>>> 0.5
>>>>>>>>>>> 0.25
>>>>>>>>>>> 0.5
>>>>>>>>>>> 0.75
>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>> -gpu
>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>>> If so, applications like Preview can lock up the GPU. I know of
>>>>>>>>>> no way to test this in CUDA while running. I just close
>>>>>>>>>> apps until it runs.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <
>>>>>>>>>>>> bourdin at lsu.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>>>> using the Python build system
>>>>>>>>>>>>>
>>>>>>>>>>>>>   ./config/builder2.py check src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>
>>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>>
>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>      Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Blaise
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  --
>>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>>> Technology
>>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/a67ec439/attachment-0001.htm>

From knepley at gmail.com  Wed Apr 25 13:07:57 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 14:07:57 -0400
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwnZ2N17n-0mMF0JWvRd4YaO2dbX7U-m0e2dg+SVD5w1pA@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAM9tzSmYTHjxO5BtcQQJOKkpKaOHxSXdEp8gxMHhiP3of1hFkw@mail.gmail.com>
	<CAKsPKwnZ2N17n-0mMF0JWvRd4YaO2dbX7U-m0e2dg+SVD5w1pA@mail.gmail.com>
Message-ID: <CAMYG4G=-oMOmmK8u_EE6_Tpu2qYBY4kU8qze0kBPb_bp0n=8ZA@mail.gmail.com>

On Wed, Apr 25, 2012 at 1:55 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> > You are not checking error codes, otherwise you wouldn't see all the
> subsequent errors.
>
> That's correct. I don't quite remember, but I think at some point I had
> trouble with function returns and CHKERRQ( and also CHKERRV? -- not sure
> though) and so I dropped it; is this terrible or just bad? PETSc still
> produces errors when I'm not using it so I though I'm ok!
>

Terrible.

   Matt


> > The default behavior was changed in petsc-dev because forgetting to
> preallocate was too common of a mistake
>
> Thanks. In fact I do preallocate(exactly), it may be that I'm not doing it
> right; gotta check.
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/ca811ab2/attachment.htm>

From dominik at itis.ethz.ch  Wed Apr 25 13:19:23 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 20:19:23 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>
Message-ID: <CALm7otp_1yZbgP_PAwZJX07uAJBDpiA4Qkreds2AF9icNdYYhQ@mail.gmail.com>

> Notice that there is no increase in the residual as is required. Notice also
> that
> there is no decrease. This must mean that the preconditioner is rank
> deficient,
> and I think it probably means that it is 0 since I find it hard to believe
> that P x
> produces no component in the b direction.

Should this be the case, how would I get correct results for several
benchmarks without using the option?

Dominik

From knepley at gmail.com  Wed Apr 25 13:27:59 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 14:27:59 -0400
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4GnH2WSJdExhFsQ=BPp9p2MfnTMGn0b1p+2g6vB83zxfXQ@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
	<CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>
	<CAMYG4GnH2WSJdExhFsQ=BPp9p2MfnTMGn0b1p+2g6vB83zxfXQ@mail.gmail.com>
Message-ID: <CAMYG4Gnkt9z180P82=D9keP3k=AZOmqjjc34T0s98uzwULYiCg@mail.gmail.com>

On Wed, Apr 25, 2012 at 2:06 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Apr 25, 2012 at 1:50 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>
>> I'm pretty sure I pulled the latest on this petsc-dev and Generator, but
>> when i change GPU type to PetscReal or double, i'm getting key errors ?
>>
>
> Ah well, I was getting around to fixing this. The problem is a little
> deeper.  The C version uses pointers everywhere to be
> general, but I was using the CUDA types since I thought the compiler would
> be more aggressive even though it is
> limiting. However, despite having float1, float2, float3, and float4, CUDA
> provides only double2. I do all my tests in float.
> I will fix this after the baby goes down and mail back.
>

Okay, this is bizarre. It builds with double3 (not in the CUDA manual). You
will need to pull both petsc and Generator, but
it should work now. I have not really tested with double, so this will be
interesting.

   Matt


>   Thanks,
>
>      Matt
>
> hg diff
>> diff -r 4a642f9b7b70 config/PETSc/FEM.py
>> --- a/config/PETSc/FEM.py       Wed Apr 25 02:13:29 2012 -0500
>> +++ b/config/PETSc/FEM.py       Wed Apr 25 12:39:03 2012 -0500
>> @@ -434,7 +434,7 @@
>>      from Cxx import Define, Declarator
>>      dim  = element.get_reference_element().get_spatial_dimension()
>>      ext  = '_'+str(num)
>> -    real = 'float'
>> +    real = 'PetscReal'
>>
>>
>> SCRGP2$ python
>> $PETSC_DIR//bin/pythonscripts/PetscGenerateFEMQuadrature.py 3 1 1 1 laplace
>> ex52.h
>> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
>> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
>> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
>> [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
>> Perm: [0, 1, 2, 3]
>> Creating /home/fuentes/tutorials/ex52.h
>> Traceback (most recent call last):
>>   File
>> "/opt/apps/PETSC/petsc-dev//bin/pythonscripts/PetscGenerateFEMQuadrature.py",
>> line 31, in <module>
>>     generator.run(elements, numBlocks, operator, filename)
>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 977, in run
>>     self.outputElementSource(self.getElementSource(elements, numBlocks,
>> operator, sourceType = 'GPU'),
>> os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1])
>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 936, in
>> getElementSource
>>     defns.extend(self.getComputationTypes(element, n))
>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 450, in
>> getComputationTypes
>>     vecType.type       = self.Cxx.typeMap[real+str(dim)]
>> KeyError: 'PetscReal3'
>>
>>
>> do I need to add something like below ?
>>
>>
>> cd externalpackages/Generator/; hg diff
>> diff -r 361a2eec9c01 CxxHelper.py
>> --- a/CxxHelper.py      Tue Mar 27 09:01:27 2012 -0500
>> +++ b/CxxHelper.py      Wed Apr 25 12:39:12 2012 -0500
>> @@ -98,6 +98,11 @@
>>      cxxType.identifier = 'float'
>>      cxxType.baseType = True
>>      cxxType.const = True
>> +    typeMap['double3'] = cxxType
>> +    cxxType = Type()
>> +    cxxType.identifier = 'double'
>> +    cxxType.baseType = True
>> +    cxxType.const = True
>>      typeMap['const float1'] = cxxType
>>      cxxType = Type()
>>      cxxType.identifier = 'float2'
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>
>>>>> Yes. This would work.
>>>>> I had trouble compiling in single precision using the some of the
>>>>> external package options I was using for double.
>>>>>
>>>>
>>>> Okay,  getting on it now.
>>>>
>>>
>>> Okay, I have tested using PetscReal=double with floats on the GPU. If
>>> you want doubles on the GPU, you can
>>> currently change PETSc/FEM.py:434. I am deciding what the best way to
>>> specify this type is.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>    Matt
>>>>
>>>>
>>>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>
>>>>>>> works!
>>>>>>>
>>>>>>
>>>>>> Excellent. Now, my thinking was that GPUs are most useful doing
>>>>>> single work, but
>>>>>> I can see the utility of double accuracy for a residual.
>>>>>>
>>>>>> My inclination is to define another type, say GPUReal, and use it for
>>>>>> all kernels.
>>>>>> Would that do what you want?
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>> SCRGP2$ make ex52
>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>> but never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>> but never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>> declared but never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>> but never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>> never referenced
>>>>>>>
>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>> declared but never referenced
>>>>>>>
>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>  -lpetsc
>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>
>>>>>>>
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>  N_t: 3, N_cb: 1
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>>   type: seq
>>>>>>> -0.25
>>>>>>> -0.5
>>>>>>> 0.25
>>>>>>> -0.5
>>>>>>> -1
>>>>>>> 0.5
>>>>>>> 0.25
>>>>>>> 0.5
>>>>>>> 0.75
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>>   type: seq
>>>>>>> -0.25
>>>>>>> -0.5
>>>>>>> 0.25
>>>>>>> -0.5
>>>>>>> -1
>>>>>>> 0.5
>>>>>>> 0.25
>>>>>>> 0.5
>>>>>>> 0.75
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>>
>>>>>>>> sure. will do.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <
>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> thanks! its running, but I seem to be getting different answer
>>>>>>>>>> for cpu/gpu ?
>>>>>>>>>> i had some floating point problems on this Tesla M2070 gpu
>>>>>>>>>> before, but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>>>>> have not decided exactly how to handle it. Can you
>>>>>>>>> make another ARCH --with-precision=single and make sure it works,
>>>>>>>>> and then we can fix the double issue?
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>>>> Residual:
>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>   type: seq
>>>>>>>>>> 0
>>>>>>>>>> 755712
>>>>>>>>>> 0
>>>>>>>>>> -58720
>>>>>>>>>> -2953.13
>>>>>>>>>> 0.375
>>>>>>>>>> 1.50323e+07
>>>>>>>>>> 0.875
>>>>>>>>>> 0
>>>>>>>>>> SCRGP2$
>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>> Residual:
>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>   type: seq
>>>>>>>>>> -0.25
>>>>>>>>>> -0.5
>>>>>>>>>> 0.25
>>>>>>>>>> -0.5
>>>>>>>>>> -1
>>>>>>>>>> 0.5
>>>>>>>>>> 0.25
>>>>>>>>>> 0.5
>>>>>>>>>> 0.75
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>>>
>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>
>>>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> No.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt
>>>>>>>>>>>> * sizeof(float));CHKERRQ(ierr);
>>>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt
>>>>>>>>>>>> * sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> SCRGP2$ python
>>>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>>>> laplacian ex52.h
>>>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>>> []}}
>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>>> []}}
>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>>>
>>>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>>>>    -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>>>>> but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>
>>>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>> Residual:
>>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>>   type: seq
>>>>>>>>>>>> -0.25
>>>>>>>>>>>> -0.5
>>>>>>>>>>>> 0.25
>>>>>>>>>>>> -0.5
>>>>>>>>>>>> -1
>>>>>>>>>>>> 0.5
>>>>>>>>>>>> 0.25
>>>>>>>>>>>> 0.5
>>>>>>>>>>>> 0.75
>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>> -gpu
>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>>>> If so, applications like Preview can lock up the GPU. I know of
>>>>>>>>>>> no way to test this in CUDA while running. I just close
>>>>>>>>>>> apps until it runs.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <
>>>>>>>>>>>>> bourdin at lsu.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>>>>> using the Python build system
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   ./config/builder2.py check
>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blaise
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  --
>>>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>>>> Technology
>>>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/5fa4456a/attachment-0001.htm>

From hzhang at mcs.anl.gov  Wed Apr 25 13:31:16 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Wed, 25 Apr 2012 13:31:16 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
Message-ID: <CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>

Mohammad:

>
> MatCreate(comm, &A);
> MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
> globalColumnSize);
> MatSetType(A, MATMPIAIJ);
> MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
> MatSetFromOptions(A);
> MatGetOwnershipRange(A, &rStart, &rEnd);
>
>
> This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
> dev. The only difference I can see is 1) the order of MatSetFromOptions and
> 2) I do not call MatSeqAIJSetPreallocation which I think I do not need
> anyway. Is there something I'm doing wrong?
>

MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
If user set mattype at runtime, MatSetFromOptions() picks it and set the
type accordingly. SetPreallocation()
will be called after the type is set.

Hong

>
> Mohammd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/8e9fc0b5/attachment.htm>

From knepley at gmail.com  Wed Apr 25 13:31:03 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 14:31:03 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otp_1yZbgP_PAwZJX07uAJBDpiA4Qkreds2AF9icNdYYhQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>
	<CALm7otp_1yZbgP_PAwZJX07uAJBDpiA4Qkreds2AF9icNdYYhQ@mail.gmail.com>
Message-ID: <CAMYG4Gmdx1Cosvu6qBSxzZjLLZw3cJUVUdC6dQTWLe0Da9oGyQ@mail.gmail.com>

On Wed, Apr 25, 2012 at 2:19 PM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> > Notice that there is no increase in the residual as is required. Notice
> also
> > that
> > there is no decrease. This must mean that the preconditioner is rank
> > deficient,
> > and I think it probably means that it is 0 since I find it hard to
> believe
> > that P x
> > produces no component in the b direction.
>
> Should this be the case, how would I get correct results for several
> benchmarks without using the option?


This is not what I said. Look closely at the above. I said the
preconditioner you give
back when using -snes_mf_operator is not right. This has nothing to do with
normal
operation.

   Matt


>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/46cbfb13/attachment.htm>

From bsmith at mcs.anl.gov  Wed Apr 25 15:02:56 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Apr 2012 15:02:56 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
Message-ID: <8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>


   The true residual decreases nicely without matrix free operator, otherwise it does not decrease.  This likely means the preconditioner is no good for the matrix free, but is good for the matrix based. This means that the matrix based multiply is not the same as the matrix free multiply. These means one of the matrix multiplies is wrong. My guess is that your "exact" Jacobian  is wrong. Run with -snes_type test and see what it reports.

   You can also run the code with the options -snes_mf -pc_type none and then again with just -pc_type none (just for a few KSP iterations) and compare the residual norms. If the matrix free and the matrix define the same matrix vector product then these residuals for the first few iterations should be pretty similar.


   Barry

On Apr 25, 2012, at 12:42 PM, Dominik Szczerba wrote:

>> Start by comparing residuals, with an without. Is SNES 0 the same? Is KSP 0
>> the same? KSP 1?
> 
> BCGS, with the option:
> 
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.112827145626e-01 true resid norm
> 1.751036369876e+01 ||r(i)||/||b|| 1.002002869425e+00
> 
> BCGS, without:
> 
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 2.930616102426e-01 true resid norm
> 2.930616102426e-01 ||r(i)||/||b|| 1.676998715922e-02
> 
> GMRES, with the option:
> 
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.122286256494e-01 true resid norm
> 1.747536251439e+01 ||r(i)||/||b|| 9.999999820055e-01
> 
> GMRES, without:
> 
>  0 SNES Function norm 1.747536282885e+01
>  0 KSP unpreconditioned resid norm 1.747536282885e+01 true resid norm
> 1.747536282885e+01 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP unpreconditioned resid norm 6.122286294217e-01 true resid norm
> 6.122286294218e-01 ||r(i)||/||b|| 3.503381505824e-02
> 
> Dominik


From dominik at itis.ethz.ch  Wed Apr 25 15:03:39 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 22:03:39 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAMYG4Gmdx1Cosvu6qBSxzZjLLZw3cJUVUdC6dQTWLe0Da9oGyQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>
	<CALm7otp_1yZbgP_PAwZJX07uAJBDpiA4Qkreds2AF9icNdYYhQ@mail.gmail.com>
	<CAMYG4Gmdx1Cosvu6qBSxzZjLLZw3cJUVUdC6dQTWLe0Da9oGyQ@mail.gmail.com>
Message-ID: <CALm7otoak-4GiEZv2XduU4O8Qwko_s7DVkuxsDGhGFnPCG3VrQ@mail.gmail.com>

>> Should this be the case, how would I get correct results for several
>> benchmarks without using the option?
>
>
> This is not what I said. Look closely at the above. I said the
> preconditioner you give
> back when using -snes_mf_operator is not right. This has nothing to do with
> normal
> operation.

I am sorry, I do not follow what this might possibly mean for me.

Thanks
Dominik

From fuentesdt at gmail.com  Wed Apr 25 15:38:08 2012
From: fuentesdt at gmail.com (David Fuentes)
Date: Wed, 25 Apr 2012 15:38:08 -0500
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAMYG4Gnkt9z180P82=D9keP3k=AZOmqjjc34T0s98uzwULYiCg@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
	<CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>
	<CAMYG4GnH2WSJdExhFsQ=BPp9p2MfnTMGn0b1p+2g6vB83zxfXQ@mail.gmail.com>
	<CAMYG4Gnkt9z180P82=D9keP3k=AZOmqjjc34T0s98uzwULYiCg@mail.gmail.com>
Message-ID: <CAKnkmZxW9vTcFcXDzOXt39TRdPGk51_naG5mWiPvjvuGdH5_6A@mail.gmail.com>

thanks! I seem to be getting the same answer on cpu and gpu in 3d w/ double.
However,  increasing the dof with -refinement_limit 0.01, the residual is
differing slightly (see below).

in single precision, do you have an idea of the upper bound limit on
degrees of freedom
such that the sieve gpu FEM solver would provide reasonable accuracy in the
solution
(using gpu for assembly of both residual and jacobain with single precision
solve ) ?
a friend was showing me some plots indicating that floating point error
would become significant around O(10^4) dof on a related problem.
would you expect similar ?


SCRGP2$ hg diff
diff -r be5f9068076c config/PETSc/FEM.py
--- a/config/PETSc/FEM.py       Wed Apr 25 13:40:46 2012 -0500
+++ b/config/PETSc/FEM.py       Wed Apr 25 14:58:40 2012 -0500
@@ -10,7 +10,7 @@
     import os
     self.baseDir    = os.getcwd()
     self.quadDegree = -1
-    self.gpuScalarType = 'float'
+    self.gpuScalarType = 'double'
     return


SCRGP2$ make ex52
/usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
-I/opt/apps/PETSC/petsc-dev/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
-I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
-I/opt/MATLAB/R2011a/extern/include -I/usr/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
-I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
 -I/opt/apps/PETSC/petsc-dev/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
-I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
-I/opt/MATLAB/R2011a/extern/include -I/usr/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
-I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
-I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
 ex52_integrateElement.cu
ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
referenced

ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
referenced

ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
referenced

ex52_gpu_inline.h(14): warning: variable "weights_0" was declared but never
referenced

ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
referenced

ex52_gpu_inline.h(30): warning: variable "BasisDerivatives_0" was declared
but never referenced

ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
referenced

ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
referenced

ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
referenced

ex52_gpu_inline.h(14): warning: variable "weights_0" was declared but never
referenced

ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
referenced

ex52_gpu_inline.h(30): warning: variable "BasisDerivatives_0" was declared
but never referenced

/opt/epd-7.1-2-rh5-x86_64/bin/python
/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py 3
1 1 1 laplacian ex52.h
[{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
{(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
{0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
[], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
Perm: [0, 1, 2, 3]
Creating /home/fuentes/tutorials/ex52.h
Creating /home/fuentes/tutorials/ex52_gpu.h
[{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
{(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
{0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
[], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
Perm: [0, 1, 2, 3]
Creating /home/fuentes/tutorials/ex52_gpu_inline.h
/usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
 -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
-L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
-Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
-ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
-lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
-Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
-lcufft -lcublas -lcudart -lcusparse
-Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
-L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
-leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
-Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
-lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
-lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
-L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
-lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
-lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
/bin/rm -f ex52.o ex52_integrateElement.o


SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
-refinement_limit 0.0 -section_view
PetscSection
Process 0:
  (   0) dim  0 offset   0
  (   1) dim  0 offset   0
  (   2) dim  0 offset   0
  (   3) dim  0 offset   0
  (   4) dim  0 offset   0
  (   5) dim  0 offset   0
  (   6) dim  1 offset   0
  (   7) dim  1 offset   1
  (   8) dim  1 offset   2
  (   9) dim  1 offset   3
  (  10) dim  1 offset   4
  (  11) dim  1 offset   5
  (  12) dim  1 offset   6
  (  13) dim  1 offset   7
Residual:
Vector Object: 1 MPI processes
  type: seq
-0.666667
0.166667
0.166667
0.333333
-0.333333
-0.166667
-0.166667
0.666667


SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
-refinement_limit 0.0 -section_view -gpu
PetscSection
Process 0:
  (   0) dim  0 offset   0
  (   1) dim  0 offset   0
  (   2) dim  0 offset   0
  (   3) dim  0 offset   0
  (   4) dim  0 offset   0
  (   5) dim  0 offset   0
  (   6) dim  1 offset   0
  (   7) dim  1 offset   1
  (   8) dim  1 offset   2
  (   9) dim  1 offset   3
  (  10) dim  1 offset   4
  (  11) dim  1 offset   5
  (  12) dim  1 offset   6
  (  13) dim  1 offset   7
GPU layout grid(1,1,1) block(4,1,1) with 1 batches
 N_t: 4, N_cb: 1
Residual:
Vector Object: 1 MPI processes
  type: seq
-0.666667
0.166667
0.166667
0.333333
-0.333333
-0.166667
-0.166667
0.666667


SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
-refinement_limit 0.01
Residual:
Vector Object: 1 MPI processes
  type: seq
-0.0833333
0.0833333
0.0507812
0.11849
-0.0833333
0.0833333
0.40743
0.428263
0.0598958
0.988459
1.10304
0.138428
-0.0833333
-0.125
-0.125
0.0833333
-0.10026
0.0507812
-0.0803223
0.0446777
-0.25
-0.117187
2.06367
1.51159
-0.181424
-0.418944
0.0195312
0.0282118
-0.574219
0.368924
-0.474826
0.0195312
-0.0238715
0.0195312
-0.0792359
-1.49715
1.10186
0.923154
-1.31844
-2.03658
0.0797244
0.0884049
-2.21192



SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
-refinement_limit 0.01 -gpu
GPU layout grid(3,7,1) block(4,1,1) with 1 batches
 N_t: 4, N_cb: 1
Residual:
Vector Object: 1 MPI processes
  type: seq
-0.0833333
0.0833333
0.0507812
0.11849
-0.0833333
0.0833333
0.0576986
0.0785319
0.0598958
0.0832038
0.197787
0.0932617
-0.0833333
-0.125
-0.125
0.0833333
-0.10026
0.0507812
-0.0803223
0.0446777
-0.25
-0.117187
0.345882
-0.206202
-0.181424
0.101074
0.0195312
0.0282118
0.0859375
0.148872
-0.0347222
0.0195312
-0.0238715
0.0195312
-0.588018
-0.117594
0.12749
0.13118
-0.121284
0.0692962
0.118305
0.126986
-0.106051




On Wed, Apr 25, 2012 at 1:27 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Apr 25, 2012 at 2:06 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Apr 25, 2012 at 1:50 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>
>>> I'm pretty sure I pulled the latest on this petsc-dev and Generator, but
>>> when i change GPU type to PetscReal or double, i'm getting key errors ?
>>>
>>
>> Ah well, I was getting around to fixing this. The problem is a little
>> deeper.  The C version uses pointers everywhere to be
>> general, but I was using the CUDA types since I thought the compiler
>> would be more aggressive even though it is
>> limiting. However, despite having float1, float2, float3, and float4,
>> CUDA provides only double2. I do all my tests in float.
>> I will fix this after the baby goes down and mail back.
>>
>
> Okay, this is bizarre. It builds with double3 (not in the CUDA manual).
> You will need to pull both petsc and Generator, but
> it should work now. I have not really tested with double, so this will be
> interesting.
>
>    Matt
>
>
>>   Thanks,
>>
>>      Matt
>>
>> hg diff
>>> diff -r 4a642f9b7b70 config/PETSc/FEM.py
>>> --- a/config/PETSc/FEM.py       Wed Apr 25 02:13:29 2012 -0500
>>> +++ b/config/PETSc/FEM.py       Wed Apr 25 12:39:03 2012 -0500
>>> @@ -434,7 +434,7 @@
>>>      from Cxx import Define, Declarator
>>>      dim  = element.get_reference_element().get_spatial_dimension()
>>>      ext  = '_'+str(num)
>>> -    real = 'float'
>>> +    real = 'PetscReal'
>>>
>>>
>>> SCRGP2$ python
>>> $PETSC_DIR//bin/pythonscripts/PetscGenerateFEMQuadrature.py 3 1 1 1 laplace
>>> ex52.h
>>> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
>>> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
>>> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
>>> [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
>>> Perm: [0, 1, 2, 3]
>>> Creating /home/fuentes/tutorials/ex52.h
>>> Traceback (most recent call last):
>>>   File
>>> "/opt/apps/PETSC/petsc-dev//bin/pythonscripts/PetscGenerateFEMQuadrature.py",
>>> line 31, in <module>
>>>     generator.run(elements, numBlocks, operator, filename)
>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 977, in run
>>>     self.outputElementSource(self.getElementSource(elements, numBlocks,
>>> operator, sourceType = 'GPU'),
>>> os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1])
>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 936, in
>>> getElementSource
>>>     defns.extend(self.getComputationTypes(element, n))
>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 450, in
>>> getComputationTypes
>>>     vecType.type       = self.Cxx.typeMap[real+str(dim)]
>>> KeyError: 'PetscReal3'
>>>
>>>
>>> do I need to add something like below ?
>>>
>>>
>>> cd externalpackages/Generator/; hg diff
>>> diff -r 361a2eec9c01 CxxHelper.py
>>> --- a/CxxHelper.py      Tue Mar 27 09:01:27 2012 -0500
>>> +++ b/CxxHelper.py      Wed Apr 25 12:39:12 2012 -0500
>>> @@ -98,6 +98,11 @@
>>>      cxxType.identifier = 'float'
>>>      cxxType.baseType = True
>>>      cxxType.const = True
>>> +    typeMap['double3'] = cxxType
>>> +    cxxType = Type()
>>> +    cxxType.identifier = 'double'
>>> +    cxxType.baseType = True
>>> +    cxxType.const = True
>>>      typeMap['const float1'] = cxxType
>>>      cxxType = Type()
>>>      cxxType.identifier = 'float2'
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>
>>>>>> Yes. This would work.
>>>>>> I had trouble compiling in single precision using the some of the
>>>>>> external package options I was using for double.
>>>>>>
>>>>>
>>>>> Okay,  getting on it now.
>>>>>
>>>>
>>>> Okay, I have tested using PetscReal=double with floats on the GPU. If
>>>> you want doubles on the GPU, you can
>>>> currently change PETSc/FEM.py:434. I am deciding what the best way to
>>>> specify this type is.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>>
>>>>>>>> works!
>>>>>>>>
>>>>>>>
>>>>>>> Excellent. Now, my thinking was that GPUs are most useful doing
>>>>>>> single work, but
>>>>>>> I can see the utility of double accuracy for a residual.
>>>>>>>
>>>>>>> My inclination is to define another type, say GPUReal, and use it
>>>>>>> for all kernels.
>>>>>>> Would that do what you want?
>>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>> SCRGP2$ make ex52
>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>  -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>> but never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>> but never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>> declared but never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>> but never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>>>>>> never referenced
>>>>>>>>
>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>> declared but never referenced
>>>>>>>>
>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>>  -lpetsc
>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>
>>>>>>>>
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> -0.25
>>>>>>>> -0.5
>>>>>>>> 0.25
>>>>>>>> -0.5
>>>>>>>> -1
>>>>>>>> 0.5
>>>>>>>> 0.25
>>>>>>>> 0.5
>>>>>>>> 0.75
>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>> Residual:
>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>   type: seq
>>>>>>>> -0.25
>>>>>>>> -0.5
>>>>>>>> 0.25
>>>>>>>> -0.5
>>>>>>>> -1
>>>>>>>> 0.5
>>>>>>>> 0.25
>>>>>>>> 0.5
>>>>>>>> 0.75
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> sure. will do.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <
>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> thanks! its running, but I seem to be getting different answer
>>>>>>>>>>> for cpu/gpu ?
>>>>>>>>>>> i had some floating point problems on this Tesla M2070 gpu
>>>>>>>>>>> before, but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>>>>>> have not decided exactly how to handle it. Can you
>>>>>>>>>> make another ARCH --with-precision=single and make sure it works,
>>>>>>>>>> and then we can fix the double issue?
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>> -gpu
>>>>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>>>>> Residual:
>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>   type: seq
>>>>>>>>>>> 0
>>>>>>>>>>> 755712
>>>>>>>>>>> 0
>>>>>>>>>>> -58720
>>>>>>>>>>> -2953.13
>>>>>>>>>>> 0.375
>>>>>>>>>>> 1.50323e+07
>>>>>>>>>>> 0.875
>>>>>>>>>>> 0
>>>>>>>>>>> SCRGP2$
>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>> Residual:
>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>   type: seq
>>>>>>>>>>> -0.25
>>>>>>>>>>> -0.5
>>>>>>>>>>> 0.25
>>>>>>>>>>> -0.5
>>>>>>>>>>> -1
>>>>>>>>>>> 0.5
>>>>>>>>>>> 0.25
>>>>>>>>>>> 0.5
>>>>>>>>>>> 0.75
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>>>>
>>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>
>>>>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt
>>>>>>>>>>>>> * sizeof(float));CHKERRQ(ierr);
>>>>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,         Ne*N_bt
>>>>>>>>>>>>> * sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> SCRGP2$ python
>>>>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>>>>> laplacian ex52.h
>>>>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>>>> []}}
>>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0:
>>>>>>>>>>>>> []}}
>>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>>>>
>>>>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>>>>>    -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>>
>>>>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o
>>>>>>>>>>>>> ex52_integrateElement.o
>>>>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>>> Residual:
>>>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>>>   type: seq
>>>>>>>>>>>>> -0.25
>>>>>>>>>>>>> -0.5
>>>>>>>>>>>>> 0.25
>>>>>>>>>>>>> -0.5
>>>>>>>>>>>>> -1
>>>>>>>>>>>>> 0.5
>>>>>>>>>>>>> 0.25
>>>>>>>>>>>>> 0.5
>>>>>>>>>>>>> 0.75
>>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>>> -gpu
>>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>>>>> If so, applications like Preview can lock up the GPU. I know of
>>>>>>>>>>>> no way to test this in CUDA while running. I just close
>>>>>>>>>>>> apps until it runs.
>>>>>>>>>>>>
>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>      Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <
>>>>>>>>>>>>>> bourdin at lsu.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>>>>>> using the Python build system
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   ./config/builder2.py check
>>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Blaise
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  --
>>>>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>>>>> Technology
>>>>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/33475587/attachment-0001.htm>

From dominik at itis.ethz.ch  Wed Apr 25 16:06:16 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 23:06:16 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
Message-ID: <CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>

> My guess is that your "exact" Jacobian ?is wrong

I am happy to be guilty if I know how I can improve: why do I get the
right results without this option, for several cases?

> Run with -snes_type test and see what it reports.

The moment I specify this option I get:

[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Cannot test with alternative preconditioner!

tracing back to SNESSolve()

Dominik

From jedbrown at mcs.anl.gov  Wed Apr 25 16:07:31 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 16:07:31 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otoak-4GiEZv2XduU4O8Qwko_s7DVkuxsDGhGFnPCG3VrQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<CAMYG4GmXC8NUXQBhKn6L0ccS1kvSEi0sRj=8jh3EnJU4gwZ0_w@mail.gmail.com>
	<CALm7otp_1yZbgP_PAwZJX07uAJBDpiA4Qkreds2AF9icNdYYhQ@mail.gmail.com>
	<CAMYG4Gmdx1Cosvu6qBSxzZjLLZw3cJUVUdC6dQTWLe0Da9oGyQ@mail.gmail.com>
	<CALm7otoak-4GiEZv2XduU4O8Qwko_s7DVkuxsDGhGFnPCG3VrQ@mail.gmail.com>
Message-ID: <CAM9tzSn_eFPzoAtQZTUEqEUUcG8vi=A+vYEoagDMfur8jikpew@mail.gmail.com>

On Wed, Apr 25, 2012 at 15:03, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> > This is not what I said. Look closely at the above. I said the
> > preconditioner you give
> > back when using -snes_mf_operator is not right. This has nothing to do
> with
> > normal
> > operation.
>
> I am sorry, I do not follow what this might possibly mean for me.


The operator you provided for preconditioning is not close to the finite
differenced operator. I.e. your residual evaluation and Jacobian assembly
are different to first order, probably one or both is buggy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/4fc6bf4d/attachment.htm>

From jedbrown at mcs.anl.gov  Wed Apr 25 16:16:33 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 25 Apr 2012 16:16:33 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
Message-ID: <CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>

On Wed, Apr 25, 2012 at 16:06, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I am happy to be guilty if I know how I can improve: why do I get the
> right results without this option, for several cases?
>

How do you define "right results"? Is SNES converging quadratically when
you do not use -snes_mf_operator?


>
> > Run with -snes_type test and see what it reports.
>
> The moment I specify this option I get:
>
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Cannot test with alternative preconditioner!
>
> tracing back to SNESSolve()
>

You either (a) set different matrices for the operator and preconditioning
matrix or (b) passed -snes_mf. Just pass one matrix.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/19a14a00/attachment.htm>

From bsmith at mcs.anl.gov  Wed Apr 25 16:23:31 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Apr 2012 16:23:31 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
Message-ID: <54C3A798-900A-439E-9951-E3C214B23257@mcs.anl.gov>


   Do not pass -snes_mf or -snes_mf_operator with the -snes_type test   option

   Barry

On Apr 25, 2012, at 4:16 PM, Jed Brown wrote:

> On Wed, Apr 25, 2012 at 16:06, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
> I am happy to be guilty if I know how I can improve: why do I get the
> right results without this option, for several cases?
> 
> How do you define "right results"? Is SNES converging quadratically when you do not use -snes_mf_operator?
>  
> 
> > Run with -snes_type test and see what it reports.
> 
> The moment I specify this option I get:
> 
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Cannot test with alternative preconditioner!
> 
> tracing back to SNESSolve()
> 
> You either (a) set different matrices for the operator and preconditioning matrix or (b) passed -snes_mf. Just pass one matrix.


From dominik at itis.ethz.ch  Wed Apr 25 16:31:01 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Wed, 25 Apr 2012 23:31:01 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <54C3A798-900A-439E-9951-E3C214B23257@mcs.anl.gov>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<54C3A798-900A-439E-9951-E3C214B23257@mcs.anl.gov>
Message-ID: <CALm7otpuay1zz5k8o2RVaikW5FoeuodVPWco7+CQaESTKPEchA@mail.gmail.com>

> ? Do not pass -snes_mf or -snes_mf_operator with the -snes_type test ? option

Sorry, I am lost.

I provide an analytical Jacobian to a polynomial function. Sure I can
make mistakes even in such trivial derivations, but how would I get
the results I expect from a book without the option?

Dominik

From bsmith at mcs.anl.gov  Wed Apr 25 16:36:50 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 25 Apr 2012 16:36:50 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otpuay1zz5k8o2RVaikW5FoeuodVPWco7+CQaESTKPEchA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<54C3A798-900A-439E-9951-E3C214B23257@mcs.anl.gov>
	<CALm7otpuay1zz5k8o2RVaikW5FoeuodVPWco7+CQaESTKPEchA@mail.gmail.com>
Message-ID: <6EBCF2CD-4881-4F89-994F-42F936FBBD22@mcs.anl.gov>


   Dominik,

     The -snes_type test option    compares the full Jacobian you provide with one that PETSc computes to try to determine if your Jacobian is correct. Please run with this option and see what happens and send us result. It should print a few numbers and a statement of whether the Jacobian provided is correct or not.

    The option cannot be used with -snes_mf or -snes_mf_operator because the option -snes_type test is ONLY to test the computation of the Jacobin, it does not actually try to solve the system.

   Barry



On Apr 25, 2012, at 4:31 PM, Dominik Szczerba wrote:

>>   Do not pass -snes_mf or -snes_mf_operator with the -snes_type test   option
> 
> Sorry, I am lost.
> 
> I provide an analytical Jacobian to a polynomial function. Sure I can
> make mistakes even in such trivial derivations, but how would I get
> the results I expect from a book without the option?
> 
> Dominik


From mirzadeh at gmail.com  Wed Apr 25 18:04:24 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 25 Apr 2012 16:04:24 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
	<CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
Message-ID: <CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>

Thanks Hong; that fixed the problem.

On Wed, Apr 25, 2012 at 11:31 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:

> Mohammad:
>
>>
>> MatCreate(comm, &A);
>> MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
>> globalColumnSize);
>> MatSetType(A, MATMPIAIJ);
>> MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
>> MatSetFromOptions(A);
>> MatGetOwnershipRange(A, &rStart, &rEnd);
>>
>>
>> This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
>> dev. The only difference I can see is 1) the order of MatSetFromOptions and
>> 2) I do not call MatSeqAIJSetPreallocation which I think I do not need
>> anyway. Is there something I'm doing wrong?
>>
>
> MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
> If user set mattype at runtime, MatSetFromOptions() picks it and set the
> type accordingly. SetPreallocation()
> will be called after the type is set.
>
> Hong
>
>>
>> Mohammd
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/d8747353/attachment.htm>

From iamkyungjoo at gmail.com  Wed Apr 25 18:06:39 2012
From: iamkyungjoo at gmail.com (Kyungjoo Kim)
Date: Wed, 25 Apr 2012 18:06:39 -0500
Subject: [petsc-users] Question about MatDestroy...
Message-ID: <83B566E5-2F9A-4A99-A44A-A339794631E2@gmail.com>

Hello, 

Lately I experienced very weird behavior in PETSC.

While solving small size problem (25k) on my laptop, it MatDestroy clean the objects.
Another test (40k unknowns) problem use most of my memory and left around 500 MB free while solving the problem.

Then when I call MatDestroy, PETSC request more memory and the available memory goes down 8MB (It start scratching my hdd, so I stopped).

Why MatDestroy need more memory than solving phase ?


Any idea ?


Thank you in advance.

From knepley at gmail.com  Wed Apr 25 19:22:24 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 20:22:24 -0400
Subject: [petsc-users] Question about MatDestroy...
In-Reply-To: <83B566E5-2F9A-4A99-A44A-A339794631E2@gmail.com>
References: <83B566E5-2F9A-4A99-A44A-A339794631E2@gmail.com>
Message-ID: <CAMYG4G=DTscj0Fto3Bds7E-oR1MiYSdO_i6n+C5cLwLt2Qy5JA@mail.gmail.com>

On Wed, Apr 25, 2012 at 7:06 PM, Kyungjoo Kim <iamkyungjoo at gmail.com> wrote:

> Hello,
>
> Lately I experienced very weird behavior in PETSC.
>
> While solving small size problem (25k) on my laptop, it MatDestroy clean
> the objects.
> Another test (40k unknowns) problem use most of my memory and left around
> 500 MB free while solving the problem.
>
> Then when I call MatDestroy, PETSC request more memory and the available
> memory goes down 8MB (It start scratching my hdd, so I stopped).
>
> Why MatDestroy need more memory than solving phase ?
>

This is a misinterpretation. MatDestroy() does not allocate memory.

  Thanks,

     Matt


> Any idea ?
>
>
> Thank you in advance.




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/c16879d5/attachment-0001.htm>

From knepley at gmail.com  Wed Apr 25 19:40:00 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 25 Apr 2012 20:40:00 -0400
Subject: [petsc-users] ex52_integrateElement.cu
In-Reply-To: <CAKnkmZxW9vTcFcXDzOXt39TRdPGk51_naG5mWiPvjvuGdH5_6A@mail.gmail.com>
References: <CAKnkmZy6E08r1eYcOwF3AEZMfRNc9jqfYtDJTKnMBY1LnfACgg@mail.gmail.com>
	<CAMYG4Gk3Y05nMZgs8UXwixJf+V4mfZ79MzqpLZbfkxaE379_PA@mail.gmail.com>
	<99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu>
	<CAMYG4GnzDm4JEw8xijwt72gHOr1z6vPgeS2DgBNhV_jeTXXmMQ@mail.gmail.com>
	<CAKnkmZwMUzqEX=y+DQU=L-M8M8bKYjBCThA1otyX_Ayf=Dq_3g@mail.gmail.com>
	<CAMYG4Gnrsh6PQmkkGyh_4VMozNKoi=E_usX_-a1pg2=6CYYjiw@mail.gmail.com>
	<CAKnkmZzHGyZBGXoDtAYtQbwCrEUDJ26Vv7Dd0k7uCW=60GLnCQ@mail.gmail.com>
	<CAMYG4G=YNBh+nCKM_BpYLRjWFTK6sn+7GUS6Wg0dGVz8fr4S2g@mail.gmail.com>
	<CAKnkmZxbcLaoVPP9akk+jsLO8x3HHKiwESRBkW3VUnFy8Q86+g@mail.gmail.com>
	<CAKnkmZz7bLTik=csTAiv1crWYWRvtPmco3PROTK9DkLRpSxsMw@mail.gmail.com>
	<CAMYG4G=M+ZLuO0EUds_2qe83K7hLPG_+1wNLeo7LFeGCVU09jA@mail.gmail.com>
	<CAKnkmZzzWFZOvJkJeNZw53u7dMFV53MXnMOpPUNyED2dfo85Zg@mail.gmail.com>
	<CAMYG4G=6UZvB9rnYRqDHEGFSEOsWe-uCq92yAPTKLbOdk45qmQ@mail.gmail.com>
	<CAMYG4G=+dETft0b4GTmizBCKn_GpeN1yCDc5=emAfWV7W7Qmqw@mail.gmail.com>
	<CAKnkmZzEOBHDihdpCX5-ZtaMCnANH82kO7wXDgnCAw=dE_Wpiw@mail.gmail.com>
	<CAMYG4GnH2WSJdExhFsQ=BPp9p2MfnTMGn0b1p+2g6vB83zxfXQ@mail.gmail.com>
	<CAMYG4Gnkt9z180P82=D9keP3k=AZOmqjjc34T0s98uzwULYiCg@mail.gmail.com>
	<CAKnkmZxW9vTcFcXDzOXt39TRdPGk51_naG5mWiPvjvuGdH5_6A@mail.gmail.com>
Message-ID: <CAMYG4Gn37aBaO-VQE2HSHaiBf9AB=pRMcmU_YBPJjUMN7+9fiA@mail.gmail.com>

On Wed, Apr 25, 2012 at 4:38 PM, David Fuentes <fuentesdt at gmail.com> wrote:

> thanks! I seem to be getting the same answer on cpu and gpu in 3d w/
> double.
> However,  increasing the dof with -refinement_limit 0.01, the residual is
> differing slightly (see below).
>
> in single precision, do you have an idea of the upper bound limit on
> degrees of freedom
> such that the sieve gpu FEM solver would provide reasonable accuracy in
> the solution
> (using gpu for assembly of both residual and jacobain with single
> precision solve ) ?
> a friend was showing me some plots indicating that floating point error
> would become significant around O(10^4) dof on a related problem.
> would you expect similar ?
>

I guess it would depend of "significant". It can really only deliver 1e-7
residuals, but using single for FMM
we did 1e7 unknowns easily. I have not done anything big with FEM yet. The
eventual plan is to do loe fidelity
for everything but the last residual check. This is easy now since we can
make different GPU kernels.

   Matt


> SCRGP2$ hg diff
> diff -r be5f9068076c config/PETSc/FEM.py
> --- a/config/PETSc/FEM.py       Wed Apr 25 13:40:46 2012 -0500
> +++ b/config/PETSc/FEM.py       Wed Apr 25 14:58:40 2012 -0500
> @@ -10,7 +10,7 @@
>      import os
>      self.baseDir    = os.getcwd()
>      self.quadDegree = -1
> -    self.gpuScalarType = 'float'
> +    self.gpuScalarType = 'double'
>      return
>
>
> SCRGP2$ make ex52
> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
> -I/opt/apps/PETSC/petsc-dev/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g   -fPIC
>  -I/opt/apps/PETSC/petsc-dev/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>  ex52_integrateElement.cu
> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(14): warning: variable "weights_0" was declared but
> never referenced
>
> ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(30): warning: variable "BasisDerivatives_0" was declared
> but never referenced
>
> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(14): warning: variable "weights_0" was declared but
> never referenced
>
> ex52_gpu_inline.h(22): warning: variable "Basis_0" was declared but never
> referenced
>
> ex52_gpu_inline.h(30): warning: variable "BasisDerivatives_0" was declared
> but never referenced
>
> /opt/epd-7.1-2-rh5-x86_64/bin/python
> /opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py 3
> 1 1 1 laplacian ex52.h
> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
> [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
> Perm: [0, 1, 2, 3]
> Creating /home/fuentes/tutorials/ex52.h
> Creating /home/fuentes/tutorials/ex52_gpu.h
> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4:
> [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
> Perm: [0, 1, 2, 3]
> Creating /home/fuentes/tutorials/ex52_gpu_inline.h
> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
> -lcufft -lcublas -lcudart -lcusparse
> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
> /bin/rm -f ex52.o ex52_integrateElement.o
>
>
> SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
> -refinement_limit 0.0 -section_view
> PetscSection
> Process 0:
>   (   0) dim  0 offset   0
>   (   1) dim  0 offset   0
>   (   2) dim  0 offset   0
>   (   3) dim  0 offset   0
>   (   4) dim  0 offset   0
>   (   5) dim  0 offset   0
>   (   6) dim  1 offset   0
>   (   7) dim  1 offset   1
>   (   8) dim  1 offset   2
>   (   9) dim  1 offset   3
>   (  10) dim  1 offset   4
>   (  11) dim  1 offset   5
>   (  12) dim  1 offset   6
>   (  13) dim  1 offset   7
> Residual:
> Vector Object: 1 MPI processes
>   type: seq
> -0.666667
> 0.166667
> 0.166667
> 0.333333
> -0.333333
> -0.166667
> -0.166667
> 0.666667
>
>
> SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
> -refinement_limit 0.0 -section_view -gpu
> PetscSection
> Process 0:
>   (   0) dim  0 offset   0
>   (   1) dim  0 offset   0
>   (   2) dim  0 offset   0
>   (   3) dim  0 offset   0
>   (   4) dim  0 offset   0
>   (   5) dim  0 offset   0
>   (   6) dim  1 offset   0
>   (   7) dim  1 offset   1
>   (   8) dim  1 offset   2
>   (   9) dim  1 offset   3
>   (  10) dim  1 offset   4
>   (  11) dim  1 offset   5
>   (  12) dim  1 offset   6
>   (  13) dim  1 offset   7
> GPU layout grid(1,1,1) block(4,1,1) with 1 batches
>  N_t: 4, N_cb: 1
> Residual:
> Vector Object: 1 MPI processes
>   type: seq
> -0.666667
> 0.166667
> 0.166667
> 0.333333
> -0.333333
> -0.166667
> -0.166667
> 0.666667
>
>
> SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
> -refinement_limit 0.01
> Residual:
> Vector Object: 1 MPI processes
>   type: seq
> -0.0833333
> 0.0833333
> 0.0507812
> 0.11849
> -0.0833333
> 0.0833333
> 0.40743
> 0.428263
> 0.0598958
> 0.988459
> 1.10304
> 0.138428
> -0.0833333
> -0.125
> -0.125
> 0.0833333
> -0.10026
> 0.0507812
> -0.0803223
> 0.0446777
> -0.25
> -0.117187
> 2.06367
> 1.51159
> -0.181424
> -0.418944
> 0.0195312
> 0.0282118
> -0.574219
> 0.368924
> -0.474826
> 0.0195312
> -0.0238715
> 0.0195312
> -0.0792359
> -1.49715
> 1.10186
> 0.923154
> -1.31844
> -2.03658
> 0.0797244
> 0.0884049
> -2.21192
>
>
>
> SCRGP2$ ./ex52 -dim 3 -compute_function  -show_residual -batch
> -refinement_limit 0.01 -gpu
> GPU layout grid(3,7,1) block(4,1,1) with 1 batches
>  N_t: 4, N_cb: 1
> Residual:
> Vector Object: 1 MPI processes
>   type: seq
> -0.0833333
> 0.0833333
> 0.0507812
> 0.11849
> -0.0833333
> 0.0833333
> 0.0576986
> 0.0785319
> 0.0598958
> 0.0832038
> 0.197787
> 0.0932617
> -0.0833333
> -0.125
> -0.125
> 0.0833333
> -0.10026
> 0.0507812
> -0.0803223
> 0.0446777
> -0.25
> -0.117187
> 0.345882
> -0.206202
> -0.181424
> 0.101074
> 0.0195312
> 0.0282118
> 0.0859375
> 0.148872
> -0.0347222
> 0.0195312
> -0.0238715
> 0.0195312
> -0.588018
> -0.117594
> 0.12749
> 0.13118
> -0.121284
> 0.0692962
> 0.118305
> 0.126986
> -0.106051
>
>
>
>
> On Wed, Apr 25, 2012 at 1:27 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Wed, Apr 25, 2012 at 2:06 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Apr 25, 2012 at 1:50 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>
>>>> I'm pretty sure I pulled the latest on this petsc-dev and Generator,
>>>> but when i change GPU type to PetscReal or double, i'm getting key errors ?
>>>>
>>>
>>> Ah well, I was getting around to fixing this. The problem is a little
>>> deeper.  The C version uses pointers everywhere to be
>>> general, but I was using the CUDA types since I thought the compiler
>>> would be more aggressive even though it is
>>> limiting. However, despite having float1, float2, float3, and float4,
>>> CUDA provides only double2. I do all my tests in float.
>>> I will fix this after the baby goes down and mail back.
>>>
>>
>> Okay, this is bizarre. It builds with double3 (not in the CUDA manual).
>> You will need to pull both petsc and Generator, but
>> it should work now. I have not really tested with double, so this will be
>> interesting.
>>
>>    Matt
>>
>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>> hg diff
>>>> diff -r 4a642f9b7b70 config/PETSc/FEM.py
>>>> --- a/config/PETSc/FEM.py       Wed Apr 25 02:13:29 2012 -0500
>>>> +++ b/config/PETSc/FEM.py       Wed Apr 25 12:39:03 2012 -0500
>>>> @@ -434,7 +434,7 @@
>>>>      from Cxx import Define, Declarator
>>>>      dim  = element.get_reference_element().get_spatial_dimension()
>>>>      ext  = '_'+str(num)
>>>> -    real = 'float'
>>>> +    real = 'PetscReal'
>>>>
>>>>
>>>> SCRGP2$ python
>>>> $PETSC_DIR//bin/pythonscripts/PetscGenerateFEMQuadrature.py 3 1 1 1 laplace
>>>> ex52.h
>>>> [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]},
>>>> {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}]
>>>> {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [],
>>>> 4: [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}}
>>>> Perm: [0, 1, 2, 3]
>>>> Creating /home/fuentes/tutorials/ex52.h
>>>> Traceback (most recent call last):
>>>>   File
>>>> "/opt/apps/PETSC/petsc-dev//bin/pythonscripts/PetscGenerateFEMQuadrature.py",
>>>> line 31, in <module>
>>>>     generator.run(elements, numBlocks, operator, filename)
>>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 977, in run
>>>>     self.outputElementSource(self.getElementSource(elements, numBlocks,
>>>> operator, sourceType = 'GPU'),
>>>> os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1])
>>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 936, in
>>>> getElementSource
>>>>     defns.extend(self.getComputationTypes(element, n))
>>>>   File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 450, in
>>>> getComputationTypes
>>>>     vecType.type       = self.Cxx.typeMap[real+str(dim)]
>>>> KeyError: 'PetscReal3'
>>>>
>>>>
>>>> do I need to add something like below ?
>>>>
>>>>
>>>> cd externalpackages/Generator/; hg diff
>>>> diff -r 361a2eec9c01 CxxHelper.py
>>>> --- a/CxxHelper.py      Tue Mar 27 09:01:27 2012 -0500
>>>> +++ b/CxxHelper.py      Wed Apr 25 12:39:12 2012 -0500
>>>> @@ -98,6 +98,11 @@
>>>>      cxxType.identifier = 'float'
>>>>      cxxType.baseType = True
>>>>      cxxType.const = True
>>>> +    typeMap['double3'] = cxxType
>>>> +    cxxType = Type()
>>>> +    cxxType.identifier = 'double'
>>>> +    cxxType.baseType = True
>>>> +    cxxType.const = True
>>>>      typeMap['const float1'] = cxxType
>>>>      cxxType = Type()
>>>>      cxxType.identifier = 'float2'
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 2, 2012 at 9:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>>>>>>
>>>>>>> Yes. This would work.
>>>>>>> I had trouble compiling in single precision using the some of the
>>>>>>> external package options I was using for double.
>>>>>>>
>>>>>>
>>>>>> Okay,  getting on it now.
>>>>>>
>>>>>
>>>>> Okay, I have tested using PetscReal=double with floats on the GPU. If
>>>>> you want doubles on the GPU, you can
>>>>> currently change PETSc/FEM.py:434. I am deciding what the best way to
>>>>> specify this type is.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> works!
>>>>>>>>>
>>>>>>>>
>>>>>>>> Excellent. Now, my thinking was that GPUs are most useful doing
>>>>>>>> single work, but
>>>>>>>> I can see the utility of double accuracy for a residual.
>>>>>>>>
>>>>>>>> My inclination is to define another type, say GPUReal, and use it
>>>>>>>> for all kernels.
>>>>>>>> Would that do what you want?
>>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>> nvcc -G -O0 -g -arch=sm_10  -c --compiler-options="-O0 -g   -fPIC
>>>>>>>>>    -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>>>>>>>  -D__INSDIR__=src/snes/examples/tutorials/"  ex52_integrateElement.cu
>>>>>>>>>  ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>>>  -lpetsc
>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>>>>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>>>>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>   type: seq
>>>>>>>>> -0.25
>>>>>>>>> -0.5
>>>>>>>>> 0.25
>>>>>>>>> -0.5
>>>>>>>>> -1
>>>>>>>>> 0.5
>>>>>>>>> 0.25
>>>>>>>>> 0.5
>>>>>>>>> 0.75
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>   type: seq
>>>>>>>>> -0.25
>>>>>>>>> -0.5
>>>>>>>>> 0.25
>>>>>>>>> -0.5
>>>>>>>>> -1
>>>>>>>>> 0.5
>>>>>>>>> 0.25
>>>>>>>>> 0.5
>>>>>>>>> 0.75
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <
>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> sure. will do.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <
>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> thanks! its running, but I seem to be getting different answer
>>>>>>>>>>>> for cpu/gpu ?
>>>>>>>>>>>> i had some floating point problems on this Tesla M2070 gpu
>>>>>>>>>>>> before, but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yep, that is the problem. I have not tested anything in double.
>>>>>>>>>>> I have not decided exactly how to handle it. Can you
>>>>>>>>>>> make another ARCH --with-precision=single and make sure it
>>>>>>>>>>> works, and then we can fix the double issue?
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>> -gpu
>>>>>>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>>>>>>>  N_t: 3, N_cb: 1
>>>>>>>>>>>> Residual:
>>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>>   type: seq
>>>>>>>>>>>> 0
>>>>>>>>>>>> 755712
>>>>>>>>>>>> 0
>>>>>>>>>>>> -58720
>>>>>>>>>>>> -2953.13
>>>>>>>>>>>> 0.375
>>>>>>>>>>>> 1.50323e+07
>>>>>>>>>>>> 0.875
>>>>>>>>>>>> 0
>>>>>>>>>>>> SCRGP2$
>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>> Residual:
>>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>>   type: seq
>>>>>>>>>>>> -0.25
>>>>>>>>>>>> -0.5
>>>>>>>>>>>> 0.25
>>>>>>>>>>>> -0.5
>>>>>>>>>>>> -1
>>>>>>>>>>>> 0.5
>>>>>>>>>>>> 0.25
>>>>>>>>>>>> 0.5
>>>>>>>>>>>> 0.75
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> No.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -  ierr = cudaMalloc((void**) &d_coefficients,
>>>>>>>>>>>>>> Ne*N_bt * sizeof(float));CHKERRQ(ierr);
>>>>>>>>>>>>>> +  ierr = cudaMalloc((void**) &d_coefficients,
>>>>>>>>>>>>>> Ne*N_bt * sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SCRGP2$ python
>>>>>>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>>>>>>> laplacian ex52.h
>>>>>>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2:
>>>>>>>>>>>>>> {0: []}}
>>>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]},
>>>>>>>>>>>>>> {(-1.0, 1.0): [(1.0, ())]}]
>>>>>>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2:
>>>>>>>>>>>>>> {0: []}}
>>>>>>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>>>>>>> nvcc -O0 -g -arch=sm_20  -c --compiler-options="-O0 -g
>>>>>>>>>>>>>> -fPIC    -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>>>>>>> -I/usr/include/mpich2    -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>>>>>>>  ex52_integrateElement.cu
>>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was
>>>>>>>>>>>>>> declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0"
>>>>>>>>>>>>>> was declared but never referenced
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /usr/bin/mpicxx -O0 -g   -o ex52 ex52.o
>>>>>>>>>>>>>> ex52_integrateElement.o
>>>>>>>>>>>>>>  -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib  -lpetsc
>>>>>>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps
>>>>>>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>>>>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>>>> Residual:
>>>>>>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>>>>>>>   type: seq
>>>>>>>>>>>>>> -0.25
>>>>>>>>>>>>>> -0.5
>>>>>>>>>>>>>> 0.25
>>>>>>>>>>>>>> -0.5
>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>> 0.5
>>>>>>>>>>>>>> 0.25
>>>>>>>>>>>>>> 0.5
>>>>>>>>>>>>>> 0.75
>>>>>>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>>>>>>> -gpu
>>>>>>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>>>>>>> If so, applications like Preview can lock up the GPU. I know
>>>>>>>>>>>>> of no way to test this in CUDA while running. I just close
>>>>>>>>>>>>> apps until it runs.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <
>>>>>>>>>>>>>>> bourdin at lsu.edu> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done
>>>>>>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be
>>>>>>>>>>>>>>>> run using the Python build system
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   ./config/builder2.py check
>>>>>>>>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   in fact, you can build any set of files this way.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex
>>>>>>>>>>>>>>>> from DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>>>>>>>      so maybe there will be help :) You will just replace
>>>>>>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request
>>>>>>>>>>>>>>>>      it, I will bump it up the list.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element
>>>>>>>>>>>>>>>> types and should have a much lower memory footprint. The code should be
>>>>>>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too
>>>>>>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the
>>>>>>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one
>>>>>>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> David and Blaise,  I have converted this function, now
>>>>>>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Blaise
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   Thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  --
>>>>>>>>>>>>>>>> Department of Mathematics and Center for Computation &
>>>>>>>>>>>>>>>> Technology
>>>>>>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
>>>>>>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120425/3d2e8259/attachment-0001.htm>

From dominik at itis.ethz.ch  Thu Apr 26 06:28:10 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Thu, 26 Apr 2012 13:28:10 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
Message-ID: <CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>

Jed, Barry, Matt:

Below a collection of all pending answers in one mail.

> How do you define "right results"?

I run the exact same problem on in a commercial FEM package and
compare the results. The test problem is diffusion-reaction, with a
nonlinear reaction term of a polynomial form (tested up to order 5). I
Form the Jacobian as follows. We have a problem Ax = b(x), with A
constant and b depending pointwise on x (i.e. db/dxn does not depend
on xm, m!=n). Therefore I provide a simple Jacobian equal to A with
the diagonal augmented with -db/dx. The relevant code is below. I do
hope I do something wrong and you can point it out.

	ierr = MatCopy(that->system.A, *B, SAME_NONZERO_PATTERN); CHKERRQ(ierr);

	ierr = VecScale(that->dbdx, -1.0); CHKERRQ(ierr);
	ierr = MatDiagonalSet(*B, that->dbdx, ADD_VALUES); CHKERRQ(ierr);

	ierr = MatAssemblyBegin(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
	ierr = MatAssemblyEnd(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);

	if(*J != *B)
	{
		ierr = MatAssemblyBegin(*J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
		ierr = MatAssemblyEnd(*J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
	}

	*flag = SAME_NONZERO_PATTERN;


> Is SNES converging quadratically when you
> do not use -snes_mf_operator?

Without -snes_mf_operator, providing analytically derived Jacobian:

  0 SNES Function norm 1.747536282885e+01
  1 SNES Function norm 1.918464419679e-03
  2 SNES Function norm 1.086746779744e-03
  3 SNES Function norm 2.904556511511e-06
  4 SNES Function norm 7.804808098453e-08

Without -snes_mf_operator, passing A as the jacobian (which you call
Picard), and slightly underrelaxing the linear solves (otherwise it
would not converge):

  0 SNES Function norm 1.747886139999e+01
  1 SNES Function norm 3.495796117011e+00
  2 SNES Function norm 6.991806248330e-01
  3 SNES Function norm 1.398879936054e-01
  4 SNES Function norm 2.809047018752e-02
  5 SNES Function norm 5.843745932752e-03
  6 SNES Function norm 1.594583643926e-03
  7 SNES Function norm 7.712066092135e-04
  8 SNES Function norm 4.932447251451e-04
  9 SNES Function norm 3.217945635817e-04
 10 SNES Function norm 2.129867244811e-04
 11 SNES Function norm 1.399983127264e-04
 12 SNES Function norm 9.246882776024e-05
 13 SNES Function norm 6.088284618615e-05
 14 SNES Function norm 4.017029247381e-05
 15 SNES Function norm 2.646770266042e-05
 16 SNES Function norm 1.745497080382e-05
 17 SNES Function norm 1.150437737331e-05
 18 SNES Function norm 7.585434690336e-06
 19 SNES Function norm 5.000147553531e-06
 20 SNES Function norm 3.296560754852e-06
 21 SNES Function norm 2.173152296533e-06
 22 SNES Function norm 1.432685503660e-06
 23 SNES Function norm 9.444717211995e-07
 24 SNES Function norm 6.226464090397e-07
 25 SNES Function norm 4.104757979681e-07
 26 SNES Function norm 2.706068836265e-07
 27 SNES Function norm 1.783963930887e-07
 28 SNES Function norm 1.176079057620e-07

Dominik

> Run with -snes_type test and see what it reports.

Running with the options:

-ksp_type bcgs -pc_type jacobi -ksp_rtol 1e-4 -ksp_max_it 1000
-ksp_converged_use_initial_residual_norm -ksp_norm_type
unpreconditioned -ksp_monitor_true_residual -ksp_converged_reason
-ksp_view -snes_monitor -snes_converged_reason -snes_ls_monitor
-snes_view -snes_type test -snes_test_display

I get the output:

Testing hand-coded Jacobian, if the ratio is
O(1.e-8), the hand-coded Jacobian is probably correct.

and the procedure stagnates, calling FormJacobian only once, and then
FormFunction infinitely, with my own computation of ||F|| indicating
stagnation.

Regards,
Dominik

From zonexo at gmail.com  Thu Apr 26 07:27:09 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Thu, 26 Apr 2012 14:27:09 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F97E8F2.5050803@gmail.com>
References: <4F97E8F2.5050803@gmail.com>
Message-ID: <4F993F1D.9020105@gmail.com>

Hi,

Is there an answer to the question below?
  I still can't get it working.

Thanks.


Hi,

There's no error now but I didn't get the same ans as ex29. I believe 
the error lies in the boundary evaluation of v.

In c, the code is :

else if (user->bcType == NEUMANN) {
           num = 0;
           if (j!=0) {
             v[num] = -rho*HxdHy;              col[num].i = i;   
col[num].j = j-1;
             num++;
           }
           if (i!=0) {
             v[num] = -rho*HydHx;              col[num].i = i-1; 
col[num].j = j;
             num++;
           }
           if (i!=mx-1) {
             v[num] = -rho*HydHx;              col[num].i = i+1; 
col[num].j = j;
             num++;
           }
           if (j!=my-1) {
             v[num] = -rho*HxdHy;              col[num].i = i;   
col[num].j = j+1;
             num++;
           }
           v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;   
col[num].j = j;
           num++;
           ierr = 
MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);

In fortran, I have changed to :

else if (bc_cond == 2) then

                 num = 1

                 if (j/=0) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,1) = i

                     col(MatStencil_j,2) = j-1

                     num = num + 1

                 end if

                 if (i/=0) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,1) = i-1

                     col(MatStencil_j,2) = j

                     num = num + 1

                 end if

                 if (i/=mx-1) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,1) = i+1

                     col(MatStencil_j,2) = j

                     num = num + 1

                 end if

                 if (j/=my-1) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,1) = i

                     col(MatStencil_j,2) = j+1

                     num = num + 1

                 end if

                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)

                 col(MatStencil_i,1) = i

                 col(MatStencil_j,2) = j+1

                 num = num + 1

                 call 
MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)

However, I got the error:

[0]PETSC ERROR: --------------------- Error Message 
---------------------------
--------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
[0]PETSC ERROR: 
---------------------------------------------------------------

How such is be defined in Fortran?

Thank you

Yours sincerely,

TAY wee-beng


On 25/4/2012 12:06 AM, Matthew Knepley wrote:
> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I still got the same error the moment this subroutine is called,
>     after changing to :
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>
>
> Argument 3 should be 0.
>
>   Matt
>
>     [0]PETSC ERROR: Null argument, when expecting valid pointer!
>     [0]PETSC ERROR: Null Object: Parameter # 4!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     fc934c1d84bc6ba8e2686702a8a99539d
>     50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2
>     named USER-PC by
>      User Tue Apr 24 22:54:50 2012
>     [0]PETSC ERROR: Libraries linked from
>     /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     /lib
>     [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>     [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>     --with-fc="win32fe ifor
>     t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>     --download-f-b
>     las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>     --with-debuggin
>     g=1 --useThreads=0
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>     --------
>     [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>     src/mat/interface/C:\temp\PETSC
>     -~1\src\mat\INTERF~1\matnull.c
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>     On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to
>>         fortran version. The fortran version is based on the template
>>         of ex22f, which is in 3d.
>>
>>         I have attached the code below. I have 2 problems.
>>
>>         1. When i3 = -3, in dirichlet BC, I got the same answer as
>>         ex29 in c version.
>>
>>         However, when I change i3 to -4, I got the following error:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Argument out of range!
>>         [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:08:46 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>         src/mat/impls/aij/seq/C:\temp\
>>         PETSC-~1\src\mat\impls\aij\seq\aij.c
>>         [0]PETSC ERROR: MatSetValues() line 1148 in
>>         src/mat/interface/C:\temp\PETSC-~1\s
>>         rc\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>         src/mat/interface/C:\temp\PET
>>         SC-~1\src\mat\INTERF~1\matrix.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Argument out of range!
>>         [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>
>>         ...
>>
>>         What did I do wrong?
>>
>>
>>         2. When I wanted to use the neumann BC, following ex29, I added :
>>
>>         In subroutine ComputeRHS
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>         call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         and in subroutine ComputeMatrix
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         call MatSetNullSpace(jac,nullspace,ierr)
>>
>>         call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         When I compile, it says unresolved external symbol
>>         MatNullSpaceRemove. Removing MatNullSpaceRemove, I can
>>         compile but running gives the error:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Corrupt argument:
>>         see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>         [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Invalid argument!
>>         [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>         src/mat/interface/C:\temp\PETS
>>         C-~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: ourdmfunction() line 30 in
>>         src/dm/interface/ftn-custom/C:\temp\P
>>         ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>         [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>         src/dm/interface/C:\temp\PETSC-
>>         ~1\src\dm\INTERF~1\dm.c
>>         [0]PETSC ERROR: KSPSetUp() line 218 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Corrupt argument:
>>         see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>         [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ----------------------------
>>         --------
>>         [0]PETSC ERROR: Invalid argument!
>>         [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:30:31 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>         src/mat/interface/C:\temp\PETS
>>         C-~1\src\mat\INTERF~1\matnull.c
>>         [0]PETSC ERROR: ourdmfunction() line 30 in
>>         src/dm/interface/ftn-custom/C:\temp\P
>>         ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>         [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>         src/dm/interface/C:\temp\PETSC-
>>         ~1\src\dm\INTERF~1\dm.c
>>         [0]PETSC ERROR: KSPSetUp() line 218 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         [0]PETSC ERROR: KSPSolve() line 402 in
>>         src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>         c\ksp\ksp\INTERF~1\itfunc.c
>>         Vector Object:Vec_0000000084000000_0 1 MPI processes
>>           type: mpi
>>         Process [0]
>>
>>         What's wrong?
>>
>>
>>         *program ex29f*
>>
>>         implicit none
>>
>>         ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>         - - - - - -
>>         !                    Include files
>>         ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>         - - - - - -
>>         !
>>         !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>         !     petscmat.h - matrices
>>         !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>>         preconditioners
>>
>>         #include "finclude/petsc.h90"
>>
>>         PetscErrorCode   ierr
>>
>>         DM               da
>>
>>         KSP              ksp
>>
>>         Vec              x
>>
>>         external         ComputeRHS,ComputeMatrix
>>
>>         PetscInt i1,i3
>>
>>         call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>
>>         i3 = -4
>>
>>         i1 = 1
>>
>>         call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>
>>         call
>>         DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>         call DMSetFunction(da,ComputeRHS,ierr)
>>         call DMSetJacobian(da,ComputeMatrix,ierr)
>>         call KSPSetDM(ksp,da,ierr)
>>
>>         call KSPSetFromOptions(ksp,ierr)
>>         call KSPSetUp(ksp,ierr)
>>         call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>         call KSPGetSolution(ksp,x,ierr)
>>         call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>         call KSPDestroy(ksp,ierr)
>>         call DMDestroy(da,ierr)
>>         call PetscFinalize(ierr)
>>
>>         end program ex29f
>>
>>         subroutine ComputeRHS(da,x,b,ierr)
>>
>>         implicit none
>>
>>         #include "finclude/petsc.h90"
>>
>>         PetscErrorCode  ierr
>>         PetscInt i,j,mx,my,xm,ym,xs,ys
>>         PetscScalar  h,nu,rho
>>         PetscScalar Hx,Hy
>>         PetscScalar,pointer :: array(:,:)
>>         Vec          x,b
>>         DM           da
>>         MatNullSpace nullspace    !>neumann BC
>>
>>         nu = 0.1
>>
>>         call
>>         DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>>         Hx = 1.d0 / (mx-1)
>>
>>         Hy = 1.d0 / (my-1)
>>
>>         call
>>         DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>         call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>>         do j = ys,ys+ym-1
>>
>>             do i = xs,xs+xm-1
>>
>>                 array(i,j) =
>>         exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>             end do
>>
>>         end do
>>
>>         call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>>         call VecAssemblyBegin(b,ierr)
>>
>>         call VecAssemblyEnd(b,ierr)
>>
>>         !call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>
>>     4th argument should be PETSC_NULL_OBJECT.
>>
>>        Matt
>>
>>         !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>         !call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         end subroutine ComputeRHS
>>
>>
>>         subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>
>>         implicit none
>>
>>         #include "finclude/petsc.h90"
>>
>>         Mat          jac,JJ
>>
>>         PetscErrorCode    ierr
>>
>>         DM           da
>>
>>         PetscInt    i,j,k,mx,my,xm
>>
>>         PetscInt    ym,xs,ys,i1,i5
>>
>>         PetscScalar  v(5),Hx,Hy,rho,centerRho
>>
>>         PetscScalar  HydHx
>>
>>         PetscScalar  HxdHy
>>
>>         MatStencil   row(4),col(4,5)
>>
>>         Vec          x
>>
>>         MatStructure str
>>
>>         MatNullSpace nullspace    !>neumann BC
>>
>>         rho = 1.0
>>
>>         i1 = 1
>>
>>         i5 = 5
>>
>>         centerRho = rho
>>
>>         call
>>         DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>
>>         Hx = 1.d0 / (mx-1)
>>
>>         Hy = 1.d0 / (my-1)
>>
>>         HxdHy = Hx/Hy
>>
>>         HydHx = Hy/Hx
>>
>>         call
>>         DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>
>>         do j=ys,ys+ym-1
>>
>>             do i=xs,xs+xm-1
>>
>>                 row(MatStencil_i) = i
>>
>>                 row(MatStencil_j) = j
>>
>>                 call ComputeRho(i,j,mx,my,centerRho,rho)
>>
>>                 if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1)
>>         then
>>
>>                     v(1) = 2.0*rho*(HxdHy + HydHx)
>>
>>                     call
>>         MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>
>>                 else
>>
>>                     v(1) = -rho*HxdHy
>>
>>                     col(MatStencil_i,1) = i
>>
>>                     col(MatStencil_j,2) = j-1
>>
>>
>>     Cut and paste error above.
>>
>>       Matt
>>
>>                     v(2) = -rho*HydHx
>>
>>                     col(MatStencil_i,2) = i-1
>>
>>                     col(MatStencil_j,2) = j
>>
>>                     v(3) = 2.0*rho*(HxdHy + HydHx)
>>
>>                     col(MatStencil_i,3) = i
>>
>>                     col(MatStencil_j,3) = j
>>
>>                     v(4) = -rho*HydHx
>>
>>                     col(MatStencil_i,4) = i+1
>>
>>                     col(MatStencil_j,4) = j
>>
>>                     v(5) = -rho*HxdHy
>>
>>                     col(MatStencil_i,5) = i
>>
>>                     col(MatStencil_j,5) = j+1
>>
>>                     call
>>         MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>
>>                 end if
>>
>>             end do
>>
>>         end do
>>
>>         call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>>         call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>
>>         !call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>
>>         !call MatSetNullSpace(jac,nullspace,ierr)
>>
>>         !call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         end subroutine ComputeMatrix
>>
>>         subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>
>>         PetscInt    i,j,mx,my
>>
>>         PetscScalar  rho,centerRho
>>
>>         if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0)
>>         .and. (j < 2.0*my/3.0)) then
>>
>>             rho = centerRho
>>
>>         else
>>
>>             rho = 1.0
>>
>>         end if
>>
>>
>>         end subroutine ComputeRho
>>
>>         -- 
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/59477d4e/attachment-0001.htm>

From knepley at gmail.com  Thu Apr 26 07:45:22 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 08:45:22 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F993F1D.9020105@gmail.com>
References: <4F97E8F2.5050803@gmail.com>
	<4F993F1D.9020105@gmail.com>
Message-ID: <CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>

On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> Is there an answer to the question below?
>  I still can't get it working.
>
> Thanks.
>
>
> Hi,
>
> There's no error now but I didn't get the same ans as ex29. I believe the
> error lies in the boundary evaluation of v.
>
> In c, the code is :
>
> else if (user->bcType == NEUMANN) {
>           num = 0;
>           if (j!=0) {
>             v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j
> = j-1;
>
                                                                  ^^^ See
this?

>             num++;
>           }
>           if (i!=0) {
>             v[num] = -rho*HydHx;              col[num].i = i-1; col[num].j
> = j;
>             num++;
>           }
>           if (i!=mx-1) {
>             v[num] = -rho*HydHx;              col[num].i = i+1; col[num].j
> = j;
>             num++;
>           }
>           if (j!=my-1) {
>             v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j
> = j+1;
>             num++;
>           }
>           v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;
> col[num].j = j;
>           num++;
>           ierr =
> MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>
> In fortran, I have changed to :
>
> else if (bc_cond == 2) then
>
>                 num = 1
>
>                 if (j/=0) then
>
>                     v(num) = -rho*HxdHy
>
>                     col(MatStencil_i,1) = i
>
                                               ^^ Why is this 1 in stead of
'num'?

   Matt


                    col(MatStencil_j,2) = j-1
>
>                     num = num + 1
>
>                 end if
>
>                 if (i/=0) then
>
>                     v(num) = -rho*HydHx
>
>                     col(MatStencil_i,1) = i-1
>
>                     col(MatStencil_j,2) = j
>
>                     num = num + 1
>
>                 end if
>
>                 if (i/=mx-1) then
>
>                     v(num) = -rho*HydHx
>
>                     col(MatStencil_i,1) = i+1
>
>                     col(MatStencil_j,2) = j
>
>                     num = num + 1
>
>                 end if
>
>                 if (j/=my-1) then
>
>                     v(num) = -rho*HxdHy
>
>                     col(MatStencil_i,1) = i
>
>                     col(MatStencil_j,2) = j+1
>
>                     num = num + 1
>
>                 end if
>
>                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>
>                 col(MatStencil_i,1) = i
>
>                 col(MatStencil_j,2) = j+1
>
>                 num = num + 1
>
>                 call
> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
> However, I got the error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ---------------------------
> --------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
> [0]PETSC ERROR:
> ---------------------------------------------------------------
>
> How such is be defined in Fortran?
>
> Thank you
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>
> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I still got the same error the moment this subroutine is called, after
>> changing to :
>>
>> call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>
>  Argument 3 should be 0.
>
>    Matt
>
>
>>  [0]PETSC ERROR: Null argument, when expecting valid pointer!
>> [0]PETSC ERROR: Null Object: Parameter # 4!
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> fc934c1d84bc6ba8e2686702a8a99539d
>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>> USER-PC by
>>  User Tue Apr 24 22:54:50 2012
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> /lib
>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>> --with-fc="win32fe ifor
>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>> --download-f-b
>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>> --with-debuggin
>> g=1 --useThreads=0
>> [0]PETSC ERROR:
>> ----------------------------------------------------------------
>> --------
>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>> src/mat/interface/C:\temp\PETSC
>> -~1\src\mat\INTERF~1\matnull.c
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>
>> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran
>>> version. The fortran version is based on the template of ex22f, which is in
>>> 3d.
>>>
>>> I have attached the code below. I have 2 problems.
>>>
>>> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c
>>> version.
>>>
>>> However, when I change i3 to -4, I got the following error:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:08:46 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>> src/mat/impls/aij/seq/C:\temp\
>>> PETSC-~1\src\mat\impls\aij\seq\aij.c
>>> [0]PETSC ERROR: MatSetValues() line 1148 in
>>> src/mat/interface/C:\temp\PETSC-~1\s
>>> rc\mat\INTERF~1\matrix.c
>>> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>> src/mat/interface/C:\temp\PETSC
>>> -~1\src\mat\INTERF~1\matrix.c
>>> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>> src/mat/interface/C:\temp\PET
>>> SC-~1\src\mat\INTERF~1\matrix.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>
>>> ...
>>>
>>> What did I do wrong?
>>>
>>>
>>> 2. When I wanted to use the neumann BC, following ex29, I added :
>>>
>>> In subroutine ComputeRHS
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> and in subroutine ComputeMatrix
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>> call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> When I compile, it says unresolved external symbol MatNullSpaceRemove.
>>> Removing MatNullSpaceRemove, I can compile but running gives the error:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Corrupt argument:
>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:30:31 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>> src/mat/interface/C:\temp\PETSC
>>> -~1\src\mat\INTERF~1\matnull.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Invalid argument!
>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:30:31 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>> src/mat/interface/C:\temp\PETS
>>> C-~1\src\mat\INTERF~1\matnull.c
>>> [0]PETSC ERROR: ourdmfunction() line 30 in
>>> src/dm/interface/ftn-custom/C:\temp\P
>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>> src/dm/interface/C:\temp\PETSC-
>>> ~1\src\dm\INTERF~1\dm.c
>>> [0]PETSC ERROR: KSPSetUp() line 218 in
>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>> c\ksp\ksp\INTERF~1\itfunc.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Corrupt argument:
>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:30:31 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>> src/mat/interface/C:\temp\PETSC
>>> -~1\src\mat\INTERF~1\matnull.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ----------------------------
>>> --------
>>> [0]PETSC ERROR: Invalid argument!
>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:30:31 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>> src/mat/interface/C:\temp\PETS
>>> C-~1\src\mat\INTERF~1\matnull.c
>>> [0]PETSC ERROR: ourdmfunction() line 30 in
>>> src/dm/interface/ftn-custom/C:\temp\P
>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>> src/dm/interface/C:\temp\PETSC-
>>> ~1\src\dm\INTERF~1\dm.c
>>> [0]PETSC ERROR: KSPSetUp() line 218 in
>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>> c\ksp\ksp\INTERF~1\itfunc.c
>>> [0]PETSC ERROR: KSPSolve() line 402 in
>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>> c\ksp\ksp\INTERF~1\itfunc.c
>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>   type: mpi
>>> Process [0]
>>>
>>> What's wrong?
>>>
>>>
>>> *program ex29f*
>>>
>>> implicit none
>>>
>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>> !                    Include files
>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>> !
>>> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>> !     petscmat.h - matrices
>>> !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>>> preconditioners
>>>
>>> #include "finclude/petsc.h90"
>>>
>>> PetscErrorCode   ierr
>>>
>>> DM               da
>>>
>>> KSP              ksp
>>>
>>> Vec              x
>>>
>>> external         ComputeRHS,ComputeMatrix
>>>
>>> PetscInt i1,i3
>>>
>>> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>
>>> i3 = -4
>>>
>>> i1 = 1
>>>
>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>
>>> call
>>> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>> call DMSetFunction(da,ComputeRHS,ierr)
>>> call DMSetJacobian(da,ComputeMatrix,ierr)
>>> call KSPSetDM(ksp,da,ierr)
>>>
>>> call KSPSetFromOptions(ksp,ierr)
>>> call KSPSetUp(ksp,ierr)
>>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>> call KSPGetSolution(ksp,x,ierr)
>>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>> call KSPDestroy(ksp,ierr)
>>> call DMDestroy(da,ierr)
>>> call PetscFinalize(ierr)
>>>
>>> end program ex29f
>>>
>>> subroutine ComputeRHS(da,x,b,ierr)
>>>
>>> implicit none
>>>
>>> #include "finclude/petsc.h90"
>>>
>>> PetscErrorCode  ierr
>>> PetscInt i,j,mx,my,xm,ym,xs,ys
>>> PetscScalar  h,nu,rho
>>> PetscScalar Hx,Hy
>>> PetscScalar,pointer :: array(:,:)
>>> Vec          x,b
>>> DM           da
>>> MatNullSpace nullspace    !>neumann BC
>>>
>>> nu = 0.1
>>>
>>> call
>>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>
>>> Hx = 1.d0 / (mx-1)
>>>
>>> Hy = 1.d0 / (my-1)
>>>
>>> call
>>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>
>>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>>
>>> do j = ys,ys+ym-1
>>>
>>>     do i = xs,xs+xm-1
>>>
>>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>
>>>     end do
>>>
>>> end do
>>>
>>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>
>>> call VecAssemblyBegin(b,ierr)
>>>
>>> call VecAssemblyEnd(b,ierr)
>>>
>>> !call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>
>>  4th argument should be PETSC_NULL_OBJECT.
>>
>>     Matt
>>
>>
>>>  !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> end subroutine ComputeRHS
>>>
>>>
>>> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>
>>> implicit none
>>>
>>> #include "finclude/petsc.h90"
>>>
>>> Mat          jac,JJ
>>>
>>> PetscErrorCode    ierr
>>>
>>> DM           da
>>>
>>> PetscInt    i,j,k,mx,my,xm
>>>
>>> PetscInt    ym,xs,ys,i1,i5
>>>
>>> PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>
>>> PetscScalar  HydHx
>>>
>>> PetscScalar  HxdHy
>>>
>>> MatStencil   row(4),col(4,5)
>>>
>>> Vec          x
>>>
>>> MatStructure str
>>>
>>> MatNullSpace nullspace    !>neumann BC
>>>
>>> rho = 1.0
>>>
>>> i1 = 1
>>>
>>> i5 = 5
>>>
>>> centerRho = rho
>>>
>>> call
>>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>
>>> Hx = 1.d0 / (mx-1)
>>>
>>> Hy = 1.d0 / (my-1)
>>>
>>> HxdHy = Hx/Hy
>>>
>>> HydHx = Hy/Hx
>>>
>>> call
>>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>
>>>
>>> do j=ys,ys+ym-1
>>>
>>>     do i=xs,xs+xm-1
>>>
>>>         row(MatStencil_i) = i
>>>
>>>         row(MatStencil_j) = j
>>>
>>>         call ComputeRho(i,j,mx,my,centerRho,rho)
>>>
>>>         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>>>
>>>             v(1) = 2.0*rho*(HxdHy + HydHx)
>>>
>>>             call
>>> MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>
>>>         else
>>>
>>>             v(1) = -rho*HxdHy
>>>
>>>             col(MatStencil_i,1) = i
>>>
>>>             col(MatStencil_j,2) = j-1
>>>
>>
>>  Cut and paste error above.
>>
>>    Matt
>>
>>
>>>              v(2) = -rho*HydHx
>>>
>>>             col(MatStencil_i,2) = i-1
>>>
>>>             col(MatStencil_j,2) = j
>>>
>>>             v(3) = 2.0*rho*(HxdHy + HydHx)
>>>
>>>             col(MatStencil_i,3) = i
>>>
>>>             col(MatStencil_j,3) = j
>>>
>>>             v(4) = -rho*HydHx
>>>
>>>             col(MatStencil_i,4) = i+1
>>>
>>>             col(MatStencil_j,4) = j
>>>
>>>             v(5) = -rho*HxdHy
>>>
>>>             col(MatStencil_i,5) = i
>>>
>>>             col(MatStencil_j,5) = j+1
>>>
>>>             call
>>> MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>
>>>         end if
>>>
>>>     end do
>>>
>>> end do
>>>
>>> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>
>>> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>
>>> !call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>> !call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> end subroutine ComputeMatrix
>>>
>>> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>
>>> PetscInt    i,j,mx,my
>>>
>>> PetscScalar  rho,centerRho
>>>
>>> if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j <
>>> 2.0*my/3.0)) then
>>>
>>>     rho = centerRho
>>>
>>> else
>>>
>>>     rho = 1.0
>>>
>>> end if
>>>
>>>
>>> end subroutine ComputeRho
>>>
>>> --
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/4a2ae495/attachment-0001.htm>

From bsmith at mcs.anl.gov  Thu Apr 26 08:24:52 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Apr 2012 08:24:52 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
Message-ID: <AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>


On Apr 26, 2012, at 6:28 AM, Dominik Szczerba wrote:

> Jed, Barry, Matt:
> 
> Below a collection of all pending answers in one mail.
> 
>> How do you define "right results"?
> 
> I run the exact same problem on in a commercial FEM package and
> compare the results. The test problem is diffusion-reaction, with a
> nonlinear reaction term of a polynomial form (tested up to order 5). I
> Form the Jacobian as follows. We have a problem Ax = b(x), with A
> constant and b depending pointwise on x (i.e. db/dxn does not depend
> on xm, m!=n). Therefore I provide a simple Jacobian equal to A with
> the diagonal augmented with -db/dx. The relevant code is below. I do
> hope I do something wrong and you can point it out.
> 
> 	ierr = MatCopy(that->system.A, *B, SAME_NONZERO_PATTERN); CHKERRQ(ierr);
> 
> 	ierr = VecScale(that->dbdx, -1.0); CHKERRQ(ierr);
> 	ierr = MatDiagonalSet(*B, that->dbdx, ADD_VALUES); CHKERRQ(ierr);
> 
> 	ierr = MatAssemblyBegin(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> 	ierr = MatAssemblyEnd(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> 
> 	if(*J != *B)
> 	{
> 		ierr = MatAssemblyBegin(*J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> 		ierr = MatAssemblyEnd(*J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> 	}
> 
> 	*flag = SAME_NONZERO_PATTERN;
> 
> 
>> Is SNES converging quadratically when you
>> do not use -snes_mf_operator?
> 
> Without -snes_mf_operator, providing analytically derived Jacobian:
> 
>  0 SNES Function norm 1.747536282885e+01
>  1 SNES Function norm 1.918464419679e-03
>  2 SNES Function norm 1.086746779744e-03
>  3 SNES Function norm 2.904556511511e-06
>  4 SNES Function norm 7.804808098453e-08
> 
> Without -snes_mf_operator, passing A as the jacobian (which you call
> Picard), and slightly underrelaxing the linear solves (otherwise it
> would not converge):
> 
>  0 SNES Function norm 1.747886139999e+01
>  1 SNES Function norm 3.495796117011e+00
>  2 SNES Function norm 6.991806248330e-01
>  3 SNES Function norm 1.398879936054e-01
>  4 SNES Function norm 2.809047018752e-02
>  5 SNES Function norm 5.843745932752e-03
>  6 SNES Function norm 1.594583643926e-03
>  7 SNES Function norm 7.712066092135e-04
>  8 SNES Function norm 4.932447251451e-04
>  9 SNES Function norm 3.217945635817e-04
> 10 SNES Function norm 2.129867244811e-04
> 11 SNES Function norm 1.399983127264e-04
> 12 SNES Function norm 9.246882776024e-05
> 13 SNES Function norm 6.088284618615e-05
> 14 SNES Function norm 4.017029247381e-05
> 15 SNES Function norm 2.646770266042e-05
> 16 SNES Function norm 1.745497080382e-05
> 17 SNES Function norm 1.150437737331e-05
> 18 SNES Function norm 7.585434690336e-06
> 19 SNES Function norm 5.000147553531e-06
> 20 SNES Function norm 3.296560754852e-06
> 21 SNES Function norm 2.173152296533e-06
> 22 SNES Function norm 1.432685503660e-06
> 23 SNES Function norm 9.444717211995e-07
> 24 SNES Function norm 6.226464090397e-07
> 25 SNES Function norm 4.104757979681e-07
> 26 SNES Function norm 2.706068836265e-07
> 27 SNES Function norm 1.783963930887e-07
> 28 SNES Function norm 1.176079057620e-07
> 
> Dominik
> 
>> Run with -snes_type test and see what it reports.
> 
> Running with the options:
> 
> -ksp_type bcgs -pc_type jacobi -ksp_rtol 1e-4 -ksp_max_it 1000
> -ksp_converged_use_initial_residual_norm -ksp_norm_type
> unpreconditioned -ksp_monitor_true_residual -ksp_converged_reason
> -ksp_view -snes_monitor -snes_converged_reason -snes_ls_monitor
> -snes_view -snes_type test -snes_test_display
> 
> I get the output:
> 
> Testing hand-coded Jacobian, if the ratio is
> O(1.e-8), the hand-coded Jacobian is probably correct.
> 
> and the procedure stagnates, calling FormJacobian only once, and then
> FormFunction infinitely, with my own computation of ||F|| indicating
> stagnation.

   It is not calling it infinitely. Run with the -snes_type test option for a small problem, with say 10 unknowns to start.

   Barry

> 
> Regards,
> Dominik


From knepley at gmail.com  Thu Apr 26 08:28:14 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 09:28:14 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
	<AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
Message-ID: <CAMYG4Gkp0V9YyW=fnmLmwD0e37nmZgoeO++qz2wM1UkfpB53MA@mail.gmail.com>

On Thu, Apr 26, 2012 at 9:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Apr 26, 2012, at 6:28 AM, Dominik Szczerba wrote:
>
> > Jed, Barry, Matt:
> >
> > Below a collection of all pending answers in one mail.
>

You can use -pc_type lu to test for a rank deficient preconditioning matrix.

   Matt


> >> How do you define "right results"?
> >
> > I run the exact same problem on in a commercial FEM package and
> > compare the results. The test problem is diffusion-reaction, with a
> > nonlinear reaction term of a polynomial form (tested up to order 5). I
> > Form the Jacobian as follows. We have a problem Ax = b(x), with A
> > constant and b depending pointwise on x (i.e. db/dxn does not depend
> > on xm, m!=n). Therefore I provide a simple Jacobian equal to A with
> > the diagonal augmented with -db/dx. The relevant code is below. I do
> > hope I do something wrong and you can point it out.
> >
> >       ierr = MatCopy(that->system.A, *B, SAME_NONZERO_PATTERN);
> CHKERRQ(ierr);
> >
> >       ierr = VecScale(that->dbdx, -1.0); CHKERRQ(ierr);
> >       ierr = MatDiagonalSet(*B, that->dbdx, ADD_VALUES); CHKERRQ(ierr);
> >
> >       ierr = MatAssemblyBegin(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> >       ierr = MatAssemblyEnd(*B, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
> >
> >       if(*J != *B)
> >       {
> >               ierr = MatAssemblyBegin(*J, MAT_FINAL_ASSEMBLY);
> CHKERRQ(ierr);
> >               ierr = MatAssemblyEnd(*J, MAT_FINAL_ASSEMBLY);
> CHKERRQ(ierr);
> >       }
> >
> >       *flag = SAME_NONZERO_PATTERN;
> >
> >
> >> Is SNES converging quadratically when you
> >> do not use -snes_mf_operator?
> >
> > Without -snes_mf_operator, providing analytically derived Jacobian:
> >
> >  0 SNES Function norm 1.747536282885e+01
> >  1 SNES Function norm 1.918464419679e-03
> >  2 SNES Function norm 1.086746779744e-03
> >  3 SNES Function norm 2.904556511511e-06
> >  4 SNES Function norm 7.804808098453e-08
> >
> > Without -snes_mf_operator, passing A as the jacobian (which you call
> > Picard), and slightly underrelaxing the linear solves (otherwise it
> > would not converge):
> >
> >  0 SNES Function norm 1.747886139999e+01
> >  1 SNES Function norm 3.495796117011e+00
> >  2 SNES Function norm 6.991806248330e-01
> >  3 SNES Function norm 1.398879936054e-01
> >  4 SNES Function norm 2.809047018752e-02
> >  5 SNES Function norm 5.843745932752e-03
> >  6 SNES Function norm 1.594583643926e-03
> >  7 SNES Function norm 7.712066092135e-04
> >  8 SNES Function norm 4.932447251451e-04
> >  9 SNES Function norm 3.217945635817e-04
> > 10 SNES Function norm 2.129867244811e-04
> > 11 SNES Function norm 1.399983127264e-04
> > 12 SNES Function norm 9.246882776024e-05
> > 13 SNES Function norm 6.088284618615e-05
> > 14 SNES Function norm 4.017029247381e-05
> > 15 SNES Function norm 2.646770266042e-05
> > 16 SNES Function norm 1.745497080382e-05
> > 17 SNES Function norm 1.150437737331e-05
> > 18 SNES Function norm 7.585434690336e-06
> > 19 SNES Function norm 5.000147553531e-06
> > 20 SNES Function norm 3.296560754852e-06
> > 21 SNES Function norm 2.173152296533e-06
> > 22 SNES Function norm 1.432685503660e-06
> > 23 SNES Function norm 9.444717211995e-07
> > 24 SNES Function norm 6.226464090397e-07
> > 25 SNES Function norm 4.104757979681e-07
> > 26 SNES Function norm 2.706068836265e-07
> > 27 SNES Function norm 1.783963930887e-07
> > 28 SNES Function norm 1.176079057620e-07
> >
> > Dominik
> >
> >> Run with -snes_type test and see what it reports.
> >
> > Running with the options:
> >
> > -ksp_type bcgs -pc_type jacobi -ksp_rtol 1e-4 -ksp_max_it 1000
> > -ksp_converged_use_initial_residual_norm -ksp_norm_type
> > unpreconditioned -ksp_monitor_true_residual -ksp_converged_reason
> > -ksp_view -snes_monitor -snes_converged_reason -snes_ls_monitor
> > -snes_view -snes_type test -snes_test_display
> >
> > I get the output:
> >
> > Testing hand-coded Jacobian, if the ratio is
> > O(1.e-8), the hand-coded Jacobian is probably correct.
> >
> > and the procedure stagnates, calling FormJacobian only once, and then
> > FormFunction infinitely, with my own computation of ||F|| indicating
> > stagnation.
>
>    It is not calling it infinitely. Run with the -snes_type test option
> for a small problem, with say 10 unknowns to start.
>
>   Barry
>
> >
> > Regards,
> > Dominik
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/22b24afe/attachment.htm>

From dominik at itis.ethz.ch  Thu Apr 26 08:33:32 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Thu, 26 Apr 2012 15:33:32 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAMYG4Gkp0V9YyW=fnmLmwD0e37nmZgoeO++qz2wM1UkfpB53MA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
	<AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
	<CAMYG4Gkp0V9YyW=fnmLmwD0e37nmZgoeO++qz2wM1UkfpB53MA@mail.gmail.com>
Message-ID: <CALm7otpzWi_wE3o+90GWb03X6YzMeV0MrHuwTTXU42kxGyZbeA@mail.gmail.com>

> You can use -pc_type lu to test for a rank deficient preconditioning matrix.
>
> ? ?Matt

I get:


[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Matrix format mpiaij does not have a built-in PETSc LU!


Dominik

From knepley at gmail.com  Thu Apr 26 08:45:54 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 09:45:54 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otpzWi_wE3o+90GWb03X6YzMeV0MrHuwTTXU42kxGyZbeA@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
	<AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
	<CAMYG4Gkp0V9YyW=fnmLmwD0e37nmZgoeO++qz2wM1UkfpB53MA@mail.gmail.com>
	<CALm7otpzWi_wE3o+90GWb03X6YzMeV0MrHuwTTXU42kxGyZbeA@mail.gmail.com>
Message-ID: <CAMYG4GmeRqPL-ewujO+Ln4mRSaDc9QSN6xqGF9jxUM9Y+=0KWQ@mail.gmail.com>

On Thu, Apr 26, 2012 at 9:33 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> > You can use -pc_type lu to test for a rank deficient preconditioning
> matrix.
> >
> >    Matt
>
> I get:
>
>
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Matrix format mpiaij does not have a built-in PETSc LU!


What are you doing trying to debug something in parallel that has nothing
to do with that?
Run in serial, on a small problem. Just messing around will not find the
problem.

   Matt


>
> Dominik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/db4716de/attachment.htm>

From behzad.baghapour at gmail.com  Thu Apr 26 08:47:40 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 26 Apr 2012 18:17:40 +0430
Subject: [petsc-users] How to change KSP tolerance during SNES steps
Message-ID: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>

Dear All,

I set SNES with -snes_ksp_ew to solve the system with inexact method. I
want to reset the SNES into exact Newton with changing the KSP rtol at
final iterations (in full Newton phase). How can I reset the KSP tolerance
after specific iterations of SNES?

Regards,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/099738da/attachment.htm>

From dominik at itis.ethz.ch  Thu Apr 26 08:59:19 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Thu, 26 Apr 2012 15:59:19 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CAM9tzS=7O5RXW6rLHoZ7Pu6A_dKF3YFCa00YSsfeSWj0=YcARQ@mail.gmail.com>
	<CALm7otoRxUFq-g_EphOw1FwsZm4DbL2KmE828AMp0ikf5Pwmtg@mail.gmail.com>
	<CAMYG4G=XTEHh_nbNucou1M9Kc8NFtrBttdxkagBtgTa1=WaF0Q@mail.gmail.com>
	<CALm7otqd5WbD=v6oE_19fBANThm3efwhCsEA845-L0F-oOe0tQ@mail.gmail.com>
	<8F093516-4E79-4A3A-99BF-BC176EBD2B1B@mcs.anl.gov>
	<CALm7otq1sVsM=ap8NrOkkuuMDNLa_mYEHz9VEAZfkKPdveHjxQ@mail.gmail.com>
	<CAM9tzSkaDnH5PW3CoX6XEAWqdHp8=XoZNKwCMFD2MkLBcozYfA@mail.gmail.com>
	<CALm7otpJmCUNFNGDCp8JbkTambUCLh94Ewj5qidc_CYfGWR7eA@mail.gmail.com>
	<AF692A26-6283-4D09-A6E6-19ECCAD2C208@mcs.anl.gov>
Message-ID: <CALm7otrT1zDVu=yb9xiab-4f6J3ebpBNWO3Rpss3TOcLQ7Q1ew@mail.gmail.com>

>> Running with the options:
>>
>> -ksp_type bcgs -pc_type jacobi -ksp_rtol 1e-4 -ksp_max_it 1000
>> -ksp_converged_use_initial_residual_norm -ksp_norm_type
>> unpreconditioned -ksp_monitor_true_residual -ksp_converged_reason
>> -ksp_view -snes_monitor -snes_converged_reason -snes_ls_monitor
>> -snes_view -snes_type test -snes_test_display
>>
>> I get the output:
>>
>> Testing hand-coded Jacobian, if the ratio is
>> O(1.e-8), the hand-coded Jacobian is probably correct.
>>
>> and the procedure stagnates, calling FormJacobian only once, and then
>> FormFunction infinitely, with my own computation of ||F|| indicating
>> stagnation.
>
> ? It is not calling it infinitely. Run with the -snes_type test option for a small problem, with say 10 unknowns to start.
>
> ? Barry

Thanks for a useful hint. The smallest problem I was able to generate
has around 1000 unknowns but it finally outputs something. I got some
matrix, with the biggest absolute entry order 0.1 the smallest
absolute entry order 1e-14, average values seem to be order 1e-8 or
so. End line of the output says: Norm of matrix ratio 0.579314
difference 10.9951 (constant state 1.0)

Dominik

From hzhang at mcs.anl.gov  Thu Apr 26 09:27:00 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 26 Apr 2012 09:27:00 -0500
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
Message-ID: <CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>

behzad :

>
> I set SNES with -snes_ksp_ew to solve the system with inexact method. I
> want to reset the SNES into exact Newton with changing the KSP rtol at
> final iterations (in full Newton phase). How can I reset the KSP tolerance
> after specific iterations of SNES?
>
You can use SNESMonitorSet() and write your own monitor function to do this:
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html


Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/78f1c8ed/attachment.htm>

From behzad.baghapour at gmail.com  Thu Apr 26 09:42:54 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 26 Apr 2012 19:12:54 +0430
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
Message-ID: <CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>

I already did it by changing level of fill of PC of KSP, but I received the
following error:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Cannot change levels after use!
[0]PETSC ERROR:
------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/5ed2a528/attachment.htm>

From knepley at gmail.com  Thu Apr 26 09:56:09 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 10:56:09 -0400
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
Message-ID: <CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>

On Thu, Apr 26, 2012 at 10:42 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> I already did it by changing level of fill of PC of KSP, but I received
> the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Cannot change levels after use!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
>

You will have to recreate the ILU object if you want to change the level of
fill.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/1524c3c3/attachment.htm>

From behzad.baghapour at gmail.com  Thu Apr 26 10:10:48 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 26 Apr 2012 19:40:48 +0430
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
	<CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
Message-ID: <CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>

Here is my procedure:

Define:

KSP mon_ksp;
PC mon_pre;
.
.
.
if( res_n < 1e-1 && c_test == 0 )
{
        KSPGetPC  ( mon_ksp,  &mon_pre );
        ierr = PCFactorSetLevels( mon_pre, 4 ); CHKERRQ( ierr );
        KSPSetPC  ( mon_ksp,   mon_pre );
        SNESSetKSP( snes, mon_ksp );
        ierr = SNESSetUp( snes ); CHKERRQ( ierr );
        c_test = 1;
}

and did get the error: "Object is in wrong state..."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/0ed5fc5b/attachment.htm>

From knepley at gmail.com  Thu Apr 26 10:12:06 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 11:12:06 -0400
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
	<CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
	<CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>
Message-ID: <CAMYG4G=3Tj6qTkjn9U8hV8s7cFGMT7n9BGqpAf3E4ewG+zBJoQ@mail.gmail.com>

On Thu, Apr 26, 2012 at 11:10 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Here is my procedure:
>
> Define:
>
> KSP mon_ksp;
> PC mon_pre;
> .
> .
> .
> if( res_n < 1e-1 && c_test == 0 )
> {
>         KSPGetPC  ( mon_ksp,  &mon_pre );
>         ierr = PCFactorSetLevels( mon_pre, 4 ); CHKERRQ( ierr );
>

Here you would have to PCReset(), set the type again, and set the options

   Matt


>         KSPSetPC  ( mon_ksp,   mon_pre );
>         SNESSetKSP( snes, mon_ksp );
>         ierr = SNESSetUp( snes ); CHKERRQ( ierr );
>         c_test = 1;
> }
>
> and did get the error: "Object is in wrong state..."
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/b6801feb/attachment.htm>

From behzad.baghapour at gmail.com  Thu Apr 26 10:31:33 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 26 Apr 2012 20:01:33 +0430
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CAMYG4G=3Tj6qTkjn9U8hV8s7cFGMT7n9BGqpAf3E4ewG+zBJoQ@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
	<CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
	<CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>
	<CAMYG4G=3Tj6qTkjn9U8hV8s7cFGMT7n9BGqpAf3E4ewG+zBJoQ@mail.gmail.com>
Message-ID: <CA+jOtDLAPfpPJmwqLuYFhpSWKJXdR18gYnaan5BLZE+6YnWQsg@mail.gmail.com>

Thanks. It did work!

Then how I should do for changing the tolerance of ksp? (changing from
inexact into exact Newton phase)

Thanks a lot,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/2b5cc391/attachment.htm>

From knepley at gmail.com  Thu Apr 26 10:37:40 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 11:37:40 -0400
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CA+jOtDLAPfpPJmwqLuYFhpSWKJXdR18gYnaan5BLZE+6YnWQsg@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
	<CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
	<CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>
	<CAMYG4G=3Tj6qTkjn9U8hV8s7cFGMT7n9BGqpAf3E4ewG+zBJoQ@mail.gmail.com>
	<CA+jOtDLAPfpPJmwqLuYFhpSWKJXdR18gYnaan5BLZE+6YnWQsg@mail.gmail.com>
Message-ID: <CAMYG4Gk3tLxqz_Ry2jeJnzxUEpKLDxmn=GRiLf6pq5aWPRGNRg@mail.gmail.com>

On Thu, Apr 26, 2012 at 11:31 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Thanks. It did work!
>
> Then how I should do for changing the tolerance of ksp? (changing from
> inexact into exact Newton phase)
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESKSPSetUseEW.html

   Matt


> Thanks a lot,
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/eec05d7b/attachment.htm>

From domel07 at gmail.com  Thu Apr 26 12:14:08 2012
From: domel07 at gmail.com (Dominik Szczerba)
Date: Thu, 26 Apr 2012 19:14:08 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
Message-ID: <CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>

> Thanks for a useful hint. The smallest problem I was able to generate
> has around 1000 unknowns but it finally outputs something. I got some
> matrix, with the biggest absolute entry order 0.1 the smallest
> absolute entry order 1e-14, average values seem to be order 1e-8 or
> so. End line of the output says: Norm of matrix ratio 0.579314
> difference 10.9951 (constant state 1.0)
>
> Dominik

So do you think this is reasonable or you still think I evaluate the
Jacobian incorrectly? Is there still something I can do to find out where
the problem is? Unfortunately, there is no easy way I can run the code
serially.

Dominik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/7c438bb8/attachment.htm>

From knepley at gmail.com  Thu Apr 26 13:02:14 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 14:02:14 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
Message-ID: <CAMYG4GmNuWRw1Hyofa3DgtChceEjbCjWscLH=QboPAYwtxzyRA@mail.gmail.com>

On Thu, Apr 26, 2012 at 1:14 PM, Dominik Szczerba <domel07 at gmail.com> wrote:

> > Thanks for a useful hint. The smallest problem I was able to generate
> > has around 1000 unknowns but it finally outputs something. I got some
> > matrix, with the biggest absolute entry order 0.1 the smallest
> > absolute entry order 1e-14, average values seem to be order 1e-8 or
> > so. End line of the output says: Norm of matrix ratio 0.579314
> > difference 10.9951 (constant state 1.0)
> >
> > Dominik
>
> So do you think this is reasonable or you still think I evaluate the
> Jacobian incorrectly? Is there still something I can do to find out where
> the problem is? Unfortunately, there is no easy way I can run the code
> serially.
>
Yes, it looks incorrect. The difference should be very small.

   Matt


> Dominik
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/293b20ea/attachment.htm>

From zonexo at gmail.com  Thu Apr 26 14:26:19 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Thu, 26 Apr 2012 21:26:19 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
Message-ID: <4F99A15B.1010500@gmail.com>

Hi,

Thanks for pointing out my mistake. I'm still learning about the code. I 
corrected but I still get the error:

[0]PETSC ERROR: --------------------- Error Message 
----------------------------
--------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (2,4) caused a malloc!
[0]PETSC ERROR: 
----------------------------------------------------------------

boundary condition region:

num = 1

                 if (j/=0) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,num) = i

                     col(MatStencil_j,num) = j-1

                     num = num + 1

                 end if

                 if (i/=0) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,num) = i-1

                     col(MatStencil_j,num) = j

                     num = num + 1

                 end if

                 if (i/=mx-1) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,num) = i+1

                     col(MatStencil_j,num) = j

                     num = num + 1

                 end if

                 if (j/=my-1) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,num) = i

                     col(MatStencil_j,num) = j+1

                     num = num + 1

                 end if

                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)

                 col(MatStencil_i,num) = i

                 col(MatStencil_j,num) = j+1

                 num = num + 1

                 call 
MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)

Yours sincerely,

TAY wee-beng


On 26/4/2012 2:45 PM, Matthew Knepley wrote:
> On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     Is there an answer to the question below?
>      I still can't get it working.
>
>     Thanks.
>
>
>     Hi,
>
>     There's no error now but I didn't get the same ans as ex29. I
>     believe the error lies in the boundary evaluation of v.
>
>     In c, the code is :
>
>     else if (user->bcType == NEUMANN) {
>               num = 0;
>               if (j!=0) {
>                 v[num] = -rho*HxdHy;              col[num].i = i;  
>     col[num].j = j-1;
>
>                                                                   ^^^ 
> See this?
>
>                 num++;
>               }
>               if (i!=0) {
>                 v[num] = -rho*HydHx;              col[num].i = i-1;
>     col[num].j = j;
>                 num++;
>               }
>               if (i!=mx-1) {
>                 v[num] = -rho*HydHx;              col[num].i = i+1;
>     col[num].j = j;
>                 num++;
>               }
>               if (j!=my-1) {
>                 v[num] = -rho*HxdHy;              col[num].i = i;  
>     col[num].j = j+1;
>                 num++;
>               }
>               v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i =
>     i;   col[num].j = j;
>               num++;
>               ierr =
>     MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>
>     In fortran, I have changed to :
>
>     else if (bc_cond == 2) then
>
>                     num = 1
>
>                     if (j/=0) then
>
>                         v(num) = -rho*HxdHy
>
>                         col(MatStencil_i,1) = i
>
>                                                ^^ Why is this 1 in 
> stead of 'num'?
>
>    Matt
>
>
>                         col(MatStencil_j,2) = j-1
>
>                         num = num + 1
>
>                     end if
>
>                     if (i/=0) then
>
>                         v(num) = -rho*HydHx
>
>                         col(MatStencil_i,1) = i-1
>
>                         col(MatStencil_j,2) = j
>
>                         num = num + 1
>
>                     end if
>
>                     if (i/=mx-1) then
>
>                         v(num) = -rho*HydHx
>
>                         col(MatStencil_i,1) = i+1
>
>                         col(MatStencil_j,2) = j
>
>                         num = num + 1
>
>                     end if
>
>                     if (j/=my-1) then
>
>                         v(num) = -rho*HxdHy
>
>                         col(MatStencil_i,1) = i
>
>                         col(MatStencil_j,2) = j+1
>
>                         num = num + 1
>
>                     end if
>
>                     v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>
>                     col(MatStencil_i,1) = i
>
>                     col(MatStencil_j,2) = j+1
>
>                     num = num + 1
>
>                     call
>     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
>     However, I got the error:
>
>     [0]PETSC ERROR: --------------------- Error Message
>     ---------------------------
>     --------
>     [0]PETSC ERROR: Argument out of range!
>     [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
>     [0]PETSC ERROR:
>     ---------------------------------------------------------------
>
>     How such is be defined in Fortran?
>
>     Thank you
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>>     On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I still got the same error the moment this subroutine is
>>         called, after changing to :
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>
>>     Argument 3 should be 0.
>>
>>       Matt
>>
>>         [0]PETSC ERROR: Null argument, when expecting valid pointer!
>>         [0]PETSC ERROR: Null Object: Parameter # 4!
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         fc934c1d84bc6ba8e2686702a8a99539d
>>         50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>         petsc-3.2 named USER-PC by
>>          User Tue Apr 24 22:54:50 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         /lib
>>         [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>         [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>         --with-fc="win32fe ifor
>>         t" --with-cxx="win32fe cl"
>>         --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>         las-lapack=1
>>         --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>         --with-debuggin
>>         g=1 --useThreads=0
>>         [0]PETSC ERROR:
>>         ----------------------------------------------------------------
>>         --------
>>         [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>         src/mat/interface/C:\temp\PETSC
>>         -~1\src\mat\INTERF~1\matnull.c
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>>         On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to
>>>             fortran version. The fortran version is based on the
>>>             template of ex22f, which is in 3d.
>>>
>>>             I have attached the code below. I have 2 problems.
>>>
>>>             1. When i3 = -3, in dirichlet BC, I got the same answer
>>>             as ex29 in c version.
>>>
>>>             However, when I change i3 to -4, I got the following error:
>>>
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Argument out of range!
>>>             [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:08:46 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>>             src/mat/impls/aij/seq/C:\temp\
>>>             PETSC-~1\src\mat\impls\aij\seq\aij.c
>>>             [0]PETSC ERROR: MatSetValues() line 1148 in
>>>             src/mat/interface/C:\temp\PETSC-~1\s
>>>             rc\mat\INTERF~1\matrix.c
>>>             [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>>             src/mat/interface/C:\temp\PETSC
>>>             -~1\src\mat\INTERF~1\matrix.c
>>>             [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>>             src/mat/interface/C:\temp\PET
>>>             SC-~1\src\mat\INTERF~1\matrix.c
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Argument out of range!
>>>             [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>
>>>             ...
>>>
>>>             What did I do wrong?
>>>
>>>
>>>             2. When I wanted to use the neumann BC, following ex29,
>>>             I added :
>>>
>>>             In subroutine ComputeRHS
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>>             call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>>             call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             and in subroutine ComputeMatrix
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>>             call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>             call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             When I compile, it says unresolved external symbol
>>>             MatNullSpaceRemove. Removing MatNullSpaceRemove, I can
>>>             compile but running gives the error:
>>>
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Corrupt argument:
>>>             see
>>>             http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>             [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:30:31 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>             src/mat/interface/C:\temp\PETSC
>>>             -~1\src\mat\INTERF~1\matnull.c
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Invalid argument!
>>>             [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:30:31 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>             src/mat/interface/C:\temp\PETS
>>>             C-~1\src\mat\INTERF~1\matnull.c
>>>             [0]PETSC ERROR: ourdmfunction() line 30 in
>>>             src/dm/interface/ftn-custom/C:\temp\P
>>>             ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>             [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>             src/dm/interface/C:\temp\PETSC-
>>>             ~1\src\dm\INTERF~1\dm.c
>>>             [0]PETSC ERROR: KSPSetUp() line 218 in
>>>             src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>             c\ksp\ksp\INTERF~1\itfunc.c
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Corrupt argument:
>>>             see
>>>             http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>             [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:30:31 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>             src/mat/interface/C:\temp\PETSC
>>>             -~1\src\mat\INTERF~1\matnull.c
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ----------------------------
>>>             --------
>>>             [0]PETSC ERROR: Invalid argument!
>>>             [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:30:31 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>             src/mat/interface/C:\temp\PETS
>>>             C-~1\src\mat\INTERF~1\matnull.c
>>>             [0]PETSC ERROR: ourdmfunction() line 30 in
>>>             src/dm/interface/ftn-custom/C:\temp\P
>>>             ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>             [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>             src/dm/interface/C:\temp\PETSC-
>>>             ~1\src\dm\INTERF~1\dm.c
>>>             [0]PETSC ERROR: KSPSetUp() line 218 in
>>>             src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>             c\ksp\ksp\INTERF~1\itfunc.c
>>>             [0]PETSC ERROR: KSPSolve() line 402 in
>>>             src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>             c\ksp\ksp\INTERF~1\itfunc.c
>>>             Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>               type: mpi
>>>             Process [0]
>>>
>>>             What's wrong?
>>>
>>>
>>>             *program ex29f*
>>>
>>>             implicit none
>>>
>>>             ! - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>             - - - - - - - - -
>>>             !                    Include files
>>>             ! - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>             - - - - - - - - -
>>>             !
>>>             !     petscsys.h  - base PETSc routines      petscvec.h
>>>             - vectors
>>>             !     petscmat.h - matrices
>>>             !     petscksp.h    - Krylov subspace methods 
>>>             petscpc.h  - preconditioners
>>>
>>>             #include "finclude/petsc.h90"
>>>
>>>             PetscErrorCode   ierr
>>>
>>>             DM               da
>>>
>>>             KSP              ksp
>>>
>>>             Vec              x
>>>
>>>             external         ComputeRHS,ComputeMatrix
>>>
>>>             PetscInt i1,i3
>>>
>>>             call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>
>>>             i3 = -4
>>>
>>>             i1 = 1
>>>
>>>             call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>
>>>             call
>>>             DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>             call DMSetFunction(da,ComputeRHS,ierr)
>>>             call DMSetJacobian(da,ComputeMatrix,ierr)
>>>             call KSPSetDM(ksp,da,ierr)
>>>
>>>             call KSPSetFromOptions(ksp,ierr)
>>>             call KSPSetUp(ksp,ierr)
>>>             call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>             call KSPGetSolution(ksp,x,ierr)
>>>             call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>>             call KSPDestroy(ksp,ierr)
>>>             call DMDestroy(da,ierr)
>>>             call PetscFinalize(ierr)
>>>
>>>             end program ex29f
>>>
>>>             subroutine ComputeRHS(da,x,b,ierr)
>>>
>>>             implicit none
>>>
>>>             #include "finclude/petsc.h90"
>>>
>>>             PetscErrorCode  ierr
>>>             PetscInt i,j,mx,my,xm,ym,xs,ys
>>>             PetscScalar  h,nu,rho
>>>             PetscScalar Hx,Hy
>>>             PetscScalar,pointer :: array(:,:)
>>>             Vec          x,b
>>>             DM           da
>>>             MatNullSpace nullspace    !>neumann BC
>>>
>>>             nu = 0.1
>>>
>>>             call
>>>             DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>
>>>             Hx = 1.d0 / (mx-1)
>>>
>>>             Hy = 1.d0 / (my-1)
>>>
>>>             call
>>>             DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>
>>>             call DMDAVecGetArrayF90(da,b,array,ierr)
>>>
>>>             do j = ys,ys+ym-1
>>>
>>>                 do i = xs,xs+xm-1
>>>
>>>                     array(i,j) =
>>>             exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>
>>>                 end do
>>>
>>>             end do
>>>
>>>             call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>
>>>             call VecAssemblyBegin(b,ierr)
>>>
>>>             call VecAssemblyEnd(b,ierr)
>>>
>>>             !call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>>
>>>         4th argument should be PETSC_NULL_OBJECT.
>>>
>>>            Matt
>>>
>>>             !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>>             !call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             end subroutine ComputeRHS
>>>
>>>
>>>             subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>
>>>             implicit none
>>>
>>>             #include "finclude/petsc.h90"
>>>
>>>             Mat          jac,JJ
>>>
>>>             PetscErrorCode    ierr
>>>
>>>             DM           da
>>>
>>>             PetscInt    i,j,k,mx,my,xm
>>>
>>>             PetscInt    ym,xs,ys,i1,i5
>>>
>>>             PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>
>>>             PetscScalar  HydHx
>>>
>>>             PetscScalar  HxdHy
>>>
>>>             MatStencil   row(4),col(4,5)
>>>
>>>             Vec          x
>>>
>>>             MatStructure str
>>>
>>>             MatNullSpace nullspace    !>neumann BC
>>>
>>>             rho = 1.0
>>>
>>>             i1 = 1
>>>
>>>             i5 = 5
>>>
>>>             centerRho = rho
>>>
>>>             call
>>>             DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>
>>>             Hx = 1.d0 / (mx-1)
>>>
>>>             Hy = 1.d0 / (my-1)
>>>
>>>             HxdHy = Hx/Hy
>>>
>>>             HydHx = Hy/Hx
>>>
>>>             call
>>>             DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>
>>>
>>>             do j=ys,ys+ym-1
>>>
>>>                 do i=xs,xs+xm-1
>>>
>>>                     row(MatStencil_i) = i
>>>
>>>                     row(MatStencil_j) = j
>>>
>>>                     call ComputeRho(i,j,mx,my,centerRho,rho)
>>>
>>>                     if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or.
>>>             j.eq.my-1) then
>>>
>>>                         v(1) = 2.0*rho*(HxdHy + HydHx)
>>>
>>>                         call
>>>             MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>
>>>                     else
>>>
>>>                         v(1) = -rho*HxdHy
>>>
>>>                         col(MatStencil_i,1) = i
>>>
>>>                         col(MatStencil_j,2) = j-1
>>>
>>>
>>>         Cut and paste error above.
>>>
>>>           Matt
>>>
>>>                         v(2) = -rho*HydHx
>>>
>>>                         col(MatStencil_i,2) = i-1
>>>
>>>                         col(MatStencil_j,2) = j
>>>
>>>                         v(3) = 2.0*rho*(HxdHy + HydHx)
>>>
>>>                         col(MatStencil_i,3) = i
>>>
>>>                         col(MatStencil_j,3) = j
>>>
>>>                         v(4) = -rho*HydHx
>>>
>>>                         col(MatStencil_i,4) = i+1
>>>
>>>                         col(MatStencil_j,4) = j
>>>
>>>                         v(5) = -rho*HxdHy
>>>
>>>                         col(MatStencil_i,5) = i
>>>
>>>                         col(MatStencil_j,5) = j+1
>>>
>>>                         call
>>>             MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>
>>>                     end if
>>>
>>>                 end do
>>>
>>>             end do
>>>
>>>             call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>
>>>             call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>
>>>             !call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>
>>>             !call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>             !call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             end subroutine ComputeMatrix
>>>
>>>             subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>
>>>             PetscInt    i,j,mx,my
>>>
>>>             PetscScalar  rho,centerRho
>>>
>>>             if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j >
>>>             my/3.0) .and. (j < 2.0*my/3.0)) then
>>>
>>>                 rho = centerRho
>>>
>>>             else
>>>
>>>                 rho = 1.0
>>>
>>>             end if
>>>
>>>
>>>             end subroutine ComputeRho
>>>
>>>             -- 
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/0dc3969b/attachment-0001.htm>

From knepley at gmail.com  Thu Apr 26 14:29:21 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 26 Apr 2012 15:29:21 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F99A15B.1010500@gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
Message-ID: <CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>

On Thu, Apr 26, 2012 at 3:26 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> Thanks for pointing out my mistake. I'm still learning about the code. I
> corrected but I still get the error:
>

I am sure its another simple bug. Go in with the debugger. When it says you
are inserting a new nonzero,
back up the stack to your routine and see what indices you are tying to
insert. If its not part of the stencil,
you have found the bug.

  Matt


> [0]PETSC ERROR: --------------------- Error Message
> ----------------------------
> --------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,4) caused a malloc!
> [0]PETSC ERROR:
> ----------------------------------------------------------------
>
> boundary condition region:
>
> num = 1
>
>                 if (j/=0) then
>
>                     v(num) = -rho*HxdHy
>
>                     col(MatStencil_i,num) = i
>
>                     col(MatStencil_j,num) = j-1
>
>                     num = num + 1
>
>                 end if
>
>                 if (i/=0) then
>
>                     v(num) = -rho*HydHx
>
>                     col(MatStencil_i,num) = i-1
>
>                     col(MatStencil_j,num) = j
>
>                     num = num + 1
>
>                 end if
>
>                 if (i/=mx-1) then
>
>                     v(num) = -rho*HydHx
>
>                     col(MatStencil_i,num) = i+1
>
>                     col(MatStencil_j,num) = j
>
>                     num = num + 1
>
>                 end if
>
>                 if (j/=my-1) then
>
>                     v(num) = -rho*HxdHy
>
>                     col(MatStencil_i,num) = i
>
>                     col(MatStencil_j,num) = j+1
>
>                     num = num + 1
>
>                 end if
>
>                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>
>                 col(MatStencil_i,num) = i
>
>                 col(MatStencil_j,num) = j+1
>
>                 num = num + 1
>
>                 call
> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 26/4/2012 2:45 PM, Matthew Knepley wrote:
>
> On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> Is there an answer to the question below?
>>  I still can't get it working.
>>
>> Thanks.
>>
>>
>> Hi,
>>
>> There's no error now but I didn't get the same ans as ex29. I believe the
>> error lies in the boundary evaluation of v.
>>
>> In c, the code is :
>>
>> else if (user->bcType == NEUMANN) {
>>           num = 0;
>>           if (j!=0) {
>>             v[num] = -rho*HxdHy;              col[num].i = i;
>> col[num].j = j-1;
>>
>                                                                   ^^^ See
> this?
>
>>              num++;
>>           }
>>           if (i!=0) {
>>             v[num] = -rho*HydHx;              col[num].i = i-1;
>> col[num].j = j;
>>             num++;
>>           }
>>           if (i!=mx-1) {
>>             v[num] = -rho*HydHx;              col[num].i = i+1;
>> col[num].j = j;
>>             num++;
>>           }
>>           if (j!=my-1) {
>>             v[num] = -rho*HxdHy;              col[num].i = i;
>> col[num].j = j+1;
>>             num++;
>>           }
>>           v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;
>> col[num].j = j;
>>           num++;
>>           ierr =
>> MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>
>> In fortran, I have changed to :
>>
>> else if (bc_cond == 2) then
>>
>>                 num = 1
>>
>>                 if (j/=0) then
>>
>>                     v(num) = -rho*HxdHy
>>
>>                     col(MatStencil_i,1) = i
>>
>                                                ^^ Why is this 1 in stead
> of 'num'?
>
>     Matt
>
>
>                       col(MatStencil_j,2) = j-1
>>
>>                     num = num + 1
>>
>>                 end if
>>
>>                 if (i/=0) then
>>
>>                     v(num) = -rho*HydHx
>>
>>                     col(MatStencil_i,1) = i-1
>>
>>                     col(MatStencil_j,2) = j
>>
>>                     num = num + 1
>>
>>                 end if
>>
>>                 if (i/=mx-1) then
>>
>>                     v(num) = -rho*HydHx
>>
>>                     col(MatStencil_i,1) = i+1
>>
>>                     col(MatStencil_j,2) = j
>>
>>                     num = num + 1
>>
>>                 end if
>>
>>                 if (j/=my-1) then
>>
>>                     v(num) = -rho*HxdHy
>>
>>                     col(MatStencil_i,1) = i
>>
>>                     col(MatStencil_j,2) = j+1
>>
>>                     num = num + 1
>>
>>                 end if
>>
>>                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>>
>>                 col(MatStencil_i,1) = i
>>
>>                 col(MatStencil_j,2) = j+1
>>
>>                 num = num + 1
>>
>>                 call
>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>> However, I got the error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ---------------------------
>> --------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
>> [0]PETSC ERROR:
>> ---------------------------------------------------------------
>>
>> How such is be defined in Fortran?
>>
>> Thank you
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>>
>> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I still got the same error the moment this subroutine is called, after
>>> changing to :
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>
>>  Argument 3 should be 0.
>>
>>    Matt
>>
>>
>>>  [0]PETSC ERROR: Null argument, when expecting valid pointer!
>>> [0]PETSC ERROR: Null Object: Parameter # 4!
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> fc934c1d84bc6ba8e2686702a8a99539d
>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>> USER-PC by
>>>  User Tue Apr 24 22:54:50 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> /lib
>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>> --with-fc="win32fe ifor
>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>> --download-f-b
>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>> --with-debuggin
>>> g=1 --useThreads=0
>>> [0]PETSC ERROR:
>>> ----------------------------------------------------------------
>>> --------
>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>> src/mat/interface/C:\temp\PETSC
>>> -~1\src\mat\INTERF~1\matnull.c
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>>
>>> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran
>>>> version. The fortran version is based on the template of ex22f, which is in
>>>> 3d.
>>>>
>>>> I have attached the code below. I have 2 problems.
>>>>
>>>> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c
>>>> version.
>>>>
>>>> However, when I change i3 to -4, I got the following error:
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Argument out of range!
>>>> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision:
>>>> fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>>> USER-PC by
>>>>  User Tue Apr 24 22:08:46 2012
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>> --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>>> --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>>> src/mat/impls/aij/seq/C:\temp\
>>>> PETSC-~1\src\mat\impls\aij\seq\aij.c
>>>> [0]PETSC ERROR: MatSetValues() line 1148 in
>>>> src/mat/interface/C:\temp\PETSC-~1\s
>>>> rc\mat\INTERF~1\matrix.c
>>>> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>>> src/mat/interface/C:\temp\PETSC
>>>> -~1\src\mat\INTERF~1\matrix.c
>>>> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>>> src/mat/interface/C:\temp\PET
>>>> SC-~1\src\mat\INTERF~1\matrix.c
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Argument out of range!
>>>> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>>
>>>> ...
>>>>
>>>> What did I do wrong?
>>>>
>>>>
>>>> 2. When I wanted to use the neumann BC, following ex29, I added :
>>>>
>>>> In subroutine ComputeRHS
>>>>
>>>> call
>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> and in subroutine ComputeMatrix
>>>>
>>>> call
>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>> call MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> When I compile, it says unresolved external symbol MatNullSpaceRemove.
>>>> Removing MatNullSpaceRemove, I can compile but running gives the error:
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Corrupt argument:
>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision:
>>>> fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>>> USER-PC by
>>>>  User Tue Apr 24 22:30:31 2012
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>> --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>>> --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>> src/mat/interface/C:\temp\PETSC
>>>> -~1\src\mat\INTERF~1\matnull.c
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Invalid argument!
>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision:
>>>> fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>>> USER-PC by
>>>>  User Tue Apr 24 22:30:31 2012
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>> --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>>> --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>> src/mat/interface/C:\temp\PETS
>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>> [0]PETSC ERROR: ourdmfunction() line 30 in
>>>> src/dm/interface/ftn-custom/C:\temp\P
>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>> src/dm/interface/C:\temp\PETSC-
>>>> ~1\src\dm\INTERF~1\dm.c
>>>> [0]PETSC ERROR: KSPSetUp() line 218 in
>>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Corrupt argument:
>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision:
>>>> fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>>> USER-PC by
>>>>  User Tue Apr 24 22:30:31 2012
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>> --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>>> --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>> src/mat/interface/C:\temp\PETSC
>>>> -~1\src\mat\INTERF~1\matnull.c
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ----------------------------
>>>> --------
>>>> [0]PETSC ERROR: Invalid argument!
>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision:
>>>> fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named
>>>> USER-PC by
>>>>  User Tue Apr 24 22:30:31 2012
>>>> [0]PETSC ERROR: Libraries linked from
>>>> /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>> --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/
>>>> --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR:
>>>> ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>> src/mat/interface/C:\temp\PETS
>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>> [0]PETSC ERROR: ourdmfunction() line 30 in
>>>> src/dm/interface/ftn-custom/C:\temp\P
>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>> src/dm/interface/C:\temp\PETSC-
>>>> ~1\src\dm\INTERF~1\dm.c
>>>> [0]PETSC ERROR: KSPSetUp() line 218 in
>>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>> [0]PETSC ERROR: KSPSolve() line 402 in
>>>> src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>>
>>>> What's wrong?
>>>>
>>>>
>>>> *program ex29f*
>>>>
>>>> implicit none
>>>>
>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> -
>>>> !                    Include files
>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> -
>>>> !
>>>> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>>> !     petscmat.h - matrices
>>>> !     petscksp.h    - Krylov subspace methods  petscpc.h  -
>>>> preconditioners
>>>>
>>>> #include "finclude/petsc.h90"
>>>>
>>>> PetscErrorCode   ierr
>>>>
>>>> DM               da
>>>>
>>>> KSP              ksp
>>>>
>>>> Vec              x
>>>>
>>>> external         ComputeRHS,ComputeMatrix
>>>>
>>>> PetscInt i1,i3
>>>>
>>>> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>>
>>>> i3 = -4
>>>>
>>>> i1 = 1
>>>>
>>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>>
>>>> call
>>>> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>> call DMSetFunction(da,ComputeRHS,ierr)
>>>> call DMSetJacobian(da,ComputeMatrix,ierr)
>>>> call KSPSetDM(ksp,da,ierr)
>>>>
>>>> call KSPSetFromOptions(ksp,ierr)
>>>> call KSPSetUp(ksp,ierr)
>>>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>> call KSPGetSolution(ksp,x,ierr)
>>>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>>> call KSPDestroy(ksp,ierr)
>>>> call DMDestroy(da,ierr)
>>>> call PetscFinalize(ierr)
>>>>
>>>> end program ex29f
>>>>
>>>> subroutine ComputeRHS(da,x,b,ierr)
>>>>
>>>> implicit none
>>>>
>>>> #include "finclude/petsc.h90"
>>>>
>>>> PetscErrorCode  ierr
>>>> PetscInt i,j,mx,my,xm,ym,xs,ys
>>>> PetscScalar  h,nu,rho
>>>> PetscScalar Hx,Hy
>>>> PetscScalar,pointer :: array(:,:)
>>>> Vec          x,b
>>>> DM           da
>>>> MatNullSpace nullspace    !>neumann BC
>>>>
>>>> nu = 0.1
>>>>
>>>> call
>>>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>
>>>> Hx = 1.d0 / (mx-1)
>>>>
>>>> Hy = 1.d0 / (my-1)
>>>>
>>>> call
>>>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>
>>>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>>>
>>>> do j = ys,ys+ym-1
>>>>
>>>>     do i = xs,xs+xm-1
>>>>
>>>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>>
>>>>     end do
>>>>
>>>> end do
>>>>
>>>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>>
>>>> call VecAssemblyBegin(b,ierr)
>>>>
>>>> call VecAssemblyEnd(b,ierr)
>>>>
>>>> !call
>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>
>>>  4th argument should be PETSC_NULL_OBJECT.
>>>
>>>     Matt
>>>
>>>
>>>>  !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> end subroutine ComputeRHS
>>>>
>>>>
>>>> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>>
>>>> implicit none
>>>>
>>>> #include "finclude/petsc.h90"
>>>>
>>>> Mat          jac,JJ
>>>>
>>>> PetscErrorCode    ierr
>>>>
>>>> DM           da
>>>>
>>>> PetscInt    i,j,k,mx,my,xm
>>>>
>>>> PetscInt    ym,xs,ys,i1,i5
>>>>
>>>> PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>>
>>>> PetscScalar  HydHx
>>>>
>>>> PetscScalar  HxdHy
>>>>
>>>> MatStencil   row(4),col(4,5)
>>>>
>>>> Vec          x
>>>>
>>>> MatStructure str
>>>>
>>>> MatNullSpace nullspace    !>neumann BC
>>>>
>>>> rho = 1.0
>>>>
>>>> i1 = 1
>>>>
>>>> i5 = 5
>>>>
>>>> centerRho = rho
>>>>
>>>> call
>>>> DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>
>>>> Hx = 1.d0 / (mx-1)
>>>>
>>>> Hy = 1.d0 / (my-1)
>>>>
>>>> HxdHy = Hx/Hy
>>>>
>>>> HydHx = Hy/Hx
>>>>
>>>> call
>>>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>
>>>>
>>>> do j=ys,ys+ym-1
>>>>
>>>>     do i=xs,xs+xm-1
>>>>
>>>>         row(MatStencil_i) = i
>>>>
>>>>         row(MatStencil_j) = j
>>>>
>>>>         call ComputeRho(i,j,mx,my,centerRho,rho)
>>>>
>>>>         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>>>>
>>>>             v(1) = 2.0*rho*(HxdHy + HydHx)
>>>>
>>>>             call
>>>> MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>>
>>>>         else
>>>>
>>>>             v(1) = -rho*HxdHy
>>>>
>>>>             col(MatStencil_i,1) = i
>>>>
>>>>             col(MatStencil_j,2) = j-1
>>>>
>>>
>>>  Cut and paste error above.
>>>
>>>    Matt
>>>
>>>
>>>>              v(2) = -rho*HydHx
>>>>
>>>>             col(MatStencil_i,2) = i-1
>>>>
>>>>             col(MatStencil_j,2) = j
>>>>
>>>>             v(3) = 2.0*rho*(HxdHy + HydHx)
>>>>
>>>>             col(MatStencil_i,3) = i
>>>>
>>>>             col(MatStencil_j,3) = j
>>>>
>>>>             v(4) = -rho*HydHx
>>>>
>>>>             col(MatStencil_i,4) = i+1
>>>>
>>>>             col(MatStencil_j,4) = j
>>>>
>>>>             v(5) = -rho*HxdHy
>>>>
>>>>             col(MatStencil_i,5) = i
>>>>
>>>>             col(MatStencil_j,5) = j+1
>>>>
>>>>             call
>>>> MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>>
>>>>         end if
>>>>
>>>>     end do
>>>>
>>>> end do
>>>>
>>>> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>
>>>> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>
>>>> !call
>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>> !call MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> end subroutine ComputeMatrix
>>>>
>>>> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>>
>>>> PetscInt    i,j,mx,my
>>>>
>>>> PetscScalar  rho,centerRho
>>>>
>>>> if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j <
>>>> 2.0*my/3.0)) then
>>>>
>>>>     rho = centerRho
>>>>
>>>> else
>>>>
>>>>     rho = 1.0
>>>>
>>>> end if
>>>>
>>>>
>>>> end subroutine ComputeRho
>>>>
>>>> --
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120426/5e426494/attachment-0001.htm>

From bsmith at mcs.anl.gov  Thu Apr 26 14:57:45 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Apr 2012 14:57:45 -0500
Subject: [petsc-users] How to change KSP tolerance during SNES steps
In-Reply-To: <CA+jOtDLAPfpPJmwqLuYFhpSWKJXdR18gYnaan5BLZE+6YnWQsg@mail.gmail.com>
References: <CA+jOtDLvXvSOqbc03oQ7H24QTX97bYyD0BEvkWShkckq2AoAYQ@mail.gmail.com>
	<CAGCphBusDP8u630jVo9Z7-1u3P9_1j1n0Crwxr-OtrQ5DsA_3A@mail.gmail.com>
	<CA+jOtDKHSt_96tioV8PEw9p_Ft8=GtpgQF5nJ=xoN-ojcy8a6w@mail.gmail.com>
	<CAMYG4G=VduibZ+O4kBmnYvazgCp80My78=ys1F5=U5oecorh0Q@mail.gmail.com>
	<CA+jOtDJF1uU9Hy+m5BWAN1ROm9wXwNirASkXes5Uk8gvOUzFhQ@mail.gmail.com>
	<CAMYG4G=3Tj6qTkjn9U8hV8s7cFGMT7n9BGqpAf3E4ewG+zBJoQ@mail.gmail.com>
	<CA+jOtDLAPfpPJmwqLuYFhpSWKJXdR18gYnaan5BLZE+6YnWQsg@mail.gmail.com>
Message-ID: <5D808164-D426-4DCA-B404-80AC63EFC379@mcs.anl.gov>


On Apr 26, 2012, at 10:31 AM, behzad baghapour wrote:

> Thanks. It did work!
> 
> Then how I should do for changing the tolerance of ksp? (changing from inexact into exact Newton phase)
> 
> Thanks a lot,
> 


  Hong already answered that


behzad :

I set SNES with -snes_ksp_ew to solve the system with inexact method. I want to reset the SNES into exact Newton with changing the KSP rtol at final iterations (in full Newton phase). How can I reset the KSP tolerance after specific iterations of SNES?
You can use SNESMonitorSet() and write your own monitor function to do this:
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html 

Hong

From bsmith at mcs.anl.gov  Thu Apr 26 15:05:35 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Apr 2012 15:05:35 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
Message-ID: <D34711D9-5687-4EA7-98F5-C105ECE84966@mcs.anl.gov>


On Apr 26, 2012, at 12:14 PM, Dominik Szczerba wrote:

> > Thanks for a useful hint. The smallest problem I was able to generate
> > has around 1000 unknowns

(1) 

> but it finally outputs something. I got some
> > matrix, with the biggest absolute entry order 0.1 the smallest
> > absolute entry order 1e-14, average values seem to be order 1e-8 or
> > so. End line of the output says: Norm of matrix ratio 0.579314
> > difference 10.9951 (constant state 1.0)
> >
> > Dominik
> 
> So do you think this is reasonable or you still think I evaluate the Jacobian incorrectly? Is there still something I can do to find out where the problem is? Unfortunately, there is no easy way I can run the code serially.

(2)
> 
> Dominik

  You set yourself up for a very painful life. It is simply insane to have a code that cannot run with tiny problems for debugging and understanding and similarly run sequentially. Parallel computing with MPI is HARD and the only way to make it at all manageable is to organize the code so that it is easy to debug and understand.  You may think it is silly to waste time making the code run on small problems sequentially since you want to run big problems but it is not silly, it is crucial. You would already have debugged and fixed this problem if you could run the code sequentially on tiny problems. The extra time in making the code debuggable is much much less than the extra time wasted futzing around trying to find a bug with no tools to debug.

    The -snes_type test option has another option -snes_test_display which will display the two Jacobians and the difference between them allowing one to see which entries in the Jacobian (if not all of them) are wrong.  If you could run on a small problem sequentially you would run with these options, see what Jacobian entries are wrong and look back at the code that generates those entries and likely see the bug.

   Barry


From zonexo at gmail.com  Thu Apr 26 17:21:00 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 27 Apr 2012 00:21:00 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
Message-ID: <4F99CA4C.1080609@gmail.com>

Hi,

I managed to remove the out of range error by removing

*num = num + 1* from

col(MatStencil_i,num) = i

                 col(MatStencil_j,num) = j+1

*num = num + 1*

                 call 
MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)

In C, num will give the number 5 since num start from 0.

In fortran, num starts from 1 so num = num + 1 before 
MatSetValuesStencil must be removed.

I still can't get the same ans. I will try to figure it out. Just to 
confirm, this subroutine:

call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)

is not added in fortran, although it is used in C.

Yours sincerely,

TAY wee-beng


On 26/4/2012 9:29 PM, Matthew Knepley wrote:
> On Thu, Apr 26, 2012 at 3:26 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     Thanks for pointing out my mistake. I'm still learning about the
>     code. I corrected but I still get the error:
>
>
> I am sure its another simple bug. Go in with the debugger. When it 
> says you are inserting a new nonzero,
> back up the stack to your routine and see what indices you are tying 
> to insert. If its not part of the stencil,
> you have found the bug.
>
>   Matt
>
>     [0]PETSC ERROR: --------------------- Error Message
>     ----------------------------
>     --------
>     [0]PETSC ERROR: Argument out of range!
>     [0]PETSC ERROR: New nonzero at (2,4) caused a malloc!
>     [0]PETSC ERROR:
>     ----------------------------------------------------------------
>
>     boundary condition region:
>
>     num = 1
>
>                     if (j/=0) then
>
>                         v(num) = -rho*HxdHy
>
>                         col(MatStencil_i,num) = i
>
>                         col(MatStencil_j,num) = j-1
>
>                         num = num + 1
>
>                     end if
>
>                     if (i/=0) then
>
>                         v(num) = -rho*HydHx
>
>                         col(MatStencil_i,num) = i-1
>
>                         col(MatStencil_j,num) = j
>
>                         num = num + 1
>
>                     end if
>
>                     if (i/=mx-1) then
>
>                         v(num) = -rho*HydHx
>
>                         col(MatStencil_i,num) = i+1
>
>                         col(MatStencil_j,num) = j
>
>                         num = num + 1
>
>                     end if
>
>                     if (j/=my-1) then
>
>                         v(num) = -rho*HxdHy
>
>                         col(MatStencil_i,num) = i
>
>                         col(MatStencil_j,num) = j+1
>
>                         num = num + 1
>
>                     end if
>
>                     v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>
>                     col(MatStencil_i,num) = i
>
>                     col(MatStencil_j,num) = j+1
>
>                     num = num + 1
>
>                     call
>     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 26/4/2012 2:45 PM, Matthew Knepley wrote:
>>     On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         Is there an answer to the question below?
>>          I still can't get it working.
>>
>>         Thanks.
>>
>>
>>         Hi,
>>
>>         There's no error now but I didn't get the same ans as ex29. I
>>         believe the error lies in the boundary evaluation of v.
>>
>>         In c, the code is :
>>
>>         else if (user->bcType == NEUMANN) {
>>                   num = 0;
>>                   if (j!=0) {
>>                     v[num] = -rho*HxdHy;              col[num].i =
>>         i;   col[num].j = j-1;
>>
>>                                                                      
>>     ^^^ See this?
>>
>>                     num++;
>>                   }
>>                   if (i!=0) {
>>                     v[num] = -rho*HydHx;              col[num].i =
>>         i-1; col[num].j = j;
>>                     num++;
>>                   }
>>                   if (i!=mx-1) {
>>                     v[num] = -rho*HydHx;              col[num].i =
>>         i+1; col[num].j = j;
>>                     num++;
>>                   }
>>                   if (j!=my-1) {
>>                     v[num] = -rho*HxdHy;              col[num].i =
>>         i;   col[num].j = j+1;
>>                     num++;
>>                   }
>>                   v[num]   = (num/2.0)*rho*(HxdHy + HydHx);
>>         col[num].i = i;   col[num].j = j;
>>                   num++;
>>                   ierr =
>>         MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>
>>         In fortran, I have changed to :
>>
>>         else if (bc_cond == 2) then
>>
>>                         num = 1
>>
>>                         if (j/=0) then
>>
>>                             v(num) = -rho*HxdHy
>>
>>                             col(MatStencil_i,1) = i
>>
>>                                                    ^^ Why is this 1
>>     in stead of 'num'?
>>
>>        Matt
>>
>>
>>                             col(MatStencil_j,2) = j-1
>>
>>                             num = num + 1
>>
>>                         end if
>>
>>                         if (i/=0) then
>>
>>                             v(num) = -rho*HydHx
>>
>>                             col(MatStencil_i,1) = i-1
>>
>>                             col(MatStencil_j,2) = j
>>
>>                             num = num + 1
>>
>>                         end if
>>
>>                         if (i/=mx-1) then
>>
>>                             v(num) = -rho*HydHx
>>
>>                             col(MatStencil_i,1) = i+1
>>
>>                             col(MatStencil_j,2) = j
>>
>>                             num = num + 1
>>
>>                         end if
>>
>>                         if (j/=my-1) then
>>
>>                             v(num) = -rho*HxdHy
>>
>>                             col(MatStencil_i,1) = i
>>
>>                             col(MatStencil_j,2) = j+1
>>
>>                             num = num + 1
>>
>>                         end if
>>
>>                         v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>>
>>                         col(MatStencil_i,1) = i
>>
>>                         col(MatStencil_j,2) = j+1
>>
>>                         num = num + 1
>>
>>                         call
>>         MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>>         However, I got the error:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ---------------------------
>>         --------
>>         [0]PETSC ERROR: Argument out of range!
>>         [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
>>         [0]PETSC ERROR:
>>         ---------------------------------------------------------------
>>
>>         How such is be defined in Fortran?
>>
>>         Thank you
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>>>         On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I still got the same error the moment this subroutine is
>>>             called, after changing to :
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>
>>>         Argument 3 should be 0.
>>>
>>>           Matt
>>>
>>>             [0]PETSC ERROR: Null argument, when expecting valid pointer!
>>>             [0]PETSC ERROR: Null Object: Parameter # 4!
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             fc934c1d84bc6ba8e2686702a8a99539d
>>>             50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a
>>>             petsc-3.2 named USER-PC by
>>>              User Tue Apr 24 22:54:50 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             /lib
>>>             [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>             [0]PETSC ERROR: Configure options --with-cc="win32fe cl"
>>>             --with-fc="win32fe ifor
>>>             t" --with-cxx="win32fe cl"
>>>             --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>             las-lapack=1
>>>             --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>             --with-debuggin
>>>             g=1 --useThreads=0
>>>             [0]PETSC ERROR:
>>>             ----------------------------------------------------------------
>>>             --------
>>>             [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>             src/mat/interface/C:\temp\PETSC
>>>             -~1\src\mat\INTERF~1\matnull.c
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>             On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>>>             On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from
>>>>                 c to fortran version. The fortran version is based
>>>>                 on the template of ex22f, which is in 3d.
>>>>
>>>>                 I have attached the code below. I have 2 problems.
>>>>
>>>>                 1. When i3 = -3, in dirichlet BC, I got the same
>>>>                 answer as ex29 in c version.
>>>>
>>>>                 However, when I change i3 to -4, I got the
>>>>                 following error:
>>>>
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Argument out of range!
>>>>                 [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Petsc Development HG revision:
>>>>                 fc934c1d84bc6ba8e2686702a8a99539d
>>>>                 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>                 [0]PETSC ERROR: See docs/changes/index.html for
>>>>                 recent updates.
>>>>                 [0]PETSC ERROR: See docs/faq.html for hints about
>>>>                 trouble shooting.
>>>>                 [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on
>>>>                 a petsc-3.2 named USER-PC by
>>>>                  User Tue Apr 24 22:08:46 2012
>>>>                 [0]PETSC ERROR: Libraries linked from
>>>>                 /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 /lib
>>>>                 [0]PETSC ERROR: Configure run at Mon Apr 23
>>>>                 21:45:20 2012
>>>>                 [0]PETSC ERROR: Configure options
>>>>                 --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>                 t" --with-cxx="win32fe cl"
>>>>                 --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>                 las-lapack=1
>>>>                 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 --with-debuggin
>>>>                 g=1 --useThreads=0
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in
>>>>                 src/mat/impls/aij/seq/C:\temp\
>>>>                 PETSC-~1\src\mat\impls\aij\seq\aij.c
>>>>                 [0]PETSC ERROR: MatSetValues() line 1148 in
>>>>                 src/mat/interface/C:\temp\PETSC-~1\s
>>>>                 rc\mat\INTERF~1\matrix.c
>>>>                 [0]PETSC ERROR: MatSetValuesLocal() line 2003 in
>>>>                 src/mat/interface/C:\temp\PETSC
>>>>                 -~1\src\mat\INTERF~1\matrix.c
>>>>                 [0]PETSC ERROR: MatSetValuesStencil() line 1379 in
>>>>                 src/mat/interface/C:\temp\PET
>>>>                 SC-~1\src\mat\INTERF~1\matrix.c
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Argument out of range!
>>>>                 [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>>
>>>>                 ...
>>>>
>>>>                 What did I do wrong?
>>>>
>>>>
>>>>                 2. When I wanted to use the neumann BC, following
>>>>                 ex29, I added :
>>>>
>>>>                 In subroutine ComputeRHS
>>>>
>>>>                 call
>>>>                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>>                 call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                 and in subroutine ComputeMatrix
>>>>
>>>>                 call
>>>>                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>>                 call MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                 When I compile, it says unresolved external symbol
>>>>                 MatNullSpaceRemove. Removing MatNullSpaceRemove, I
>>>>                 can compile but running gives the error:
>>>>
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Corrupt argument:
>>>>                 see
>>>>                 http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>                 [0]PETSC ERROR: Invalid Pointer to Object:
>>>>                 Parameter # 4!
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Petsc Development HG revision:
>>>>                 fc934c1d84bc6ba8e2686702a8a99539d
>>>>                 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>                 [0]PETSC ERROR: See docs/changes/index.html for
>>>>                 recent updates.
>>>>                 [0]PETSC ERROR: See docs/faq.html for hints about
>>>>                 trouble shooting.
>>>>                 [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on
>>>>                 a petsc-3.2 named USER-PC by
>>>>                  User Tue Apr 24 22:30:31 2012
>>>>                 [0]PETSC ERROR: Libraries linked from
>>>>                 /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 /lib
>>>>                 [0]PETSC ERROR: Configure run at Mon Apr 23
>>>>                 21:45:20 2012
>>>>                 [0]PETSC ERROR: Configure options
>>>>                 --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>                 t" --with-cxx="win32fe cl"
>>>>                 --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>                 las-lapack=1
>>>>                 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 --with-debuggin
>>>>                 g=1 --useThreads=0
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>>                 src/mat/interface/C:\temp\PETSC
>>>>                 -~1\src\mat\INTERF~1\matnull.c
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Invalid argument!
>>>>                 [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Petsc Development HG revision:
>>>>                 fc934c1d84bc6ba8e2686702a8a99539d
>>>>                 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>                 [0]PETSC ERROR: See docs/changes/index.html for
>>>>                 recent updates.
>>>>                 [0]PETSC ERROR: See docs/faq.html for hints about
>>>>                 trouble shooting.
>>>>                 [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on
>>>>                 a petsc-3.2 named USER-PC by
>>>>                  User Tue Apr 24 22:30:31 2012
>>>>                 [0]PETSC ERROR: Libraries linked from
>>>>                 /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 /lib
>>>>                 [0]PETSC ERROR: Configure run at Mon Apr 23
>>>>                 21:45:20 2012
>>>>                 [0]PETSC ERROR: Configure options
>>>>                 --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>                 t" --with-cxx="win32fe cl"
>>>>                 --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>                 las-lapack=1
>>>>                 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 --with-debuggin
>>>>                 g=1 --useThreads=0
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>>                 src/mat/interface/C:\temp\PETS
>>>>                 C-~1\src\mat\INTERF~1\matnull.c
>>>>                 [0]PETSC ERROR: ourdmfunction() line 30 in
>>>>                 src/dm/interface/ftn-custom/C:\temp\P
>>>>                 ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>                 [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>>                 src/dm/interface/C:\temp\PETSC-
>>>>                 ~1\src\dm\INTERF~1\dm.c
>>>>                 [0]PETSC ERROR: KSPSetUp() line 218 in
>>>>                 src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>                 c\ksp\ksp\INTERF~1\itfunc.c
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Corrupt argument:
>>>>                 see
>>>>                 http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>                 [0]PETSC ERROR: Invalid Pointer to Object:
>>>>                 Parameter # 4!
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Petsc Development HG revision:
>>>>                 fc934c1d84bc6ba8e2686702a8a99539d
>>>>                 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>                 [0]PETSC ERROR: See docs/changes/index.html for
>>>>                 recent updates.
>>>>                 [0]PETSC ERROR: See docs/faq.html for hints about
>>>>                 trouble shooting.
>>>>                 [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on
>>>>                 a petsc-3.2 named USER-PC by
>>>>                  User Tue Apr 24 22:30:31 2012
>>>>                 [0]PETSC ERROR: Libraries linked from
>>>>                 /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 /lib
>>>>                 [0]PETSC ERROR: Configure run at Mon Apr 23
>>>>                 21:45:20 2012
>>>>                 [0]PETSC ERROR: Configure options
>>>>                 --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>                 t" --with-cxx="win32fe cl"
>>>>                 --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>                 las-lapack=1
>>>>                 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 --with-debuggin
>>>>                 g=1 --useThreads=0
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: MatNullSpaceCreate() line 250 in
>>>>                 src/mat/interface/C:\temp\PETSC
>>>>                 -~1\src\mat\INTERF~1\matnull.c
>>>>                 [0]PETSC ERROR: --------------------- Error Message
>>>>                 ----------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Invalid argument!
>>>>                 [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: Petsc Development HG revision:
>>>>                 fc934c1d84bc6ba8e2686702a8a99539d
>>>>                 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>                 [0]PETSC ERROR: See docs/changes/index.html for
>>>>                 recent updates.
>>>>                 [0]PETSC ERROR: See docs/faq.html for hints about
>>>>                 trouble shooting.
>>>>                 [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on
>>>>                 a petsc-3.2 named USER-PC by
>>>>                  User Tue Apr 24 22:30:31 2012
>>>>                 [0]PETSC ERROR: Libraries linked from
>>>>                 /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 /lib
>>>>                 [0]PETSC ERROR: Configure run at Mon Apr 23
>>>>                 21:45:20 2012
>>>>                 [0]PETSC ERROR: Configure options
>>>>                 --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>                 t" --with-cxx="win32fe cl"
>>>>                 --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>                 las-lapack=1
>>>>                 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>                 --with-debuggin
>>>>                 g=1 --useThreads=0
>>>>                 [0]PETSC ERROR:
>>>>                 ----------------------------------------------------------------
>>>>                 --------
>>>>                 [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in
>>>>                 src/mat/interface/C:\temp\PETS
>>>>                 C-~1\src\mat\INTERF~1\matnull.c
>>>>                 [0]PETSC ERROR: ourdmfunction() line 30 in
>>>>                 src/dm/interface/ftn-custom/C:\temp\P
>>>>                 ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>                 [0]PETSC ERROR: DMComputeFunction() line 1783 in
>>>>                 src/dm/interface/C:\temp\PETSC-
>>>>                 ~1\src\dm\INTERF~1\dm.c
>>>>                 [0]PETSC ERROR: KSPSetUp() line 218 in
>>>>                 src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>                 c\ksp\ksp\INTERF~1\itfunc.c
>>>>                 [0]PETSC ERROR: KSPSolve() line 402 in
>>>>                 src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>                 c\ksp\ksp\INTERF~1\itfunc.c
>>>>                 Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>
>>>>                 What's wrong?
>>>>
>>>>
>>>>                 *program ex29f*
>>>>
>>>>                 implicit none
>>>>
>>>>                 ! - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>                 - - - - - - - - - - -
>>>>                 !                    Include files
>>>>                 ! - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>                 - - - - - - - - - - -
>>>>                 !
>>>>                 !     petscsys.h  - base PETSc routines     
>>>>                 petscvec.h - vectors
>>>>                 !     petscmat.h - matrices
>>>>                 !     petscksp.h    - Krylov subspace methods 
>>>>                 petscpc.h  - preconditioners
>>>>
>>>>                 #include "finclude/petsc.h90"
>>>>
>>>>                 PetscErrorCode   ierr
>>>>
>>>>                 DM               da
>>>>
>>>>                 KSP              ksp
>>>>
>>>>                 Vec              x
>>>>
>>>>                 external         ComputeRHS,ComputeMatrix
>>>>
>>>>                 PetscInt i1,i3
>>>>
>>>>                 call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>>
>>>>                 i3 = -4
>>>>
>>>>                 i1 = 1
>>>>
>>>>                 call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>>
>>>>                 call
>>>>                 DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>>                 call DMSetFunction(da,ComputeRHS,ierr)
>>>>                 call DMSetJacobian(da,ComputeMatrix,ierr)
>>>>                 call KSPSetDM(ksp,da,ierr)
>>>>
>>>>                 call KSPSetFromOptions(ksp,ierr)
>>>>                 call KSPSetUp(ksp,ierr)
>>>>                 call
>>>>                 KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>>                 call KSPGetSolution(ksp,x,ierr)
>>>>                 call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>>>                 call KSPDestroy(ksp,ierr)
>>>>                 call DMDestroy(da,ierr)
>>>>                 call PetscFinalize(ierr)
>>>>
>>>>                 end program ex29f
>>>>
>>>>                 subroutine ComputeRHS(da,x,b,ierr)
>>>>
>>>>                 implicit none
>>>>
>>>>                 #include "finclude/petsc.h90"
>>>>
>>>>                 PetscErrorCode  ierr
>>>>                 PetscInt i,j,mx,my,xm,ym,xs,ys
>>>>                 PetscScalar  h,nu,rho
>>>>                 PetscScalar Hx,Hy
>>>>                 PetscScalar,pointer :: array(:,:)
>>>>                 Vec          x,b
>>>>                 DM           da
>>>>                 MatNullSpace nullspace    !>neumann BC
>>>>
>>>>                 nu = 0.1
>>>>
>>>>                 call
>>>>                 DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>
>>>>                 Hx = 1.d0 / (mx-1)
>>>>
>>>>                 Hy = 1.d0 / (my-1)
>>>>
>>>>                 call
>>>>                 DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>
>>>>                 call DMDAVecGetArrayF90(da,b,array,ierr)
>>>>
>>>>                 do j = ys,ys+ym-1
>>>>
>>>>                     do i = xs,xs+xm-1
>>>>
>>>>                         array(i,j) =
>>>>                 exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>>
>>>>                     end do
>>>>
>>>>                 end do
>>>>
>>>>                 call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>>
>>>>                 call VecAssemblyBegin(b,ierr)
>>>>
>>>>                 call VecAssemblyEnd(b,ierr)
>>>>
>>>>                 !call
>>>>                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>>
>>>>             4th argument should be PETSC_NULL_OBJECT.
>>>>
>>>>                Matt
>>>>
>>>>                 !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>>                 !call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                 end subroutine ComputeRHS
>>>>
>>>>
>>>>                 subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>>
>>>>                 implicit none
>>>>
>>>>                 #include "finclude/petsc.h90"
>>>>
>>>>                 Mat          jac,JJ
>>>>
>>>>                 PetscErrorCode    ierr
>>>>
>>>>                 DM           da
>>>>
>>>>                 PetscInt    i,j,k,mx,my,xm
>>>>
>>>>                 PetscInt    ym,xs,ys,i1,i5
>>>>
>>>>                 PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>>
>>>>                 PetscScalar  HydHx
>>>>
>>>>                 PetscScalar  HxdHy
>>>>
>>>>                 MatStencil   row(4),col(4,5)
>>>>
>>>>                 Vec          x
>>>>
>>>>                 MatStructure str
>>>>
>>>>                 MatNullSpace nullspace    !>neumann BC
>>>>
>>>>                 rho = 1.0
>>>>
>>>>                 i1 = 1
>>>>
>>>>                 i5 = 5
>>>>
>>>>                 centerRho = rho
>>>>
>>>>                 call
>>>>                 DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>
>>>>                 Hx = 1.d0 / (mx-1)
>>>>
>>>>                 Hy = 1.d0 / (my-1)
>>>>
>>>>                 HxdHy = Hx/Hy
>>>>
>>>>                 HydHx = Hy/Hx
>>>>
>>>>                 call
>>>>                 DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>
>>>>
>>>>                 do j=ys,ys+ym-1
>>>>
>>>>                     do i=xs,xs+xm-1
>>>>
>>>>                         row(MatStencil_i) = i
>>>>
>>>>                         row(MatStencil_j) = j
>>>>
>>>>                         call ComputeRho(i,j,mx,my,centerRho,rho)
>>>>
>>>>                         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or.
>>>>                 j.eq.my-1) then
>>>>
>>>>                             v(1) = 2.0*rho*(HxdHy + HydHx)
>>>>
>>>>                             call
>>>>                 MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>>
>>>>                         else
>>>>
>>>>                             v(1) = -rho*HxdHy
>>>>
>>>>                             col(MatStencil_i,1) = i
>>>>
>>>>                             col(MatStencil_j,2) = j-1
>>>>
>>>>
>>>>             Cut and paste error above.
>>>>
>>>>               Matt
>>>>
>>>>                             v(2) = -rho*HydHx
>>>>
>>>>                             col(MatStencil_i,2) = i-1
>>>>
>>>>                             col(MatStencil_j,2) = j
>>>>
>>>>                             v(3) = 2.0*rho*(HxdHy + HydHx)
>>>>
>>>>                             col(MatStencil_i,3) = i
>>>>
>>>>                             col(MatStencil_j,3) = j
>>>>
>>>>                             v(4) = -rho*HydHx
>>>>
>>>>                             col(MatStencil_i,4) = i+1
>>>>
>>>>                             col(MatStencil_j,4) = j
>>>>
>>>>                             v(5) = -rho*HxdHy
>>>>
>>>>                             col(MatStencil_i,5) = i
>>>>
>>>>                             col(MatStencil_j,5) = j+1
>>>>
>>>>                             call
>>>>                 MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>>
>>>>                         end if
>>>>
>>>>                     end do
>>>>
>>>>                 end do
>>>>
>>>>                 call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>
>>>>                 call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>
>>>>                 !call
>>>>                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>
>>>>                 !call MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>>                 !call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                 end subroutine ComputeMatrix
>>>>
>>>>                 subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>>
>>>>                 PetscInt    i,j,mx,my
>>>>
>>>>                 PetscScalar  rho,centerRho
>>>>
>>>>                 if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j >
>>>>                 my/3.0) .and. (j < 2.0*my/3.0)) then
>>>>
>>>>                     rho = centerRho
>>>>
>>>>                 else
>>>>
>>>>                     rho = 1.0
>>>>
>>>>                 end if
>>>>
>>>>
>>>>                 end subroutine ComputeRho
>>>>
>>>>                 -- 
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/502a644b/attachment-0001.htm>

From bsmith at mcs.anl.gov  Thu Apr 26 18:22:35 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Apr 2012 18:22:35 -0500
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4F99CA4C.1080609@gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
Message-ID: <C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>


  From the manual page for MatSetValuesStencil():

   MatSetValuesStencil() uses 0-based row and column numbers in Fortran 
   as well as in C.


On Apr 26, 2012, at 5:21 PM, TAY wee-beng wrote:

> Hi,
> 
> I managed to remove the out of range error by removing 
> 
>    num = num + 1 from
> 
> col(MatStencil_i,num) = i
> 
>                 col(MatStencil_j,num) = j+1
> 
>                 num = num + 1
>             
>                 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
> 
> In C, num will give the number 5 since num start from 0.
> 
> In fortran, num starts from 1 so num = num + 1 before MatSetValuesStencil must be removed.
> 
> I still can't get the same ans. I will try to figure it out. Just to confirm, this subroutine:
> 
> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
> 
> is not added in fortran, although it is used in C.
> Yours sincerely,
> 
> TAY wee-beng
> 
> 
> On 26/4/2012 9:29 PM, Matthew Knepley wrote:
>> On Thu, Apr 26, 2012 at 3:26 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>> Hi,
>> 
>> Thanks for pointing out my mistake. I'm still learning about the code. I corrected but I still get the error:
>> 
>> I am sure its another simple bug. Go in with the debugger. When it says you are inserting a new nonzero,
>> back up the stack to your routine and see what indices you are tying to insert. If its not part of the stencil,
>> you have found the bug.
>> 
>>   Matt
>>  
>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>> --------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: New nonzero at (2,4) caused a malloc!
>> [0]PETSC ERROR: ----------------------------------------------------------------
>> 
>> boundary condition region:
>> 
>> num = 1
>> 
>>                 if (j/=0) then
>>                 
>>                     v(num) = -rho*HxdHy
>> 
>>                     col(MatStencil_i,num) = i
>> 
>>                     col(MatStencil_j,num) = j-1
>> 
>>                     num = num + 1
>> 
>>                 end if
>> 
>>                 if (i/=0) then
>>                 
>>                     v(num) = -rho*HydHx
>> 
>>                     col(MatStencil_i,num) = i-1
>> 
>>                     col(MatStencil_j,num) = j
>> 
>>                     num = num + 1
>> 
>>                 end if
>> 
>>                 if (i/=mx-1) then
>>                 
>>                     v(num) = -rho*HydHx
>> 
>>                     col(MatStencil_i,num) = i+1
>> 
>>                     col(MatStencil_j,num) = j
>> 
>>                     num = num + 1
>> 
>>                 end if
>> 
>>                 if (j/=my-1) then
>>                 
>>                     v(num) = -rho*HxdHy
>> 
>>                     col(MatStencil_i,num) = i
>> 
>>                     col(MatStencil_j,num) = j+1
>> 
>>                     num = num + 1
>> 
>>                 end if
>>                 
>>                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>> 
>>                 col(MatStencil_i,num) = i
>> 
>>                 col(MatStencil_j,num) = j+1
>> 
>>                 num = num + 1
>>             
>>                 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>> Yours sincerely,
>> 
>> TAY wee-beng
>> 
>> 
>> On 26/4/2012 2:45 PM, Matthew Knepley wrote:
>>> On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>> Hi,
>>> 
>>> Is there an answer to the question below?
>>>  I still can't get it working.
>>> 
>>> Thanks.
>>> 
>>> 
>>> Hi,
>>> 
>>> There's no error now but I didn't get the same ans as ex29. I believe the error lies in the boundary evaluation of v.
>>> 
>>> In c, the code is :
>>> 
>>> else if (user->bcType == NEUMANN) {
>>>           num = 0;
>>>           if (j!=0) {
>>>             v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j-1;
>>>                                                                   ^^^ See this? 
>>>             num++;
>>>           }
>>>           if (i!=0) {
>>>             v[num] = -rho*HydHx;              col[num].i = i-1; col[num].j = j;
>>>             num++;
>>>           }
>>>           if (i!=mx-1) {
>>>             v[num] = -rho*HydHx;              col[num].i = i+1; col[num].j = j;
>>>             num++;
>>>           }
>>>           if (j!=my-1) {
>>>             v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j+1;
>>>             num++;
>>>           }
>>>           v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;   col[num].j = j;
>>>           num++;
>>>           ierr = MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>> 
>>> In fortran, I have changed to :
>>> 
>>> else if (bc_cond == 2) then
>>> 
>>>                 num = 1
>>> 
>>>                 if (j/=0) then
>>>                 
>>>                     v(num) = -rho*HxdHy
>>> 
>>>                     col(MatStencil_i,1) = i
>>>                                                ^^ Why is this 1 in stead of 'num'? 
>>> 
>>>    Matt
>>> 
>>> 
>>>                     col(MatStencil_j,2) = j-1
>>> 
>>>                     num = num + 1
>>> 
>>>                 end if
>>> 
>>>                 if (i/=0) then
>>>                 
>>>                     v(num) = -rho*HydHx
>>> 
>>>                     col(MatStencil_i,1) = i-1
>>> 
>>>                     col(MatStencil_j,2) = j
>>> 
>>>                     num = num + 1
>>> 
>>>                 end if
>>> 
>>>                 if (i/=mx-1) then
>>>                 
>>>                     v(num) = -rho*HydHx
>>> 
>>>                     col(MatStencil_i,1) = i+1
>>> 
>>>                     col(MatStencil_j,2) = j
>>> 
>>>                     num = num + 1
>>> 
>>>                 end if
>>> 
>>>                 if (j/=my-1) then
>>>                 
>>>                     v(num) = -rho*HxdHy
>>> 
>>>                     col(MatStencil_i,1) = i
>>> 
>>>                     col(MatStencil_j,2) = j+1
>>> 
>>>                     num = num + 1
>>> 
>>>                 end if
>>>                 
>>>                 v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>>> 
>>>                 col(MatStencil_i,1) = i
>>> 
>>>                 col(MatStencil_j,2) = j+1
>>> 
>>>                 num = num + 1
>>>             
>>>                 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>> 
>>> However, I got the error: 
>>> 
>>> [0]PETSC ERROR: --------------------- Error Message ---------------------------
>>> --------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
>>> [0]PETSC ERROR: ---------------------------------------------------------------
>>> 
>>> How such is be defined in Fortran?
>>> 
>>> Thank you
>>> Yours sincerely,
>>> 
>>> TAY wee-beng
>>> 
>>> 
>>> On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>>>> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> I still got the same error the moment this subroutine is called, after changing to :
>>>> 
>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>>> 
>>>> Argument 3 should be 0.
>>>> 
>>>>   Matt
>>>>  
>>>> [0]PETSC ERROR: Null argument, when expecting valid pointer!
>>>> [0]PETSC ERROR: Null Object: Parameter # 4!
>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>  User Tue Apr 24 22:54:50 2012
>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>> /lib
>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>> g=1 --useThreads=0
>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>> --------
>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>> -~1\src\mat\INTERF~1\matnull.c
>>>> Yours sincerely,
>>>> 
>>>> TAY wee-beng
>>>> 
>>>> 
>>>> On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>>>> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>> Hi,
>>>>> 
>>>>> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran version. The fortran version is based on the template of ex22f, which is in 3d.
>>>>> 
>>>>> I have attached the code below. I have 2 problems.
>>>>> 
>>>>> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c version.
>>>>> 
>>>>> However, when I change i3 to -4, I got the following error:
>>>>> 
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Argument out of range!
>>>>> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>  User Tue Apr 24 22:08:46 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in src/mat/impls/aij/seq/C:\temp\
>>>>> PETSC-~1\src\mat\impls\aij\seq\aij.c
>>>>> [0]PETSC ERROR: MatSetValues() line 1148 in src/mat/interface/C:\temp\PETSC-~1\s
>>>>> rc\mat\INTERF~1\matrix.c
>>>>> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in src/mat/interface/C:\temp\PETSC
>>>>> -~1\src\mat\INTERF~1\matrix.c
>>>>> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in src/mat/interface/C:\temp\PET
>>>>> SC-~1\src\mat\INTERF~1\matrix.c
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Argument out of range!
>>>>> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>>> 
>>>>> ...
>>>>> 
>>>>> What did I do wrong?
>>>>> 
>>>>> 
>>>>> 2. When I wanted to use the neumann BC, following ex29, I added :
>>>>> 
>>>>> In subroutine ComputeRHS
>>>>> 
>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>> 
>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>> 
>>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> and in subroutine ComputeMatrix
>>>>> 
>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>> 
>>>>> call MatSetNullSpace(jac,nullspace,ierr)
>>>>> 
>>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> When I compile, it says unresolved external symbol MatNullSpaceRemove. Removing MatNullSpaceRemove, I can compile but running gives the error:
>>>>> 
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Corrupt argument:
>>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>  User Tue Apr 24 22:30:31 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>>> -~1\src\mat\INTERF~1\matnull.c
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Invalid argument!
>>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>  User Tue Apr 24 22:30:31 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in src/mat/interface/C:\temp\PETS
>>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>>> [0]PETSC ERROR: ourdmfunction() line 30 in src/dm/interface/ftn-custom/C:\temp\P
>>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in src/dm/interface/C:\temp\PETSC-
>>>>> ~1\src\dm\INTERF~1\dm.c
>>>>> [0]PETSC ERROR: KSPSetUp() line 218 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Corrupt argument:
>>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>  User Tue Apr 24 22:30:31 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>>> -~1\src\mat\INTERF~1\matnull.c
>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Invalid argument!
>>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>  User Tue Apr 24 22:30:31 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in src/mat/interface/C:\temp\PETS
>>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>>> [0]PETSC ERROR: ourdmfunction() line 30 in src/dm/interface/ftn-custom/C:\temp\P
>>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in src/dm/interface/C:\temp\PETSC-
>>>>> ~1\src\dm\INTERF~1\dm.c
>>>>> [0]PETSC ERROR: KSPSetUp() line 218 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>> [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>   type: mpi
>>>>> Process [0]
>>>>> 
>>>>> What's wrong?
>>>>> 
>>>>> 
>>>>> program ex29f
>>>>> 
>>>>> implicit none
>>>>> 
>>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>> !                    Include files
>>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>> !
>>>>> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>>>> !     petscmat.h - matrices
>>>>> !     petscksp.h    - Krylov subspace methods  petscpc.h  - preconditioners
>>>>> 
>>>>> #include "finclude/petsc.h90"
>>>>> 
>>>>> PetscErrorCode   ierr
>>>>> 
>>>>> DM               da
>>>>> 
>>>>> KSP              ksp
>>>>> 
>>>>> Vec              x
>>>>> 
>>>>> external         ComputeRHS,ComputeMatrix
>>>>> 
>>>>> PetscInt i1,i3
>>>>> 
>>>>> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>>> 
>>>>> i3 = -4
>>>>> 
>>>>> i1 = 1
>>>>> 
>>>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>>> 
>>>>> call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>>> call DMSetFunction(da,ComputeRHS,ierr)
>>>>> call DMSetJacobian(da,ComputeMatrix,ierr)   
>>>>> call KSPSetDM(ksp,da,ierr)
>>>>> 
>>>>> call KSPSetFromOptions(ksp,ierr)
>>>>> call KSPSetUp(ksp,ierr)
>>>>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>>> call KSPGetSolution(ksp,x,ierr)
>>>>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>>>> call KSPDestroy(ksp,ierr)
>>>>> call DMDestroy(da,ierr)
>>>>> call PetscFinalize(ierr)
>>>>> 
>>>>> end program ex29f
>>>>> 
>>>>> subroutine ComputeRHS(da,x,b,ierr)
>>>>> 
>>>>> implicit none
>>>>> 
>>>>> #include "finclude/petsc.h90"
>>>>> 
>>>>> PetscErrorCode  ierr
>>>>> PetscInt i,j,mx,my,xm,ym,xs,ys
>>>>> PetscScalar  h,nu,rho
>>>>> PetscScalar Hx,Hy
>>>>> PetscScalar,pointer :: array(:,:)
>>>>> Vec          x,b
>>>>> DM           da
>>>>> MatNullSpace nullspace    !>neumann BC
>>>>> 
>>>>> nu = 0.1
>>>>> 
>>>>> call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>> 
>>>>> Hx = 1.d0 / (mx-1)
>>>>> 
>>>>> Hy = 1.d0 / (my-1)
>>>>> 
>>>>> call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>> 
>>>>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>>>> 
>>>>> do j = ys,ys+ym-1
>>>>> 
>>>>>     do i = xs,xs+xm-1
>>>>> 
>>>>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>>> 
>>>>>     end do
>>>>> 
>>>>> end do
>>>>> 
>>>>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>>> 
>>>>> call VecAssemblyBegin(b,ierr)
>>>>> 
>>>>> call VecAssemblyEnd(b,ierr)
>>>>> 
>>>>> !call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>> 
>>>>> 4th argument should be PETSC_NULL_OBJECT.
>>>>> 
>>>>>    Matt
>>>>>  
>>>>> !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>> 
>>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> end subroutine ComputeRHS
>>>>> 
>>>>> 
>>>>> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>>> 
>>>>> implicit none
>>>>> 
>>>>> #include "finclude/petsc.h90"
>>>>> 
>>>>> Mat          jac,JJ
>>>>> 
>>>>> PetscErrorCode    ierr
>>>>> 
>>>>> DM           da
>>>>> 
>>>>> PetscInt    i,j,k,mx,my,xm
>>>>> 
>>>>> PetscInt    ym,xs,ys,i1,i5
>>>>> 
>>>>> PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>>> 
>>>>> PetscScalar  HydHx
>>>>> 
>>>>> PetscScalar  HxdHy
>>>>> 
>>>>> MatStencil   row(4),col(4,5)
>>>>> 
>>>>> Vec          x
>>>>> 
>>>>> MatStructure str
>>>>> 
>>>>> MatNullSpace nullspace    !>neumann BC
>>>>> 
>>>>> rho = 1.0
>>>>> 
>>>>> i1 = 1
>>>>> 
>>>>> i5 = 5
>>>>> 
>>>>> centerRho = rho
>>>>> 
>>>>> call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>> 
>>>>> Hx = 1.d0 / (mx-1)
>>>>> 
>>>>> Hy = 1.d0 / (my-1)
>>>>> 
>>>>> HxdHy = Hx/Hy
>>>>> 
>>>>> HydHx = Hy/Hx
>>>>> 
>>>>> call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)        
>>>>> 
>>>>> do j=ys,ys+ym-1
>>>>> 
>>>>>     do i=xs,xs+xm-1
>>>>> 
>>>>>         row(MatStencil_i) = i  
>>>>> 
>>>>>         row(MatStencil_j) = j
>>>>>         
>>>>>         call ComputeRho(i,j,mx,my,centerRho,rho)
>>>>> 
>>>>>         if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>>>>> 
>>>>>             v(1) = 2.0*rho*(HxdHy + HydHx)
>>>>>             
>>>>>             call MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>>> 
>>>>>         else 
>>>>> 
>>>>>             v(1) = -rho*HxdHy
>>>>> 
>>>>>             col(MatStencil_i,1) = i
>>>>> 
>>>>>             col(MatStencil_j,2) = j-1
>>>>> 
>>>>> Cut and paste error above.
>>>>> 
>>>>>   Matt
>>>>>  
>>>>>             v(2) = -rho*HydHx
>>>>> 
>>>>>             col(MatStencil_i,2) = i-1 
>>>>> 
>>>>>             col(MatStencil_j,2) = j
>>>>>             
>>>>>             v(3) = 2.0*rho*(HxdHy + HydHx)
>>>>> 
>>>>>             col(MatStencil_i,3) = i 
>>>>> 
>>>>>             col(MatStencil_j,3) = j
>>>>> 
>>>>>             v(4) = -rho*HydHx
>>>>> 
>>>>>             col(MatStencil_i,4) = i+1 
>>>>> 
>>>>>             col(MatStencil_j,4) = j 
>>>>> 
>>>>>             v(5) = -rho*HxdHy
>>>>> 
>>>>>             col(MatStencil_i,5) = i 
>>>>> 
>>>>>             col(MatStencil_j,5) = j+1
>>>>> 
>>>>>             call MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>>> 
>>>>>         end if
>>>>> 
>>>>>     end do
>>>>> 
>>>>> end do
>>>>> 
>>>>> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>> 
>>>>> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>> 
>>>>> !call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>> 
>>>>> !call MatSetNullSpace(jac,nullspace,ierr)
>>>>> 
>>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> end subroutine ComputeMatrix
>>>>> 
>>>>> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>>> 
>>>>> PetscInt    i,j,mx,my
>>>>> 
>>>>> PetscScalar  rho,centerRho
>>>>> 
>>>>> if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j < 2.0*my/3.0)) then
>>>>> 
>>>>>     rho = centerRho
>>>>> 
>>>>> else
>>>>> 
>>>>>     rho = 1.0
>>>>> 
>>>>> end if
>>>>> 
>>>>> 
>>>>> end subroutine ComputeRho
>>>>> -- 
>>>>> Yours sincerely,
>>>>> 
>>>>> TAY wee-beng
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener


From iamkyungjoo at gmail.com  Thu Apr 26 20:52:11 2012
From: iamkyungjoo at gmail.com (Kyungjoo Kim)
Date: Thu, 26 Apr 2012 20:52:11 -0500
Subject: [petsc-users] Can I compile PETSC --with-mpi=0 when I use hypre ???
Message-ID: <DA716802-3CD2-4624-A6E8-D90A8BF931A9@gmail.com>

Hello,


While compiling PETSC without mpi, it complains that he needs mpi for hypre.
But I already install hypre also without mpi (hypre give me a option that I can install without mpi).

So, without mpi, I can compile PETSC but it removes all hypre interfaces, too. 

is there any detouring way ? Somehow my mac does not like mpi, so I would like to have no-mpi version for testing in my mac.




From bsmith at mcs.anl.gov  Thu Apr 26 20:55:31 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 26 Apr 2012 20:55:31 -0500
Subject: [petsc-users] Can I compile PETSC --with-mpi=0 when I use hypre
	???
In-Reply-To: <DA716802-3CD2-4624-A6E8-D90A8BF931A9@gmail.com>
References: <DA716802-3CD2-4624-A6E8-D90A8BF931A9@gmail.com>
Message-ID: <C48C1E09-C1A5-4120-BEE6-9CE6B10F1988@mcs.anl.gov>


  Please send to petsc-maint at mcs.anl.gov HOW it "complains" that it needs mpi for hypre. (Send ALL error messages, configure.log make.log)

  We've never tested this configuration, there may be some flags that need to be changed to get the build going; and we don't know if the build will work. But we can get started on helping you test it with a complete error message to petsc-maint

   Barry

On Apr 26, 2012, at 8:52 PM, Kyungjoo Kim wrote:

> Hello,
> 
> 
> While compiling PETSC without mpi, it complains that he needs mpi for hypre.
> But I already install hypre also without mpi (hypre give me a option that I can install without mpi).
> 
> So, without mpi, I can compile PETSC but it removes all hypre interfaces, too. 
> 
> is there any detouring way ? Somehow my mac does not like mpi, so I would like to have no-mpi version for testing in my mac.
> 
> 
> 


From balay at mcs.anl.gov  Thu Apr 26 21:09:31 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 26 Apr 2012 21:09:31 -0500 (CDT)
Subject: [petsc-users] Can I compile PETSC --with-mpi=0 when I use hypre
 ???
In-Reply-To: <C48C1E09-C1A5-4120-BEE6-9CE6B10F1988@mcs.anl.gov>
References: <DA716802-3CD2-4624-A6E8-D90A8BF931A9@gmail.com>
	<C48C1E09-C1A5-4120-BEE6-9CE6B10F1988@mcs.anl.gov>
Message-ID: <alpine.LFD.2.02.1204262106130.27238@asterix>

getting this working properly might require work - but as a hack - you
can edit config/PETSc/packages/hypre.py and change:

    self.deps       = [self.mpi,self.blasLapack]  
to
    self.deps       = [self.blasLapack]  

> Somehow my mac does not like mpi, so I would like to have no-mpi version for testing in my mac.

Don't see why mpi would be a problem - everyone over here with mac
uses mpi. If you send us logs to petsc-mait with the mpi problems [try
--download-mpich] - we can suggest fixes.

Satish


On Thu, 26 Apr 2012, Barry Smith wrote:

> 
>   Please send to petsc-maint at mcs.anl.gov HOW it "complains" that it needs mpi for hypre. (Send ALL error messages, configure.log make.log)
> 
>   We've never tested this configuration, there may be some flags that need to be changed to get the build going; and we don't know if the build will work. But we can get started on helping you test it with a complete error message to petsc-maint
> 
>    Barry
> 
> On Apr 26, 2012, at 8:52 PM, Kyungjoo Kim wrote:
> 
> > Hello,
> > 
> > 
> > While compiling PETSC without mpi, it complains that he needs mpi for hypre.
> > But I already install hypre also without mpi (hypre give me a option that I can install without mpi).
> > 
> > So, without mpi, I can compile PETSC but it removes all hypre interfaces, too. 
> > 
> > is there any detouring way ? Somehow my mac does not like mpi, so I would like to have no-mpi version for testing in my mac.
> > 
> > 
> > 
> 
> 


From balay at mcs.anl.gov  Fri Apr 27 00:08:45 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 27 Apr 2012 00:08:45 -0500 (CDT)
Subject: [petsc-users] Can I compile PETSC --with-mpi=0 when I use hypre
 ???
In-Reply-To: <C48C1E09-C1A5-4120-BEE6-9CE6B10F1988@mcs.anl.gov>
References: <DA716802-3CD2-4624-A6E8-D90A8BF931A9@gmail.com>
	<C48C1E09-C1A5-4120-BEE6-9CE6B10F1988@mcs.anl.gov>
Message-ID: <alpine.LFD.2.02.1204262344060.27238@asterix>

On Thu, 26 Apr 2012, Barry Smith wrote:

>   We've never tested this configuration, there may be some flags that need to be changed to get the build going; and we don't know if the build will work. But we can get started on helping you test it with a complete error message to petsc-maint

Ok - we don't support this configuration because both PETSc and hypre
internally use mpiuni type code. And when you combine
petsc+petsc-mpiuni with hypre+hypre-mpiuni petsc-mpiuni and
hypre-mpiuni conflict with each other - and things break.

[this is similar to mixing mpich compiled code with openmpi compiled
code]

So we'll always require a proper (common) MPI for using hypre from
PETSc.

Satish

From dominik at itis.ethz.ch  Fri Apr 27 01:00:45 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Fri, 27 Apr 2012 08:00:45 +0200
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <D34711D9-5687-4EA7-98F5-C105ECE84966@mcs.anl.gov>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
	<D34711D9-5687-4EA7-98F5-C105ECE84966@mcs.anl.gov>
Message-ID: <CALm7otp0fUFgOVmMDi3Ltkj0yPwmDgjE1DfBkKFOcryWtVXrwg@mail.gmail.com>

> ?You set yourself up for a very painful life. It is simply insane to have a code that cannot run with tiny problems for debugging and understanding and similarly run sequentially. Parallel computing with MPI is HARD and the only way to make it at all manageable is to organize the code so that it is easy to debug and understand. ?You may think it is silly to waste time making the code run on small problems sequentially since you want to run big problems but it is not silly, it is crucial. You would already have debugged and fixed this problem if you could run the code sequentially on tiny problems. The extra time in making the code debuggable is much much less than the extra time wasted futzing around trying to find a bug with no tools to debug.
>
> ? ?The -snes_type test option has another option -snes_test_display which will display the two Jacobians and the difference between them allowing one to see which entries in the Jacobian (if not all of them) are wrong. ?If you could run on a small problem sequentially you would run with these options, see what Jacobian entries are wrong and look back at the code that generates those entries and likely see the bug.
>
> ? Barry

You are perfectly right about using very small problems for debugging.
But clearly, the discussion has diverged from the original topic.
Coming therefore back to my very first email in this thread: I have
properly converging non-trivial cases with the Newton solver using
analytically derived Jacobian. I want to reproduce these results
WITHOUT using my Jacobian at all with the secant method (i.e. F'(x)
approximated with finite differences)). You say my Jacobian is wrong,
and maybe you are right and maybe not. But it does not matter: I do
not want to use my Jacobian anyway, hoping for the petsc's secant
method to approximate it. Does petsc have a secant method and does
-snes_mf_operator option enables it? Or it only works as a
preconditioner, requiring the Jacobian in any case?

Thanks,
Dominik

From jedbrown at mcs.anl.gov  Fri Apr 27 01:12:43 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 27 Apr 2012 01:12:43 -0500
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CALm7otp0fUFgOVmMDi3Ltkj0yPwmDgjE1DfBkKFOcryWtVXrwg@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
	<D34711D9-5687-4EA7-98F5-C105ECE84966@mcs.anl.gov>
	<CALm7otp0fUFgOVmMDi3Ltkj0yPwmDgjE1DfBkKFOcryWtVXrwg@mail.gmail.com>
Message-ID: <CAM9tzSnz=DUfhVS4B0m-O__r7mWByQRDTG2m=QD0JTWx=VX9Sw@mail.gmail.com>

On Fri, Apr 27, 2012 at 01:00, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I have
> properly converging non-trivial cases with the Newton solver using
> analytically derived Jacobian. I want to reproduce these results
> WITHOUT using my Jacobian at all with the secant method (i.e. F'(x)
> approximated with finite differences)). You say my Jacobian is wrong,
> and maybe you are right and maybe not. But it does not matter: I do
> not want to use my Jacobian anyway, hoping for the petsc's secant
> method to approximate it. Does petsc have a secant method and does
> -snes_mf_operator option enables it? Or it only works as a
> preconditioner, requiring the Jacobian in any case?
>

-snes_mf_operator is NOT the secant method. Secant is actually only a 1D
method, the generalizations to multiple dimensions are usually referred to
as quasi-Newton, of which BFGS is perhaps the most popular. Those are
-snes_type qn, most of that work is new in petsc-dev.

-snes_mf_operator is Newton, but with the *operator* applied using
matrix-free finite differences. It uses whatever you provide as the
preconditioning matrix.

As for what is going wrong, some of the symptoms you report often occur
when the original system is poorly scaled. What units are you using for
state variables and residuals? This has lots of useful tips

http://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging

See these entries relating to inaccurate finite differencing

* Run with -snes_type test -snes_test_display to see if the Jacobian you
are using is wrong. Compare the output when you add -mat_fd_type ds to see
if the result is sensitive to the choice of differencing parameter.
* Run with -snes_mf_operator -pc_type lu to see if the Jacobian you are
using is wrong. If the problem is too large for a direct solve, try
-snes_mf_operator -pc_type ksp -ksp_ksp_rtol 1e-12. Compare the output when
you add -mat_mffd_type ds to see if the result is sensitive to choice of
differencing parameter.
* Run with quad precision (./configure --with-precision=__float128
--download-f2cblaslapack with PETSc 3.2 and later, needs version 4.6 or
later GNU compilers)
* Change the units (nondimensionalization), boundary condition scaling, or
formulation so that the Jacobian is better conditioned.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/09a8a72d/attachment.htm>

From knepley at gmail.com  Fri Apr 27 06:08:29 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 07:08:29 -0400
Subject: [petsc-users] help needed for -snes_mf_operator
In-Reply-To: <CAM9tzSnz=DUfhVS4B0m-O__r7mWByQRDTG2m=QD0JTWx=VX9Sw@mail.gmail.com>
References: <CALm7otrmAcxUGF=fBvxp0bcKGA7DjEbJ2bVLjQo=F3iCxJcN5A@mail.gmail.com>
	<CALm7otpyhLh_VMAer-ditTdKrL+SMhrb8qq3F0w816v=TB4vdg@mail.gmail.com>
	<D34711D9-5687-4EA7-98F5-C105ECE84966@mcs.anl.gov>
	<CALm7otp0fUFgOVmMDi3Ltkj0yPwmDgjE1DfBkKFOcryWtVXrwg@mail.gmail.com>
	<CAM9tzSnz=DUfhVS4B0m-O__r7mWByQRDTG2m=QD0JTWx=VX9Sw@mail.gmail.com>
Message-ID: <CAMYG4GmeSe2bdk_rTYXGJOd62ogFw21On30BZBRS-aU0sgAj5Q@mail.gmail.com>

On Fri, Apr 27, 2012 at 2:12 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Fri, Apr 27, 2012 at 01:00, Dominik Szczerba <dominik at itis.ethz.ch>wrote:
>
>> I have
>> properly converging non-trivial cases with the Newton solver using
>> analytically derived Jacobian. I want to reproduce these results
>> WITHOUT using my Jacobian at all with the secant method (i.e. F'(x)
>> approximated with finite differences)). You say my Jacobian is wrong,
>> and maybe you are right and maybe not. But it does not matter: I do
>> not want to use my Jacobian anyway, hoping for the petsc's secant
>> method to approximate it. Does petsc have a secant method and does
>> -snes_mf_operator option enables it? Or it only works as a
>> preconditioner, requiring the Jacobian in any case?
>>
>
> -snes_mf_operator is NOT the secant method. Secant is actually only a 1D
> method, the generalizations to multiple dimensions are usually referred to
> as quasi-Newton, of which BFGS is perhaps the most popular. Those are
> -snes_type qn, most of that work is new in petsc-dev.
>
> -snes_mf_operator is Newton, but with the *operator* applied using
> matrix-free finite differences. It uses whatever you provide as the
> preconditioning matrix.
>
> As for what is going wrong, some of the symptoms you report often occur
> when the original system is poorly scaled. What units are you using for
> state variables and residuals? This has lots of useful tips
>

I disagree here. I think what is wrong is

  You are screwing up providing the preconditioner matrix. There is no
other way to explain the GMRES performance.

Try using -snes_mf and it will converge, just slowly. This is a simple
thing of correcting an error in providing that matrix I think.

    Matt


>
> http://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging
>
> See these entries relating to inaccurate finite differencing
>
> * Run with -snes_type test -snes_test_display to see if the Jacobian you
> are using is wrong. Compare the output when you add -mat_fd_type ds to see
> if the result is sensitive to the choice of differencing parameter.
> * Run with -snes_mf_operator -pc_type lu to see if the Jacobian you are
> using is wrong. If the problem is too large for a direct solve, try
> -snes_mf_operator -pc_type ksp -ksp_ksp_rtol 1e-12. Compare the output when
> you add -mat_mffd_type ds to see if the result is sensitive to choice of
> differencing parameter.
> * Run with quad precision (./configure --with-precision=__float128
> --download-f2cblaslapack with PETSc 3.2 and later, needs version 4.6 or
> later GNU compilers)
> * Change the units (nondimensionalization), boundary condition scaling, or
> formulation so that the Jacobian is better conditioned.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/cafd3d1c/attachment.htm>

From dominik at itis.ethz.ch  Fri Apr 27 06:48:52 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Fri, 27 Apr 2012 13:48:52 +0200
Subject: [petsc-users] next release
Message-ID: <CALm7otpo9kNwPy7p7FkUjpS2t-S4sBkact8CghVLWobessSJDA@mail.gmail.com>

Hi,

Is it at least approximately known when the petsc-dev will be out as
the next official release?

Thanks
Dominik

From amesga1 at tigers.lsu.edu  Fri Apr 27 08:19:43 2012
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Fri, 27 Apr 2012 08:19:43 -0500
Subject: [petsc-users] writing native PETSc binaries in python
Message-ID: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>

Dear all,

I was wondering if there is way to write numpy arrays as native PETSc
 binaries in python so I can open them inside my program with VecView?

Thanks,
Ata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/b58477c7/attachment.htm>

From knepley at gmail.com  Fri Apr 27 08:25:01 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 09:25:01 -0400
Subject: [petsc-users] writing native PETSc binaries in python
In-Reply-To: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>
References: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>
Message-ID: <CAMYG4GnnYvj_36RhLr+O-bwFkPtP0nSxjaRfjP=mvnpOySC07g@mail.gmail.com>

On Fri, Apr 27, 2012 at 9:19 AM, Ataollah Mesgarnejad <
amesga1 at tigers.lsu.edu> wrote:

> Dear all,
>
> I was wondering if there is way to write numpy arrays as native PETSc
>  binaries in python so I can open them inside my program with VecView?
>

I think what you want is to create a Vec using that array in petsc4py.

   Matt


> Thanks,
> Ata
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/bab6eda8/attachment-0001.htm>

From jiangwen84 at gmail.com  Fri Apr 27 08:53:57 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Fri, 27 Apr 2012 09:53:57 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern
Message-ID: <CAMJxm+D7kwTZPQ_7xwEKueL2z=zKL6a4A1BBTQXxACj_UKJF-Q@mail.gmail.com>

Hi,

I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
Within Newton-Raphson iteration, the system with same sparse structure but
different values will be solved many times. And I specify the
preconditioning matrix having same nonzero pattern. The first time solving
costs around 900 seconds and later solving only takes around 200 seconds.
So I am wondering why the time differs that much. By setting the same
nonzero pattern of pc, which of the mumps control parameters does PETSc
change? Thanks.

Regards,
Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/7bd8e1ad/attachment.htm>

From knepley at gmail.com  Fri Apr 27 09:11:27 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 10:11:27 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern
In-Reply-To: <CAMJxm+D7kwTZPQ_7xwEKueL2z=zKL6a4A1BBTQXxACj_UKJF-Q@mail.gmail.com>
References: <CAMJxm+D7kwTZPQ_7xwEKueL2z=zKL6a4A1BBTQXxACj_UKJF-Q@mail.gmail.com>
Message-ID: <CAMYG4GnbKCA7DOXAP=_+YHkDWaDF94TFjqr2eaZ3dQnD7Vy_Dg@mail.gmail.com>

On Fri, Apr 27, 2012 at 9:53 AM, Wen Jiang <jiangwen84 at gmail.com> wrote:

> Hi,
>
> I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
> Within Newton-Raphson iteration, the system with same sparse structure but
> different values will be solved many times. And I specify the
> preconditioning matrix having same nonzero pattern. The first time solving
> costs around 900 seconds and later solving only takes around 200 seconds.
> So I am wondering why the time differs that much. By setting the same
> nonzero pattern of pc, which of the mumps control parameters does PETSc
> change? Thanks.
>

The difference is that you do not have to perform the symbolic
factorization again if the nonzero pattern does not change,
just the numeric factorization. The symbolic factorization is the costly
step.

   Matt


> Regards,
> Wen
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/f148b130/attachment.htm>

From ucemckl at ucl.ac.uk  Fri Apr 27 05:26:33 2012
From: ucemckl at ucl.ac.uk (Christian Klettner)
Date: Fri, 27 Apr 2012 11:26:33 +0100
Subject: [petsc-users] High Re CFD method.
Message-ID: <ba0801c9e3bd11beb658cbec10d24a7f.squirrel@www.squirrelmail.ucl.ac.uk>

Dear Petsc users,

I know this is off topic but the people reading these questions are well
placed to answer the question, being actual CFD developers. Over the last
five years our group has developed a CFD code (using PETSc for parallel
and vectors) based on the characteristic based split scheme to solve the
incompressible Navier-Stokes equation (2D-3D) in a finite element
framework using unstructured meshes. It is first order accurate in time
and space. This has proved effective for high resolution, low Reynolds
number complex geometry flows (e.g. groups of bodies).

We are hoping to move onto higher Reynolds number flows, with the
intention of using a large eddy simulation model for the turbulence. The
goal is to model the flow in a hospital room with typical Reynolds numbers
of 10^5. At present we are not convinced that the CBS scheme is the best
scheme for these types of flows and are looking for other peoples opinions
on alternative methods. Does anyone have any advice in this direction?

Best regards,
Christian


From amesga1 at tigers.lsu.edu  Fri Apr 27 10:28:41 2012
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Fri, 27 Apr 2012 10:28:41 -0500
Subject: [petsc-users] writing native PETSc binaries in python
In-Reply-To: <CAMYG4GnnYvj_36RhLr+O-bwFkPtP0nSxjaRfjP=mvnpOySC07g@mail.gmail.com>
References: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>
	<CAMYG4GnnYvj_36RhLr+O-bwFkPtP0nSxjaRfjP=mvnpOySC07g@mail.gmail.com>
Message-ID: <CAC+VmGdP1V3DFrfLWextGVU23NRHtKwUoqcD4Ai__mwTdd1ZGw@mail.gmail.com>

Thanks for the replies:

Just a bug report with petsc4py:

petsc4py which downloads with --download-petsc4py=1 is the older v1.1.1
which doesn't compile with petsc-dev?

Thanks,
Ata

On Fri, Apr 27, 2012 at 8:25 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Apr 27, 2012 at 9:19 AM, Ataollah Mesgarnejad <
> amesga1 at tigers.lsu.edu> wrote:
>
>> Dear all,
>>
>> I was wondering if there is way to write numpy arrays as native PETSc
>>  binaries in python so I can open them inside my program with VecView?
>>
>
> I think what you want is to create a Vec using that array in petsc4py.
>
>    Matt
>
>
>> Thanks,
>> Ata
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
A. Mesgarnejad
PhD Student, Research Assistant
Mechanical Engineering Department
Louisiana State University
2203 Patrick F. Taylor Hall
Baton Rouge, La 70803
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/cca3e1b6/attachment.htm>

From jiangwen84 at gmail.com  Fri Apr 27 12:49:21 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Fri, 27 Apr 2012 13:49:21 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern
Message-ID: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>

Thanks for your reply. I also tested the exactly same problem with SUPERLU.
The mumps and superlu dist both are using sequential symbolic
factorization. However, superlu dist takes only 20 seconds but mumps takes
almost 700 seconds. I am wondering whether such a big difference is
possible. Do those two direct solver use quite different algorithm?

And also since I might have the same nonzero structure system to be solved
many times at different places. I am wondering whether I could save the
symbolic factorization output somewhere and then read them as the input for
future solving. Thanks.

Regards,
Wen



On Fri, Apr 27, 2012 at 1:06 PM, <petsc-users-request at mcs.anl.gov> wrote:

> Send petsc-users mailing list submissions to
>        petsc-users at mcs.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> or, via email, send a message with subject or body 'help' to
>        petsc-users-request at mcs.anl.gov
>
> You can reach the person managing the list at
>        petsc-users-owner at mcs.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of petsc-users digest..."
>
>
> Today's Topics:
>
>   1.  mumps solve with same nonzero pattern (Wen Jiang)
>   2. Re:  mumps solve with same nonzero pattern (Matthew Knepley)
>   3.  High Re CFD method. (Christian Klettner)
>   4. Re:  writing native PETSc binaries in python
>      (Ataollah Mesgarnejad)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 27 Apr 2012 09:53:57 -0400
> From: Wen Jiang <jiangwen84 at gmail.com>
> Subject: [petsc-users] mumps solve with same nonzero pattern
> To: petsc-users at mcs.anl.gov
> Message-ID:
>        <CAMJxm+D7kwTZPQ_7xwEKueL2z=zKL6a4A1BBTQXxACj_UKJF-Q at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi,
>
> I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
> Within Newton-Raphson iteration, the system with same sparse structure but
> different values will be solved many times. And I specify the
> preconditioning matrix having same nonzero pattern. The first time solving
> costs around 900 seconds and later solving only takes around 200 seconds.
> So I am wondering why the time differs that much. By setting the same
> nonzero pattern of pc, which of the mumps control parameters does PETSc
> change? Thanks.
>
> Regards,
> Wen
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/7bd8e1ad/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Fri, 27 Apr 2012 10:11:27 -0400
> From: Matthew Knepley <knepley at gmail.com>
> Subject: Re: [petsc-users] mumps solve with same nonzero pattern
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAMYG4GnbKCA7DOXAP=_+YHkDWaDF94TFjqr2eaZ3dQnD7Vy_Dg at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Fri, Apr 27, 2012 at 9:53 AM, Wen Jiang <jiangwen84 at gmail.com> wrote:
>
> > Hi,
> >
> > I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
> > Within Newton-Raphson iteration, the system with same sparse structure
> but
> > different values will be solved many times. And I specify the
> > preconditioning matrix having same nonzero pattern. The first time
> solving
> > costs around 900 seconds and later solving only takes around 200 seconds.
> > So I am wondering why the time differs that much. By setting the same
> > nonzero pattern of pc, which of the mumps control parameters does PETSc
> > change? Thanks.
> >
>
> The difference is that you do not have to perform the symbolic
> factorization again if the nonzero pattern does not change,
> just the numeric factorization. The symbolic factorization is the costly
> step.
>
>   Matt
>
>
> > Regards,
> > Wen
> >
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/f148b130/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 3
> Date: Fri, 27 Apr 2012 11:26:33 +0100
> From: "Christian Klettner" <ucemckl at ucl.ac.uk>
> Subject: [petsc-users] High Re CFD method.
> To: petsc-users at mcs.anl.gov
> Message-ID:
>        <
> ba0801c9e3bd11beb658cbec10d24a7f.squirrel at www.squirrelmail.ucl.ac.uk>
> Content-Type: text/plain;charset=iso-8859-1
>
> Dear Petsc users,
>
> I know this is off topic but the people reading these questions are well
> placed to answer the question, being actual CFD developers. Over the last
> five years our group has developed a CFD code (using PETSc for parallel
> and vectors) based on the characteristic based split scheme to solve the
> incompressible Navier-Stokes equation (2D-3D) in a finite element
> framework using unstructured meshes. It is first order accurate in time
> and space. This has proved effective for high resolution, low Reynolds
> number complex geometry flows (e.g. groups of bodies).
>
> We are hoping to move onto higher Reynolds number flows, with the
> intention of using a large eddy simulation model for the turbulence. The
> goal is to model the flow in a hospital room with typical Reynolds numbers
> of 10^5. At present we are not convinced that the CBS scheme is the best
> scheme for these types of flows and are looking for other peoples opinions
> on alternative methods. Does anyone have any advice in this direction?
>
> Best regards,
> Christian
>
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 27 Apr 2012 10:28:41 -0500
> From: Ataollah Mesgarnejad <amesga1 at tigers.lsu.edu>
> Subject: Re: [petsc-users] writing native PETSc binaries in python
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Cc: petsc-dev at mcs.anl.gov
> Message-ID:
>        <CAC+VmGdP1V3DFrfLWextGVU23NRHtKwUoqcD4Ai__mwTdd1ZGw at mail.gmail.com
> >
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thanks for the replies:
>
> Just a bug report with petsc4py:
>
> petsc4py which downloads with --download-petsc4py=1 is the older v1.1.1
> which doesn't compile with petsc-dev?
>
> Thanks,
> Ata
>
> On Fri, Apr 27, 2012 at 8:25 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
> > On Fri, Apr 27, 2012 at 9:19 AM, Ataollah Mesgarnejad <
> > amesga1 at tigers.lsu.edu> wrote:
> >
> >> Dear all,
> >>
> >> I was wondering if there is way to write numpy arrays as native PETSc
> >>  binaries in python so I can open them inside my program with VecView?
> >>
> >
> > I think what you want is to create a Vec using that array in petsc4py.
> >
> >    Matt
> >
> >
> >> Thanks,
> >> Ata
> >>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert Wiener
> >
>
>
>
> --
> A. Mesgarnejad
> PhD Student, Research Assistant
> Mechanical Engineering Department
> Louisiana State University
> 2203 Patrick F. Taylor Hall
> Baton Rouge, La 70803
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/cca3e1b6/attachment-0001.htm
> >
>
> ------------------------------
>
> _______________________________________________
> petsc-users mailing list
> petsc-users at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>
>
> End of petsc-users Digest, Vol 40, Issue 102
> ********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/49e51f1a/attachment.htm>

From bsmith at mcs.anl.gov  Fri Apr 27 13:12:19 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 27 Apr 2012 13:12:19 -0500
Subject: [petsc-users] writing native PETSc binaries in python
In-Reply-To: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>
References: <CAC+VmGciHm8rOnzwLHb23Jxrub2Qs8kNA0d-Nmj-cZ5ZYZeoaQ@mail.gmail.com>
Message-ID: <64965AEF-7161-455D-8EF8-DA3DC01FC0A9@mcs.anl.gov>


On Apr 27, 2012, at 8:19 AM, Ataollah Mesgarnejad wrote:

> Dear all,
> 
> I was wondering if there is way to write numpy arrays as native PETSc  binaries in python so I can open them inside my program with VecView?

   In petsc-dev in the directory bin/pythonscripts is PetscBinaryIO.py which has python utilities for reading and writing python arrays to PETSc binary format.

    Barry

> 
> Thanks,
> Ata


From knepley at gmail.com  Fri Apr 27 13:24:43 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 14:24:43 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern
In-Reply-To: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
References: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
Message-ID: <CAMYG4GmUVgPnrP7MEQPoHw4E9q9UKYbLxEJZ0_-c_8jv=YdDeQ@mail.gmail.com>

On Fri, Apr 27, 2012 at 1:49 PM, Wen Jiang <jiangwen84 at gmail.com> wrote:

> Thanks for your reply. I also tested the exactly same problem with
> SUPERLU. The mumps and superlu dist both are using sequential symbolic
> factorization. However, superlu dist takes only 20 seconds but mumps takes
> almost 700 seconds. I am wondering whether such a big difference is
> possible. Do those two direct solver use quite different algorithm?
>
> And also since I might have the same nonzero structure system to be solved
> many times at different places. I am wondering whether I could save the
> symbolic factorization output somewhere and then read them as the input for
> future solving. Thanks.
>

No. Those factorizations an internal data structures to the packages, so we
can't write them out.

   Matt


> Regards,
> Wen
>
>
>
> On Fri, Apr 27, 2012 at 1:06 PM, <petsc-users-request at mcs.anl.gov> wrote:
>
>> Send petsc-users mailing list submissions to
>>        petsc-users at mcs.anl.gov
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>        https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>> or, via email, send a message with subject or body 'help' to
>>        petsc-users-request at mcs.anl.gov
>>
>> You can reach the person managing the list at
>>        petsc-users-owner at mcs.anl.gov
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of petsc-users digest..."
>>
>>
>> Today's Topics:
>>
>>   1.  mumps solve with same nonzero pattern (Wen Jiang)
>>   2. Re:  mumps solve with same nonzero pattern (Matthew Knepley)
>>   3.  High Re CFD method. (Christian Klettner)
>>   4. Re:  writing native PETSc binaries in python
>>      (Ataollah Mesgarnejad)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 27 Apr 2012 09:53:57 -0400
>> From: Wen Jiang <jiangwen84 at gmail.com>
>> Subject: [petsc-users] mumps solve with same nonzero pattern
>> To: petsc-users at mcs.anl.gov
>> Message-ID:
>>        <CAMJxm+D7kwTZPQ_7xwEKueL2z=
>> zKL6a4A1BBTQXxACj_UKJF-Q at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi,
>>
>> I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
>> Within Newton-Raphson iteration, the system with same sparse structure but
>> different values will be solved many times. And I specify the
>> preconditioning matrix having same nonzero pattern. The first time solving
>> costs around 900 seconds and later solving only takes around 200 seconds.
>> So I am wondering why the time differs that much. By setting the same
>> nonzero pattern of pc, which of the mumps control parameters does PETSc
>> change? Thanks.
>>
>> Regards,
>> Wen
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/7bd8e1ad/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Fri, 27 Apr 2012 10:11:27 -0400
>> From: Matthew Knepley <knepley at gmail.com>
>> Subject: Re: [petsc-users] mumps solve with same nonzero pattern
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Message-ID:
>>        <CAMYG4GnbKCA7DOXAP=_+
>> YHkDWaDF94TFjqr2eaZ3dQnD7Vy_Dg at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> On Fri, Apr 27, 2012 at 9:53 AM, Wen Jiang <jiangwen84 at gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
>> > Within Newton-Raphson iteration, the system with same sparse structure
>> but
>> > different values will be solved many times. And I specify the
>> > preconditioning matrix having same nonzero pattern. The first time
>> solving
>> > costs around 900 seconds and later solving only takes around 200
>> seconds.
>> > So I am wondering why the time differs that much. By setting the same
>> > nonzero pattern of pc, which of the mumps control parameters does PETSc
>> > change? Thanks.
>> >
>>
>> The difference is that you do not have to perform the symbolic
>> factorization again if the nonzero pattern does not change,
>> just the numeric factorization. The symbolic factorization is the costly
>> step.
>>
>>   Matt
>>
>>
>> > Regards,
>> > Wen
>> >
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/f148b130/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Fri, 27 Apr 2012 11:26:33 +0100
>> From: "Christian Klettner" <ucemckl at ucl.ac.uk>
>> Subject: [petsc-users] High Re CFD method.
>> To: petsc-users at mcs.anl.gov
>> Message-ID:
>>        <
>> ba0801c9e3bd11beb658cbec10d24a7f.squirrel at www.squirrelmail.ucl.ac.uk>
>> Content-Type: text/plain;charset=iso-8859-1
>>
>> Dear Petsc users,
>>
>> I know this is off topic but the people reading these questions are well
>> placed to answer the question, being actual CFD developers. Over the last
>> five years our group has developed a CFD code (using PETSc for parallel
>> and vectors) based on the characteristic based split scheme to solve the
>> incompressible Navier-Stokes equation (2D-3D) in a finite element
>> framework using unstructured meshes. It is first order accurate in time
>> and space. This has proved effective for high resolution, low Reynolds
>> number complex geometry flows (e.g. groups of bodies).
>>
>> We are hoping to move onto higher Reynolds number flows, with the
>> intention of using a large eddy simulation model for the turbulence. The
>> goal is to model the flow in a hospital room with typical Reynolds numbers
>> of 10^5. At present we are not convinced that the CBS scheme is the best
>> scheme for these types of flows and are looking for other peoples opinions
>> on alternative methods. Does anyone have any advice in this direction?
>>
>> Best regards,
>> Christian
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Fri, 27 Apr 2012 10:28:41 -0500
>> From: Ataollah Mesgarnejad <amesga1 at tigers.lsu.edu>
>> Subject: Re: [petsc-users] writing native PETSc binaries in python
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Cc: petsc-dev at mcs.anl.gov
>> Message-ID:
>>        <
>> CAC+VmGdP1V3DFrfLWextGVU23NRHtKwUoqcD4Ai__mwTdd1ZGw at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Thanks for the replies:
>>
>> Just a bug report with petsc4py:
>>
>> petsc4py which downloads with --download-petsc4py=1 is the older v1.1.1
>> which doesn't compile with petsc-dev?
>>
>> Thanks,
>> Ata
>>
>> On Fri, Apr 27, 2012 at 8:25 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>> > On Fri, Apr 27, 2012 at 9:19 AM, Ataollah Mesgarnejad <
>> > amesga1 at tigers.lsu.edu> wrote:
>> >
>> >> Dear all,
>> >>
>> >> I was wondering if there is way to write numpy arrays as native PETSc
>> >>  binaries in python so I can open them inside my program with VecView?
>> >>
>> >
>> > I think what you want is to create a Vec using that array in petsc4py.
>> >
>> >    Matt
>> >
>> >
>> >> Thanks,
>> >> Ata
>> >>
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments is infinitely more interesting than any results to which
>> their
>> > experiments lead.
>> > -- Norbert Wiener
>> >
>>
>>
>>
>> --
>> A. Mesgarnejad
>> PhD Student, Research Assistant
>> Mechanical Engineering Department
>> Louisiana State University
>> 2203 Patrick F. Taylor Hall
>> Baton Rouge, La 70803
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/cca3e1b6/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> petsc-users mailing list
>> petsc-users at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>>
>>
>> End of petsc-users Digest, Vol 40, Issue 102
>> ********************************************
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/f40e5557/attachment-0001.htm>

From hzhang at mcs.anl.gov  Fri Apr 27 14:09:22 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Fri, 27 Apr 2012 14:09:22 -0500
Subject: [petsc-users] mumps solve with same nonzero pattern
In-Reply-To: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
References: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
Message-ID: <CAGCphBt68k7GUnnwScnq_TMNCCdPUUD+ccOqMuv3N9yy9Yh2oA@mail.gmail.com>

Wen :

> Thanks for your reply. I also tested the exactly same problem with
> SUPERLU. The mumps and superlu dist both are using sequential symbolic
> factorization. However, superlu dist takes only 20 seconds but mumps takes
> almost 700 seconds. I am wondering whether such a big difference is
> possible. Do those two direct solver use quite different algorithm?
>
This is weird.  Try
1) increase work  space with
-mat_mumps_icntl_14 50 (default is 20)
2) different matrix orderings with
-mat_mumps_icntl_7 2 (or number from 0 to 6)

Run your code with '-log_summary' and see which routine causes this huge
difference.

Hong

>
> And also since I might have the same nonzero structure system to be solved
> many times at different places. I am wondering whether I could save the
> symbolic factorization output somewhere and then read them as the input for
> future solving. Thanks.
>
> Regards,
> Wen
>
>
>
> On Fri, Apr 27, 2012 at 1:06 PM, <petsc-users-request at mcs.anl.gov> wrote:
>
>> Send petsc-users mailing list submissions to
>>        petsc-users at mcs.anl.gov
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>        https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>> or, via email, send a message with subject or body 'help' to
>>        petsc-users-request at mcs.anl.gov
>>
>> You can reach the person managing the list at
>>        petsc-users-owner at mcs.anl.gov
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of petsc-users digest..."
>>
>>
>> Today's Topics:
>>
>>   1.  mumps solve with same nonzero pattern (Wen Jiang)
>>   2. Re:  mumps solve with same nonzero pattern (Matthew Knepley)
>>   3.  High Re CFD method. (Christian Klettner)
>>   4. Re:  writing native PETSc binaries in python
>>      (Ataollah Mesgarnejad)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 27 Apr 2012 09:53:57 -0400
>> From: Wen Jiang <jiangwen84 at gmail.com>
>> Subject: [petsc-users] mumps solve with same nonzero pattern
>> To: petsc-users at mcs.anl.gov
>> Message-ID:
>>        <CAMJxm+D7kwTZPQ_7xwEKueL2z=
>> zKL6a4A1BBTQXxACj_UKJF-Q at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> Hi,
>>
>> I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
>> Within Newton-Raphson iteration, the system with same sparse structure but
>> different values will be solved many times. And I specify the
>> preconditioning matrix having same nonzero pattern. The first time solving
>> costs around 900 seconds and later solving only takes around 200 seconds.
>> So I am wondering why the time differs that much. By setting the same
>> nonzero pattern of pc, which of the mumps control parameters does PETSc
>> change? Thanks.
>>
>> Regards,
>> Wen
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/7bd8e1ad/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Fri, 27 Apr 2012 10:11:27 -0400
>> From: Matthew Knepley <knepley at gmail.com>
>> Subject: Re: [petsc-users] mumps solve with same nonzero pattern
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Message-ID:
>>        <CAMYG4GnbKCA7DOXAP=_+
>> YHkDWaDF94TFjqr2eaZ3dQnD7Vy_Dg at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> On Fri, Apr 27, 2012 at 9:53 AM, Wen Jiang <jiangwen84 at gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am using mumps in PETSc to solve a 0.2 million system on 128 cores.
>> > Within Newton-Raphson iteration, the system with same sparse structure
>> but
>> > different values will be solved many times. And I specify the
>> > preconditioning matrix having same nonzero pattern. The first time
>> solving
>> > costs around 900 seconds and later solving only takes around 200
>> seconds.
>> > So I am wondering why the time differs that much. By setting the same
>> > nonzero pattern of pc, which of the mumps control parameters does PETSc
>> > change? Thanks.
>> >
>>
>> The difference is that you do not have to perform the symbolic
>> factorization again if the nonzero pattern does not change,
>> just the numeric factorization. The symbolic factorization is the costly
>> step.
>>
>>   Matt
>>
>>
>> > Regards,
>> > Wen
>> >
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/f148b130/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Fri, 27 Apr 2012 11:26:33 +0100
>> From: "Christian Klettner" <ucemckl at ucl.ac.uk>
>> Subject: [petsc-users] High Re CFD method.
>> To: petsc-users at mcs.anl.gov
>> Message-ID:
>>        <
>> ba0801c9e3bd11beb658cbec10d24a7f.squirrel at www.squirrelmail.ucl.ac.uk>
>> Content-Type: text/plain;charset=iso-8859-1
>>
>> Dear Petsc users,
>>
>> I know this is off topic but the people reading these questions are well
>> placed to answer the question, being actual CFD developers. Over the last
>> five years our group has developed a CFD code (using PETSc for parallel
>> and vectors) based on the characteristic based split scheme to solve the
>> incompressible Navier-Stokes equation (2D-3D) in a finite element
>> framework using unstructured meshes. It is first order accurate in time
>> and space. This has proved effective for high resolution, low Reynolds
>> number complex geometry flows (e.g. groups of bodies).
>>
>> We are hoping to move onto higher Reynolds number flows, with the
>> intention of using a large eddy simulation model for the turbulence. The
>> goal is to model the flow in a hospital room with typical Reynolds numbers
>> of 10^5. At present we are not convinced that the CBS scheme is the best
>> scheme for these types of flows and are looking for other peoples opinions
>> on alternative methods. Does anyone have any advice in this direction?
>>
>> Best regards,
>> Christian
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Fri, 27 Apr 2012 10:28:41 -0500
>> From: Ataollah Mesgarnejad <amesga1 at tigers.lsu.edu>
>> Subject: Re: [petsc-users] writing native PETSc binaries in python
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Cc: petsc-dev at mcs.anl.gov
>> Message-ID:
>>        <
>> CAC+VmGdP1V3DFrfLWextGVU23NRHtKwUoqcD4Ai__mwTdd1ZGw at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Thanks for the replies:
>>
>> Just a bug report with petsc4py:
>>
>> petsc4py which downloads with --download-petsc4py=1 is the older v1.1.1
>> which doesn't compile with petsc-dev?
>>
>> Thanks,
>> Ata
>>
>> On Fri, Apr 27, 2012 at 8:25 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>> > On Fri, Apr 27, 2012 at 9:19 AM, Ataollah Mesgarnejad <
>> > amesga1 at tigers.lsu.edu> wrote:
>> >
>> >> Dear all,
>> >>
>> >> I was wondering if there is way to write numpy arrays as native PETSc
>> >>  binaries in python so I can open them inside my program with VecView?
>> >>
>> >
>> > I think what you want is to create a Vec using that array in petsc4py.
>> >
>> >    Matt
>> >
>> >
>> >> Thanks,
>> >> Ata
>>
>> >>
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments is infinitely more interesting than any results to which
>> their
>> > experiments lead.
>> > -- Norbert Wiener
>> >
>>
>>
>>
>> --
>> A. Mesgarnejad
>> PhD Student, Research Assistant
>> Mechanical Engineering Department
>> Louisiana State University
>> 2203 Patrick F. Taylor Hall
>> Baton Rouge, La 70803
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/cca3e1b6/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> petsc-users mailing list
>> petsc-users at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>>
>>
>> End of petsc-users Digest, Vol 40, Issue 102
>> ********************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/071853c6/attachment.htm>

From tibo at berkeley.edu  Fri Apr 27 14:50:30 2012
From: tibo at berkeley.edu (tibo at berkeley.edu)
Date: Fri, 27 Apr 2012 12:50:30 -0700
Subject: [petsc-users] Efficient Block Matrix implementation
Message-ID: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>

Hi,

I am using petsc in a Fortran code to solve large systems of linear
equations Ax = b where A is a matrix resulting from a discretization of a
reaction diffusion problem: A is block-tridiagonal, and each of the three
blocks is sparse. It turns out that I know exactly the sparsity structure
of each block (ie the number and position of non-zeros per row).

For now, I am only writing a sequential code. From the petsc manual, it
seems there are two ways to create my matrix A: Either using
MatCreateSeqAIJ, or using MatCreateSeqBAIJ. The second approach seems
recommended since A is indeed a Block matrix.

However, the command MatCreateSeqAIJ allows me to specify the number of
non zeros per row, therefore allowing me to take advantage of knowing the
sparsity structure of my blocks (but losing the advantage of using the
fact that the matrix is a block matrix). It seems that the command
MatCreateSeqBAIJ only allows me to specify the number of non zeros blocks
per block rows, therefore losing the available information on the sparsity
of each block.

My question is then: Is there a way to declare the matrix to take
advantage of both the fact that my matrix is a block matrix (or even
better, that it is a block tridiagonal matrix) and that I know the
sparsity structure of the blocks ?

I am asking this question because If I use the block matrix approach as
described above (just specifying the number of non zeros blocks per block
rows) versus the AIJ approach with specifying the number of non zeros for
each row, the computing time increases by a factor of 5, every other
parameters being unchanged.

Thank you


From knepley at gmail.com  Fri Apr 27 14:58:47 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 15:58:47 -0400
Subject: [petsc-users] Efficient Block Matrix implementation
In-Reply-To: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>
References: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>
Message-ID: <CAMYG4GkdiwigPiZeJiUhK7p9Q+8F-CRtgcNNvBudvNg2zM01Rw@mail.gmail.com>

On Fri, Apr 27, 2012 at 3:50 PM, <tibo at berkeley.edu> wrote:

> Hi,
>
> I am using petsc in a Fortran code to solve large systems of linear
> equations Ax = b where A is a matrix resulting from a discretization of a
> reaction diffusion problem: A is block-tridiagonal, and each of the three
> blocks is sparse. It turns out that I know exactly the sparsity structure
> of each block (ie the number and position of non-zeros per row).
>
> For now, I am only writing a sequential code. From the petsc manual, it
> seems there are two ways to create my matrix A: Either using
> MatCreateSeqAIJ, or using MatCreateSeqBAIJ. The second approach seems
> recommended since A is indeed a Block matrix.
>

No, I think you just want AIJ. BAIJ is for matrices made of small dense
blocks in
a sparse pattern, not big sparse blocks in a dense pattern.

   Matt


> However, the command MatCreateSeqAIJ allows me to specify the number of
> non zeros per row, therefore allowing me to take advantage of knowing the
> sparsity structure of my blocks (but losing the advantage of using the
> fact that the matrix is a block matrix). It seems that the command
> MatCreateSeqBAIJ only allows me to specify the number of non zeros blocks
> per block rows, therefore losing the available information on the sparsity
> of each block.
>
> My question is then: Is there a way to declare the matrix to take
> advantage of both the fact that my matrix is a block matrix (or even
> better, that it is a block tridiagonal matrix) and that I know the
> sparsity structure of the blocks ?
>
> I am asking this question because If I use the block matrix approach as
> described above (just specifying the number of non zeros blocks per block
> rows) versus the AIJ approach with specifying the number of non zeros for
> each row, the computing time increases by a factor of 5, every other
> parameters being unchanged.
>
> Thank you
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/2b1ae9a7/attachment.htm>

From knepley at gmail.com  Fri Apr 27 15:04:21 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 16:04:21 -0400
Subject: [petsc-users] Efficient Block Matrix implementation
In-Reply-To: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>
References: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>
Message-ID: <CAMYG4G=ZCWAfADF9O-mpJc9vm=zOzGBrPfCRG_5AzrRA=OqxNQ@mail.gmail.com>

On Fri, Apr 27, 2012 at 3:50 PM, <tibo at berkeley.edu> wrote:

> Hi,
>
> I am using petsc in a Fortran code to solve large systems of linear
> equations Ax = b where A is a matrix resulting from a discretization of a
> reaction diffusion problem: A is block-tridiagonal, and each of the three
> blocks is sparse. It turns out that I know exactly the sparsity structure
> of each block (ie the number and position of non-zeros per row).
>
> For now, I am only writing a sequential code. From the petsc manual, it
> seems there are two ways to create my matrix A: Either using
> MatCreateSeqAIJ, or using MatCreateSeqBAIJ. The second approach seems
> recommended since A is indeed a Block matrix.
>
> However, the command MatCreateSeqAIJ allows me to specify the number of
> non zeros per row, therefore allowing me to take advantage of knowing the
> sparsity structure of my blocks (but losing the advantage of using the
> fact that the matrix is a block matrix). It seems that the command
> MatCreateSeqBAIJ only allows me to specify the number of non zeros blocks
> per block rows, therefore losing the available information on the sparsity
> of each block.
>
> My question is then: Is there a way to declare the matrix to take
> advantage of both the fact that my matrix is a block matrix (or even
> better, that it is a block tridiagonal matrix) and that I know the
> sparsity structure of the blocks ?
>

What exactly do you want to do with the blocks? Its easy to extract them
with
MatGetSubmatrix() and they can be used in the PCFIELDSPLIT preconditioner.

   Matt


> I am asking this question because If I use the block matrix approach as
> described above (just specifying the number of non zeros blocks per block
> rows) versus the AIJ approach with specifying the number of non zeros for
> each row, the computing time increases by a factor of 5, every other
> parameters being unchanged.
>
> Thank you
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/ec6ba85d/attachment-0001.htm>

From tibo at berkeley.edu  Fri Apr 27 15:21:53 2012
From: tibo at berkeley.edu (tibo at berkeley.edu)
Date: Fri, 27 Apr 2012 13:21:53 -0700
Subject: [petsc-users] Efficient Block Matrix implementation
In-Reply-To: <CAMYG4G=ZCWAfADF9O-mpJc9vm=zOzGBrPfCRG_5AzrRA=OqxNQ@mail.gmail.com>
References: <7d39cf0b1afde6b14723c7cad189392f.squirrel@calmail.berkeley.edu>
	<CAMYG4G=ZCWAfADF9O-mpJc9vm=zOzGBrPfCRG_5AzrRA=OqxNQ@mail.gmail.com>
Message-ID: <32e4254c5b6ca3c74216d95feab2e999.squirrel@calmail.berkeley.edu>

> On Fri, Apr 27, 2012 at 3:50 PM, <tibo at berkeley.edu> wrote:
>
>> Hi,
>>
>> I am using petsc in a Fortran code to solve large systems of linear
>> equations Ax = b where A is a matrix resulting from a discretization of
>> a
>> reaction diffusion problem: A is block-tridiagonal, and each of the
>> three
>> blocks is sparse. It turns out that I know exactly the sparsity
>> structure
>> of each block (ie the number and position of non-zeros per row).
>>
>> For now, I am only writing a sequential code. From the petsc manual, it
>> seems there are two ways to create my matrix A: Either using
>> MatCreateSeqAIJ, or using MatCreateSeqBAIJ. The second approach seems
>> recommended since A is indeed a Block matrix.
>>
>> However, the command MatCreateSeqAIJ allows me to specify the number of
>> non zeros per row, therefore allowing me to take advantage of knowing
>> the
>> sparsity structure of my blocks (but losing the advantage of using the
>> fact that the matrix is a block matrix). It seems that the command
>> MatCreateSeqBAIJ only allows me to specify the number of non zeros
>> blocks
>> per block rows, therefore losing the available information on the
>> sparsity
>> of each block.
>>
>> My question is then: Is there a way to declare the matrix to take
>> advantage of both the fact that my matrix is a block matrix (or even
>> better, that it is a block tridiagonal matrix) and that I know the
>> sparsity structure of the blocks ?
>>
>
> What exactly do you want to do with the blocks? Its easy to extract them
> with
> MatGetSubmatrix() and they can be used in the PCFIELDSPLIT preconditioner.
>
>    Matt
>
>

Thank you for your answer (and previous one).
I don't need to use block decomposition, my goal is just to solve as
efficiently as possible systems of the form Ax=b at each time step. So
what I do now as a first approach is that I declare A as SEQAIJ, then
fills it at each time step (since A changes at each time step), and then
solves many times a system of the form Ax = b (Runge Kutta steps with
inner Newton iterations) using KSPsolve and go to the next time step.
Since my matrix is block-tridiagonal, I was just wondering if I could take
advantage of this using a SEQBAIJ declaration, which seems not to be the
case since the blocks are sparse.
The next step will be to implement this sequence in parallel, but I wanted
to make sure that I was using the appropriate sequential implementation
first to base myself upon

>> I am asking this question because If I use the block matrix approach as
>> described above (just specifying the number of non zeros blocks per
>> block
>> rows) versus the AIJ approach with specifying the number of non zeros
>> for
>> each row, the computing time increases by a factor of 5, every other
>> parameters being unchanged.
>>
>> Thank you
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



From tapitts at sandia.gov  Fri Apr 27 15:22:25 2012
From: tapitts at sandia.gov (Pitts, Todd Alan)
Date: Fri, 27 Apr 2012 20:22:25 +0000
Subject: [petsc-users] SAMRAI and Petsc
Message-ID: <8A0B84FEB45AFE4A8507E08590F747952B0B486B@EXMB02.srn.sandia.gov>

All,

I have been working with SAMRAI and Petsc to develop an implicit hyperbolic solver.  Both packages seem very well done -- I am new to both, however.  The basic physics problem is the solution to the linear wave equation  (or rather something very close to it).  I am using SAMRAI to track the mesh and Petsc to solve the resulting implicit time stepping problem.  I am giving the MPIcomm object from SAMRAI to Petsc but have noticed that SAMRAI distributes the processors across the domain seemingly randomly.  Petsc on the other hand distributes them sequentially for a matrix.  In other words, SAMRAI might give processors with adjacent rank numbers nonadjacent patches while Petsc always assigns the FIRST n rows to the rank 0 process (where n is the number of rows requested by the rank 0 process), the NEXT m rows to the rank 1 process, etc.  This means that a nice, diagonally banded matrix is no longer banded  (because we have reordered the rows according to the SAMRAI domain decomposition).  Naturally, the solution to the linear problem is the same.  However, I am concerned that I have lost the apparent diagonal structure of the matrix.   Is this a problem for the linear solvers?  I know that Petsc is used for implicit problems together with various meshing codes -- this is a common thing.  I would like to use the library in its intended fashion -- is the proper thing to do to simply fill the rows you own and forget the fact that the structure is no longer banded?  Am I missing something?

Thanks in advance for any help you might offer,

Todd

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/e6abea45/attachment.htm>

From bsmith at mcs.anl.gov  Fri Apr 27 15:32:46 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 27 Apr 2012 15:32:46 -0500
Subject: [petsc-users] SAMRAI and Petsc
In-Reply-To: <8A0B84FEB45AFE4A8507E08590F747952B0B486B@EXMB02.srn.sandia.gov>
References: <8A0B84FEB45AFE4A8507E08590F747952B0B486B@EXMB02.srn.sandia.gov>
Message-ID: <37DB072A-A359-4E57-B746-ED70A7AA6542@mcs.anl.gov>


  Todd,

    The specific diagonally banded structure is not something that can usually be taken advantage of in parallel, so don't worry about it not have this exact structure.

    However, the fact that SAMRAI is "randomly" assigning data chunks to processes seems like it will be problematic for large core counts (1000s of cores) since the ghost point updates may be slower; but this is really a SAMRAI only issue and not related to PETSc. So the question is how does SAMRAI decide where to assign chunks and is there any way to improve that. Looking its documentation and asking the SAMRAI developers about this issue may be useful.

   Barry

On Apr 27, 2012, at 3:22 PM, Pitts, Todd Alan wrote:

> All,
> 
> I have been working with SAMRAI and Petsc to develop an implicit hyperbolic solver.  Both packages seem very well done -- I am new to both, however.  The basic physics problem is the solution to the linear wave equation  (or rather something very close to it).  I am using SAMRAI to track the mesh and Petsc to solve the resulting implicit time stepping problem.  I am giving the MPIcomm object from SAMRAI to Petsc but have noticed that SAMRAI distributes the processors across the domain seemingly randomly.  Petsc on the other hand distributes them sequentially for a matrix.  In other words, SAMRAI might give processors with adjacent rank numbers nonadjacent patches while Petsc always assigns the FIRST n rows to the rank 0 process (where n is the number of rows requested by the rank 0 process), the NEXT m rows to the rank 1 process, etc.  This means that a nice, diagonally banded matrix is no longer banded  (because we have reordered the rows according to the SAMRAI domain decomposition).  Naturally, the solution to the linear problem is the same.  However, I am concerned that I have lost the apparent diagonal structure of the matrix.   Is this a problem for the linear solvers?  I know that Petsc is used for implicit problems together with various meshing codes -- this is a common thing.  I would like to use the library in its intended fashion -- is the proper thing to do to simply fill the rows you own and forget the fact that the structure is no longer banded?  Am I missing something?
> 
> Thanks in advance for any help you might offer,
> 
> Todd
> 


From andrew.spott at gmail.com  Fri Apr 27 15:57:16 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Fri, 27 Apr 2012 14:57:16 -0600
Subject: [petsc-users] Turning off PETSC errors
Message-ID: <7B5C2405-F32F-488B-86E6-06617F9C1A70@gmail.com>

I keep getting a floating point error, unfortunately I can't seem to figure out where it is coming from, because the PETSC error codes keep being kicked out before my own code can tell me anything. (And the error codes aren't telling me where my errors are).  Is there anyway to turn off the errors by petsc, without recompiling petsc?

Thanks

Andrew

From knepley at gmail.com  Fri Apr 27 16:43:39 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 17:43:39 -0400
Subject: [petsc-users] Turning off PETSC errors
In-Reply-To: <7B5C2405-F32F-488B-86E6-06617F9C1A70@gmail.com>
References: <7B5C2405-F32F-488B-86E6-06617F9C1A70@gmail.com>
Message-ID: <CAMYG4Gk9ykWSbMpeqLh96syTOBgd4Y=sj7z0-amyqcv-zRFpCg@mail.gmail.com>

On Fri, Apr 27, 2012 at 4:57 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> I keep getting a floating point error, unfortunately I can't seem to
> figure out where it is coming from, because the PETSC error codes keep
> being kicked out before my own code can tell me anything. (And the error
> codes aren't telling me where my errors are).  Is there anyway to turn off
> the errors by petsc, without recompiling petsc?
>

To be specific, are you getting a SIGFPE? If so, you should be able to pull
up the debugger and see
exactly what line it occurs on. You can turn off the trap with


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscFPTrapPush.html#PetscFPTrapPush

    Matt


> Thanks
>
> Andrew




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/16484ca8/attachment.htm>

From iamkyungjoo at gmail.com  Fri Apr 27 16:46:44 2012
From: iamkyungjoo at gmail.com (Kyungjoo Kim)
Date: Fri, 27 Apr 2012 16:46:44 -0500
Subject: [petsc-users] Question about reordering in ilu...
Message-ID: <53985B06-0C51-493D-ABE2-9AE54CDE9639@gmail.com>

Hello,


I am wondering if you do something like reordering (nested dissection or minimum degree) when you perform ilu.

I do experiments with one reordered by nested dissection and the other one is reordered opposite way. 
(BTW, ilu does not do pivoting but isn't it possible to do pivot if petsc find out block matrices such like supernodes ? )

Somehow both behave in the same way. Do you internally reorder a matrix ? ( I could not find out any clue about this in reference manual)

Thank you.






From knepley at gmail.com  Fri Apr 27 16:52:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 27 Apr 2012 17:52:34 -0400
Subject: [petsc-users] Question about reordering in ilu...
In-Reply-To: <53985B06-0C51-493D-ABE2-9AE54CDE9639@gmail.com>
References: <53985B06-0C51-493D-ABE2-9AE54CDE9639@gmail.com>
Message-ID: <CAMYG4Gk2fWN0X98FOnS286YKZ9f80S5aw9=8BnFdSjYUMH3_fg@mail.gmail.com>

On Fri, Apr 27, 2012 at 5:46 PM, Kyungjoo Kim <iamkyungjoo at gmail.com> wrote:

> Hello,
>
>
> I am wondering if you do something like reordering (nested dissection or
> minimum degree) when you perform ilu.
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFactorSetMatOrderingType.html#PCFactorSetMatOrderingType

   Matt


> I do experiments with one reordered by nested dissection and the other one
> is reordered opposite way.
> (BTW, ilu does not do pivoting but isn't it possible to do pivot if petsc
> find out block matrices such like supernodes ? )
>
> Somehow both behave in the same way. Do you internally reorder a matrix ?
> ( I could not find out any clue about this in reference manual)
>
> Thank you.
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120427/d99c77d8/attachment.htm>

From griffith at cims.nyu.edu  Fri Apr 27 16:54:58 2012
From: griffith at cims.nyu.edu (Boyce Griffith)
Date: Fri, 27 Apr 2012 17:54:58 -0400
Subject: [petsc-users] SAMRAI and Petsc
In-Reply-To: <8A0B84FEB45AFE4A8507E08590F747952B0B486B@EXMB02.srn.sandia.gov>
References: <8A0B84FEB45AFE4A8507E08590F747952B0B486B@EXMB02.srn.sandia.gov>
Message-ID: <4F9B15B2.5010907@cims.nyu.edu>



On 4/27/12 4:22 PM, Pitts, Todd Alan wrote:
> I have been working with SAMRAI and Petsc to develop an implicit
> hyperbolic solver. Both packages seem very well done -- I am new to
> both, however. The basic physics problem is the solution to the linear
> wave equation (or rather something very close to it). I am using SAMRAI
> to track the mesh and Petsc to solve the resulting implicit time
> stepping problem. I am giving the MPIcomm object from SAMRAI to Petsc
> but have noticed that SAMRAI distributes the processors across the
> domain seemingly randomly. Petsc on the other hand distributes them

SAMRAI shouldn't be *randomly* assigning patches to processors.  If you 
are using SAMRAI 3.X, the default behavior of the 
ChopAndPackLoadBalancer is to use a space-filling curve algorithm to 
assign patches to processors.  (I am not sure exactly how the other load 
balancer class implemented in SAMRAI, TreeLoadBalancer, works.)  If you 
are using SAMRAI 2.X, the default behavior is again to use a 
space-filling curve algorithm.  You can also setup your own parallel 
distribution algorithm if you are so inclined.

> sequentially for a matrix. In other words, SAMRAI might give processors
> with adjacent rank numbers nonadjacent patches while Petsc always
> assigns the FIRST n rows to the rank 0 process (where n is the number of
> rows requested by the rank 0 process), the NEXT m rows to the rank 1
> process, etc. This means that a nice, diagonally banded matrix is no
> longer banded (because we have reordered the rows according to the
> SAMRAI domain decomposition). Naturally, the solution to the linear
> problem is the same. However, I am concerned that I have lost the
> apparent diagonal structure of the matrix. Is this a problem for the
> linear solvers? I know that Petsc is used for implicit problems together
> with various meshing codes -- this is a common thing. I would like to
> use the library in its intended fashion -- is the proper thing to do to
> simply fill the rows you own and forget the fact that the structure is
> no longer banded? Am I missing something?

Another option is to use the SAMRAIVectorReal classes to provide a 
vector interface to native SAMRAI data, and to pass those to PETSc using 
a wrapper class.  This allows you to avoid having to copy data back and 
forth between SAMRAI and PETSc, although for linear solvers you will 
need to use shell matrices and preconditioners.  Of course, the big 
drawback of this approach is that you have to roll your own 
preconditioner implementations, which can be a lot of work.

There is some code to facilitate this in the SAMRAI distribution, but 
I've always found it better to maintain my own versions of these 
wrappers (there is some relevant code available from 
https://ibamr.googlecode.com).  I think Bobby Philip maintains similar 
wrappers, although I don't know whether they are readily available.

-- Boyce

From andrew.spott at gmail.com  Fri Apr 27 19:08:08 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Fri, 27 Apr 2012 18:08:08 -0600
Subject: [petsc-users] =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
Message-ID: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>

I'm honestly stumped.

I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.

I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.

any ideas?  (long error message below):

-Andrew

[14]PETSC ERROR: ------------------------------------------------------------------------
[14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[14]PETSC ERROR: likely location of problem given in stack below
[14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[14]PETSC ERROR:       INSTEAD the line number of the start of the function
[14]PETSC ERROR:       is given.
[14]PETSC ERROR: --------------------- Error Message ------------------------------------
[14]PETSC ERROR: Signal received!
[14]PETSC ERROR: ------------------------------------------------------------------------
[14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: likely location of problem given in stack below
[15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[15]PETSC ERROR:       INSTEAD the line number of the start of the function
[15]PETSC ERROR:       is given.
[15]PETSC ERROR: --------------------- Error Message ------------------------------------
[15]PETSC ERROR: Signal received!
[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
[14]PETSC ERROR: See docs/changes/index.html for recent updates.
[14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[14]PETSC ERROR: See docs/index.html for manual pages.
[14]PETSC ERROR: ------------------------------------------------------------------------
[14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
[14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
[14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
[14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
[14]PETSC ERROR: ------------------------------------------------------------------------
[14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
[15]PETSC ERROR: See docs/changes/index.html for recent updates.
[15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[15]PETSC ERROR: See docs/index.html for manual pages.
[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
[15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
[15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
[15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt

From balay at mcs.anl.gov  Fri Apr 27 20:26:41 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 27 Apr 2012 20:26:41 -0500 (CDT)
Subject: [petsc-users]
	=?utf-8?q?floating_point_exception=E2=80=A6_but_onl?=
	=?utf-8?q?y_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
Message-ID: <alpine.LFD.2.02.1204272020300.2752@asterix>

On Fri, 27 Apr 2012, Andrew Spott wrote:

> I'm honestly stumped.
> 
> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
> 
> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.

assuming you have X11 setup properly from compute nodes you can run
with the extra option '-start_in_debugger'

If X11 is not properly setup - and you'd like to run gdb on one of the
nodes [say node 14 where you see SEGV] - you can do:

-start_in-debugger noxterm -debugger_nodes 14

Or try valgrind

mpiexec -n 16 valgrind --tool=memcheck -q ./executable


For debugging - its best to install with --download-mpich [so that its
valgrind clean] - and run all mpi stuff on a single machine - [usually
X11 works well from a single machine.]

Satish

> 
> any ideas?  (long error message below):
> 
> -Andrew
> 
> [14]PETSC ERROR: ------------------------------------------------------------------------
> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [14]PETSC ERROR: likely location of problem given in stack below
> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
> [14]PETSC ERROR:       is given.
> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
> [14]PETSC ERROR: Signal received!
> [14]PETSC ERROR: ------------------------------------------------------------------------
> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [15]PETSC ERROR: likely location of problem given in stack below
> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
> [15]PETSC ERROR:       is given.
> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
> [15]PETSC ERROR: Signal received!
> [15]PETSC ERROR: ------------------------------------------------------------------------
> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [14]PETSC ERROR: See docs/index.html for manual pages.
> [14]PETSC ERROR: ------------------------------------------------------------------------
> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
> [14]PETSC ERROR: ------------------------------------------------------------------------
> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [15]PETSC ERROR: See docs/index.html for manual pages.
> [15]PETSC ERROR: ------------------------------------------------------------------------
> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
> [15]PETSC ERROR: ------------------------------------------------------------------------
> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt


From pham_ha at yahoo.com  Sat Apr 28 03:32:48 2012
From: pham_ha at yahoo.com (Pham Van)
Date: Sat, 28 Apr 2012 01:32:48 -0700 (PDT)
Subject: [petsc-users] Preallocation for regtangular matrix
In-Reply-To: <mailman.1091.1335563707.4646.petsc-users@mcs.anl.gov>
References: <mailman.1091.1335563707.4646.petsc-users@mcs.anl.gov>
Message-ID: <1335601968.53199.YahooMailNeo@web163105.mail.bf1.yahoo.com>

Dear list,

I am trying to create matrices for a rectangular matrix with number of rows is much bigger than number of columns.

I create 2 matrices together one square and one rectangular with the same number of rows. To my surprise the smaller matrix (rectangular) creation take much more time than the bigger one (rectangular). I did preallocate for both matrices with predefine number of diagonal and off-diagonal entries. But then again I did not know how to define diagonal part of a rectangular matrix. Only a very small top part of the matrix is "diagonal".

I have try both method: the first one to set diagonal and off-diagonal part as it was a square matrix; and second to set only a small top part of the matrix diagonal. Both method does not work.

Does anyone know how to preallocate a rectangular matrix.

By the way, the same code run pretty fast when I run with single process and painfully slow when 2 processes employed.

Thank you in advance.

Kind regards,
Pham Van Ha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/9d782e4d/attachment.htm>

From u.tabak at tudelft.nl  Sat Apr 28 04:25:34 2012
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Sat, 28 Apr 2012 11:25:34 +0200
Subject: [petsc-users] a cost question for sparse factorization
Message-ID: <4F9BB78E.5010806@tudelft.nl>

Dear all,

My question is not directly related to PETSc however it fits the 
audience on this list properly. Where can I find the computational cost 
analysis on

+ sparse cholesky factorization,
+ sparse LDL^T factorization,

in relation to the bandwidth of my sparse matrices. All the references I 
have present these costs for dense matrices. Where shall I look to get 
an overview on these costs for sparse/band matrices?

Best regards,
Umut

From jedbrown at mcs.anl.gov  Sat Apr 28 04:38:59 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 28 Apr 2012 04:38:59 -0500
Subject: [petsc-users] a cost question for sparse factorization
In-Reply-To: <4F9BB78E.5010806@tudelft.nl>
References: <4F9BB78E.5010806@tudelft.nl>
Message-ID: <CAM9tzSkU8vruocaONNEksXA8oSPP0R3SBZ2N3Mt9hSVAhSATkw@mail.gmail.com>

On Sat, Apr 28, 2012 at 04:25, Umut Tabak <u.tabak at tudelft.nl> wrote:

> Dear all,
>
> My question is not directly related to PETSc however it fits the audience
> on this list properly. Where can I find the computational cost analysis on
>
> + sparse cholesky factorization,
> + sparse LDL^T factorization,
>
> in relation to the bandwidth of my sparse matrices. All the references I
> have present these costs for dense matrices. Where shall I look to get an
> overview on these costs for sparse/band matrices?
>

All you will get are asymptotics, not sharp operation counts. For 2D, the
bounds are O(n^{1.5} flops and O(n log n) space; for 3D they are O(n^2)
flops and O(n^{4/3}) space. This is covered in Alan George's book,
http://www.cse.illinois.edu/courses/cs598mh/george_liu.pdf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/a1c026e8/attachment.htm>

From u.tabak at tudelft.nl  Sat Apr 28 04:56:43 2012
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Sat, 28 Apr 2012 11:56:43 +0200
Subject: [petsc-users] a cost question for sparse factorization
In-Reply-To: <CAM9tzSkU8vruocaONNEksXA8oSPP0R3SBZ2N3Mt9hSVAhSATkw@mail.gmail.com>
References: <4F9BB78E.5010806@tudelft.nl>
	<CAM9tzSkU8vruocaONNEksXA8oSPP0R3SBZ2N3Mt9hSVAhSATkw@mail.gmail.com>
Message-ID: <4F9BBEDB.1080305@tudelft.nl>


> All you will get are asymptotics, not sharp operation counts. For 2D, 
> the bounds are O(n^{1.5} flops and O(n log n) space; for 3D they are 
> O(n^2) flops and O(n^{4/3}) space. This is covered in Alan George's 
> book, http://www.cse.illinois.edu/courses/cs598mh/george_liu.pdf
>
Dear Jed,
I am not good at these cost computations, is 'n' the number of nonzeros 
in the matrices?


From jedbrown at mcs.anl.gov  Sat Apr 28 05:00:08 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 28 Apr 2012 05:00:08 -0500
Subject: [petsc-users] a cost question for sparse factorization
In-Reply-To: <4F9BBEDB.1080305@tudelft.nl>
References: <4F9BB78E.5010806@tudelft.nl>
	<CAM9tzSkU8vruocaONNEksXA8oSPP0R3SBZ2N3Mt9hSVAhSATkw@mail.gmail.com>
	<4F9BBEDB.1080305@tudelft.nl>
Message-ID: <CAM9tzSkyXDG26YTybbj0rE2-LA9eofJqfcTxiXNCOKdnxHkV7w@mail.gmail.com>

On Sat, Apr 28, 2012 at 04:56, Umut Tabak <u.tabak at tudelft.nl> wrote:

> All you will get are asymptotics, not sharp operation counts. For 2D, the
>> bounds are O(n^{1.5} flops and O(n log n) space; for 3D they are O(n^2)
>> flops and O(n^{4/3}) space. This is covered in Alan George's book,
>> http://www.cse.illinois.edu/**courses/cs598mh/george_liu.pdf<http://www.cse.illinois.edu/courses/cs598mh/george_liu.pdf>
>>
>>  Dear Jed,
> I am not good at these cost computations, is 'n' the number of nonzeros in
> the matrices?


No, n is the number of degrees of freedom in the model.

There is no estimate just in terms of the number of nonzeros because
ultimately, the cost is limited by the size of minimal vertex separators.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/9cd68706/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Sat Apr 28 05:05:22 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 28 Apr 2012 05:05:22 -0500
Subject: [petsc-users] Preallocation for regtangular matrix
In-Reply-To: <1335601968.53199.YahooMailNeo@web163105.mail.bf1.yahoo.com>
References: <mailman.1091.1335563707.4646.petsc-users@mcs.anl.gov>
	<1335601968.53199.YahooMailNeo@web163105.mail.bf1.yahoo.com>
Message-ID: <CAM9tzSnXqs5Uzx1z9Lu2BMGFrRe3PsjQpbJaMcdeDPZmQDb7vQ@mail.gmail.com>

On Sat, Apr 28, 2012 at 03:32, Pham Van <pham_ha at yahoo.com> wrote:

> I am trying to create matrices for a rectangular matrix with number of
> rows is much bigger than number of columns.
>
> I create 2 matrices together one square and one rectangular with the same
> number of rows. To my surprise the smaller matrix (rectangular) creation
> take much more time than the bigger one (rectangular). I did preallocate
> for both matrices with predefine number of diagonal and off-diagonal
> entries. But then again I did not know how to define diagonal part of a
> rectangular matrix. Only a very small top part of the matrix is "diagonal".
>
> I have try both method: the first one to set diagonal and off-diagonal
> part as it was a square matrix; and second to set only a small top part of
> the matrix diagonal. Both method does not work.
>
> Does anyone know how to preallocate a rectangular matrix.
>
> By the way, the same code run pretty fast when I run with single process
> and painfully slow when 2 processes employed.
>

The matrix has row and column ownership ranges. The "diagonal" part is any
entry that lies in both the row and column ownership range. The
"off-diagonal" part is in the row ownership range, but not in the column
ownership range.

Run with -mat_new_nonzero_allocation_err or

ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR,flg);CHKERRQ(ierr);

to find which entries start going outside your preallocation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/208b1c4c/attachment.htm>

From agrayver at gfz-potsdam.de  Sat Apr 28 09:03:58 2012
From: agrayver at gfz-potsdam.de (Alexander Grayver)
Date: Sat, 28 Apr 2012 16:03:58 +0200
Subject: [petsc-users] mumps solve with same nonzero pattern
In-Reply-To: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
References: <CAMJxm+DdrW1spX=BKb6pLTFiG7Kb7DcRCnX8RLjL_YTbr4vS=A@mail.gmail.com>
Message-ID: <4F9BF8CE.7030005@gfz-potsdam.de>

This is not quite normal.

Right now I'm dealing with 0.65 million problem and it takes 15 secs for 
symbolic factorization to be done using 64 cores (four 16x cores nodes).

As Hong suggested you should try different ordering techniques. Usually 
metis is among the best (-mat_mumps_icntl_7 5). And also any timings is 
better to do in release version (--with-debugging=0 flag for configure).

You should be able to get some output and statistics from MUMPS with this:
-mat_mumps_icntl_3 2 -mat_mumps_icntl_4 2

If you don't read MUMPS manual for that. Without any output from MUMPS 
it is not really clear what happens.

On 27.04.2012 19:49, Wen Jiang wrote:
> Thanks for your reply. I also tested the exactly same problem with 
> SUPERLU. The mumps and superlu dist both are using sequential symbolic 
> factorization. However, superlu dist takes only 20 seconds but mumps 
> takes almost 700 seconds. I am wondering whether such a big difference 
> is possible. Do those two direct solver use quite different algorithm?
>
> And also since I might have the same nonzero structure system to be 
> solved many times at different places. I am wondering whether I could 
> save the symbolic factorization output somewhere and then read them as 
> the input for future solving. Thanks.
>
> Regards,
> Wen
>


-- 
Regards,
Alexander


From jiangwen84 at gmail.com  Sat Apr 28 12:00:06 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Sat, 28 Apr 2012 13:00:06 -0400
Subject: [petsc-users] superlu dist colperm by default
Message-ID: <CAMJxm+DQwYW-6prQHAXPCuC3buvkq7md5yVaHqTMtprBjEjDsA@mail.gmail.com>

Hi,

I am using superlu dist in PETSc 3.1. The version of superlu dist installed
by petsc is SuperLU_DIST_2.4-hg-v2. Does anyone know which colperm method
superlu dist use for the default setting by petsc? I cannot got such
information by using -mat_superlu_dist_statprint, the output of which I
attached below. Thanks.

Regards,
Wen

***************************************************************************************************************
PC Object:
  type: lu
    LU: out-of-place factorization
    tolerance for zero pivot 1e-12
    matrix ordering: natural
    factor fill ratio given 0, needed 0
      Factored matrix follows:
        Matrix Object:
          type=mpiaij, rows=215883, cols=215883
          package used to perform factorization: superlu_dist
          total: nonzeros=0, allocated nonzeros=431766
            SuperLU_DIST run parameters:
              Process grid nprow 8 x npcol 8
              Equilibrate matrix TRUE
              Matrix input mode 1
              Replace tiny pivots TRUE
              Use iterative refinement FALSE
              Processors in row 8 col partition 8
              Row permutation LargeDiag
              Parallel symbolic factorization FALSE
              Repeated factorization SamePattern
******************************************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/599f3cb2/attachment.htm>

From pham_ha at yahoo.com  Sat Apr 28 12:17:48 2012
From: pham_ha at yahoo.com (Pham Van)
Date: Sat, 28 Apr 2012 10:17:48 -0700 (PDT)
Subject: [petsc-users] Preallocation for regtangular matrix
In-Reply-To: <CAM9tzSnXqs5Uzx1z9Lu2BMGFrRe3PsjQpbJaMcdeDPZmQDb7vQ@mail.gmail.com>
References: <mailman.1091.1335563707.4646.petsc-users@mcs.anl.gov>
	<1335601968.53199.YahooMailNeo@web163105.mail.bf1.yahoo.com>
	<CAM9tzSnXqs5Uzx1z9Lu2BMGFrRe3PsjQpbJaMcdeDPZmQDb7vQ@mail.gmail.com>
Message-ID: <1335633468.45720.YahooMailNeo@web163103.mail.bf1.yahoo.com>

Dear Jed Brown,

I had found a better way: treat the matrix as a square matrix. Since it is spare anyway, so there should be no memory burden by doing so. I also extended the vector size to avoid conformable problem. I guess the size of a vector does not matter that much.

This little trick does a great job.


Nevertheless it would be great if you guy (developers) could consider putting some comments on creation of rectangular matrix.

Thank you for your very quick response.


Kind regards,
Pham Van Ha



________________________________
 From: Jed Brown <jedbrown at mcs.anl.gov>
To: Pham Van <pham_ha at yahoo.com>; PETSc users list <petsc-users at mcs.anl.gov> 
Sent: Saturday, April 28, 2012 5:05 PM
Subject: Re: [petsc-users] Preallocation for regtangular matrix
 

On Sat, Apr 28, 2012 at 03:32, Pham Van <pham_ha at yahoo.com> wrote:

I am trying to create matrices for a rectangular matrix with number of rows is much bigger than number of columns.
>
>I create 2 matrices together one square and one rectangular with the same number of rows. To my surprise the smaller matrix (rectangular) creation take much more time than the bigger one (rectangular). I did preallocate for both matrices with predefine number of diagonal and off-diagonal entries. But then again I did not know how to define diagonal part of a rectangular matrix. Only a very small top part of the matrix is "diagonal".
>
>I have try both method: the first one to set diagonal and off-diagonal part as it was a square matrix; and second to set only a small top part of the matrix diagonal. Both method does not work.
>
>Does anyone know how to preallocate a rectangular
 matrix.
>
>By the way, the same code run pretty fast when I run with single process and painfully slow when 2 processes employed.

The matrix has row and column ownership ranges. The "diagonal" part is any entry that lies in both the row and column ownership range. The "off-diagonal" part is in the row ownership range, but not in the column ownership range.

Run with -mat_new_nonzero_allocation_err or

ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR,flg);CHKERRQ(ierr);

to find which entries start going outside your preallocation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/30e9f9ee/attachment.htm>

From pham_ha at yahoo.com  Sat Apr 28 12:22:28 2012
From: pham_ha at yahoo.com (Pham Van)
Date: Sat, 28 Apr 2012 10:22:28 -0700 (PDT)
Subject: [petsc-users] petsc-users Digest, Vol 40, Issue 107
In-Reply-To: <mailman.53.1335632425.18646.petsc-users@mcs.anl.gov>
References: <mailman.53.1335632425.18646.petsc-users@mcs.anl.gov>
Message-ID: <1335633748.92571.YahooMailNeo@web163105.mail.bf1.yahoo.com>

I think the sentence:

The matrix has row and column ownership ranges.

is very important. I did not get it from the manual (maybe my fault).

Kind regards,
Pham Van Ha



________________________________
 From: "petsc-users-request at mcs.anl.gov" <petsc-users-request at mcs.anl.gov>
To: petsc-users at mcs.anl.gov 
Sent: Sunday, April 29, 2012 12:00 AM
Subject: petsc-users Digest, Vol 40, Issue 107
 
Send petsc-users mailing list submissions to
??? petsc-users at mcs.anl.gov

To subscribe or unsubscribe via the World Wide Web, visit
??? https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
or, via email, send a message with subject or body 'help' to
??? petsc-users-request at mcs.anl.gov

You can reach the person managing the list at
??? petsc-users-owner at mcs.anl.gov

When replying, please edit your Subject line so it is more specific
than "Re: Contents of petsc-users digest..."


Today's Topics:

?  1. Re:? Preallocation for regtangular matrix (Jed Brown)
?  2. Re:? mumps solve with same nonzero pattern (Alexander Grayver)
?  3.? superlu dist colperm by default (Wen Jiang)


----------------------------------------------------------------------

Message: 1
Date: Sat, 28 Apr 2012 05:05:22 -0500
From: Jed Brown <jedbrown at mcs.anl.gov>
Subject: Re: [petsc-users] Preallocation for regtangular matrix
To: Pham Van <pham_ha at yahoo.com>, PETSc users list
??? <petsc-users at mcs.anl.gov>
Message-ID:
??? <CAM9tzSnXqs5Uzx1z9Lu2BMGFrRe3PsjQpbJaMcdeDPZmQDb7vQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On Sat, Apr 28, 2012 at 03:32, Pham Van <pham_ha at yahoo.com> wrote:

> I am trying to create matrices for a rectangular matrix with number of
> rows is much bigger than number of columns.
>
> I create 2 matrices together one square and one rectangular with the same
> number of rows. To my surprise the smaller matrix (rectangular) creation
> take much more time than the bigger one (rectangular). I did preallocate
> for both matrices with predefine number of diagonal and off-diagonal
> entries. But then again I did not know how to define diagonal part of a
> rectangular matrix. Only a very small top part of the matrix is "diagonal".
>
> I have try both method: the first one to set diagonal and off-diagonal
> part as it was a square matrix; and second to set only a small top part of
> the matrix diagonal. Both method does not work.
>
> Does anyone know how to preallocate a rectangular matrix.
>
> By the way, the same code run pretty fast when I run with single process
> and painfully slow when 2 processes employed.
>

The matrix has row and column ownership ranges. The "diagonal" part is any
entry that lies in both the row and column ownership range. The
"off-diagonal" part is in the row ownership range, but not in the column
ownership range.

Run with -mat_new_nonzero_allocation_err or

ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR,flg);CHKERRQ(ierr);

to find which entries start going outside your preallocation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/208b1c4c/attachment-0001.htm>

------------------------------

Message: 2
Date: Sat, 28 Apr 2012 16:03:58 +0200
From: Alexander Grayver <agrayver at gfz-potsdam.de>
Subject: Re: [petsc-users] mumps solve with same nonzero pattern
To: PETSc users list <petsc-users at mcs.anl.gov>
Message-ID: <4F9BF8CE.7030005 at gfz-potsdam.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

This is not quite normal.

Right now I'm dealing with 0.65 million problem and it takes 15 secs for 
symbolic factorization to be done using 64 cores (four 16x cores nodes).

As Hong suggested you should try different ordering techniques. Usually 
metis is among the best (-mat_mumps_icntl_7 5). And also any timings is 
better to do in release version (--with-debugging=0 flag for configure).

You should be able to get some output and statistics from MUMPS with this:
-mat_mumps_icntl_3 2 -mat_mumps_icntl_4 2

If you don't read MUMPS manual for that. Without any output from MUMPS 
it is not really clear what happens.

On 27.04.2012 19:49, Wen Jiang wrote:
> Thanks for your reply. I also tested the exactly same problem with 
> SUPERLU. The mumps and superlu dist both are using sequential symbolic 
> factorization. However, superlu dist takes only 20 seconds but mumps 
> takes almost 700 seconds. I am wondering whether such a big difference 
> is possible. Do those two direct solver use quite different algorithm?
>
> And also since I might have the same nonzero structure system to be 
> solved many times at different places. I am wondering whether I could 
> save the symbolic factorization output somewhere and then read them as 
> the input for future solving. Thanks.
>
> Regards,
> Wen
>


-- 
Regards,
Alexander



------------------------------

Message: 3
Date: Sat, 28 Apr 2012 13:00:06 -0400
From: Wen Jiang <jiangwen84 at gmail.com>
Subject: [petsc-users] superlu dist colperm by default
To: petsc-users at mcs.anl.gov
Message-ID:
??? <CAMJxm+DQwYW-6prQHAXPCuC3buvkq7md5yVaHqTMtprBjEjDsA at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I am using superlu dist in PETSc 3.1. The version of superlu dist installed
by petsc is SuperLU_DIST_2.4-hg-v2. Does anyone know which colperm method
superlu dist use for the default setting by petsc? I cannot got such
information by using -mat_superlu_dist_statprint, the output of which I
attached below. Thanks.

Regards,
Wen

***************************************************************************************************************
PC Object:
? type: lu
? ? LU: out-of-place factorization
? ? tolerance for zero pivot 1e-12
? ? matrix ordering: natural
? ? factor fill ratio given 0, needed 0
? ? ? Factored matrix follows:
? ? ? ? Matrix Object:
? ? ? ? ? type=mpiaij, rows=215883, cols=215883
? ? ? ? ? package used to perform factorization: superlu_dist
? ? ? ? ? total: nonzeros=0, allocated nonzeros=431766
? ? ? ? ? ? SuperLU_DIST run parameters:
? ? ? ? ? ? ? Process grid nprow 8 x npcol 8
? ? ? ? ? ? ? Equilibrate matrix TRUE
? ? ? ? ? ? ? Matrix input mode 1
? ? ? ? ? ? ? Replace tiny pivots TRUE
? ? ? ? ? ? ? Use iterative refinement FALSE
? ? ? ? ? ? ? Processors in row 8 col partition 8
? ? ? ? ? ? ? Row permutation LargeDiag
? ? ? ? ? ? ? Parallel symbolic factorization FALSE
? ? ? ? ? ? ? Repeated factorization SamePattern
******************************************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/599f3cb2/attachment-0001.htm>

------------------------------

_______________________________________________
petsc-users mailing list
petsc-users at mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/petsc-users


End of petsc-users Digest, Vol 40, Issue 107
********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/6f4046a4/attachment.htm>

From hzhang at mcs.anl.gov  Sat Apr 28 12:55:13 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Sat, 28 Apr 2012 12:55:13 -0500
Subject: [petsc-users] superlu dist colperm by default
In-Reply-To: <CAMJxm+DQwYW-6prQHAXPCuC3buvkq7md5yVaHqTMtprBjEjDsA@mail.gmail.com>
References: <CAMJxm+DQwYW-6prQHAXPCuC3buvkq7md5yVaHqTMtprBjEjDsA@mail.gmail.com>
Message-ID: <CAGCphBuXoiOqY24UOHGus+E+1E+NuggxNHA2m8J_OLpYSnpacQ@mail.gmail.com>

Wen:

>
> I am using superlu dist in PETSc 3.1. The version of superlu dist
> installed by petsc is SuperLU_DIST_2.4-hg-v2. Does anyone know which
> colperm method superlu dist use for the default setting by petsc? I cannot
> got such information by using -mat_superlu_dist_statprint, the output of
> which I attached below. Thanks.
>

Please update to the latest petsc and superlu_dist (preferable petsc-dev)
PETSc 3.1 and SuperLU_DIST_2.4-hg-v2 are several years old.
I recall there were bugs in colperm. Together with superlu's developer,
these bugs were fixed in latest superLU_DIST and petsc.
I guess 'colperm method superlu dist' was not supported in that version.
It is supported in petsc-dev and SuperLU_DIST_3.0-p1
With petsc-dev, I get
petsc-dev/src/ksp/ksp/examples/tutorials>mpiexec -n 2 ./ex2 -pc_type lu
-pc_factor_mat_solver_package superlu_dist -ksp_view
...
            SuperLU_DIST run parameters:
              Process grid nprow 2 x npcol 1
              Equilibrate matrix TRUE
              Matrix input mode 1
              Replace tiny pivots TRUE
              Use iterative refinement FALSE
              Processors in row 2 col partition 1
              Row permutation LargeDiag
              Column permutation METIS_AT_PLUS_A
              Parallel symbolic factorization FALSE
              Repeated factorization SamePattern_SameRowPerm


Hong

>
> Regards,
> Wen
>
>
> ***************************************************************************************************************
> PC Object:
>   type: lu
>     LU: out-of-place factorization
>     tolerance for zero pivot 1e-12
>     matrix ordering: natural
>     factor fill ratio given 0, needed 0
>       Factored matrix follows:
>         Matrix Object:
>           type=mpiaij, rows=215883, cols=215883
>           package used to perform factorization: superlu_dist
>           total: nonzeros=0, allocated nonzeros=431766
>             SuperLU_DIST run parameters:
>               Process grid nprow 8 x npcol 8
>               Equilibrate matrix TRUE
>               Matrix input mode 1
>               Replace tiny pivots TRUE
>               Use iterative refinement FALSE
>               Processors in row 8 col partition 8
>               Row permutation LargeDiag
>               Parallel symbolic factorization FALSE
>               Repeated factorization SamePattern
>
> ******************************************************************************************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/5f4dbc1f/attachment-0001.htm>

From andrew.spott at gmail.com  Sat Apr 28 17:39:32 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 16:39:32 -0600
Subject: [petsc-users]
 =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <alpine.LFD.2.02.1204272020300.2752@asterix>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
Message-ID: <63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>

>-start_in-debugger noxterm -debugger_nodes 14

All my cores are on the same machine, is this supposed to start a debugger on processor 14? or computer 14?

I don't think I have x11 setup properly for the compute nodes, so x11 isn't really an option.

Thanks for the help.

-Andrew


On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:

> On Fri, 27 Apr 2012, Andrew Spott wrote:
> 
>> I'm honestly stumped.
>> 
>> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
>> 
>> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.
> 
> assuming you have X11 setup properly from compute nodes you can run
> with the extra option '-start_in_debugger'
> 
> If X11 is not properly setup - and you'd like to run gdb on one of the
> nodes [say node 14 where you see SEGV] - you can do:
> 
> -start_in-debugger noxterm -debugger_nodes 14
> 
> Or try valgrind
> 
> mpiexec -n 16 valgrind --tool=memcheck -q ./executable
> 
> 
> For debugging - its best to install with --download-mpich [so that its
> valgrind clean] - and run all mpi stuff on a single machine - [usually
> X11 works well from a single machine.]
> 
> Satish
> 
>> 
>> any ideas?  (long error message below):
>> 
>> -Andrew
>> 
>> [14]PETSC ERROR: ------------------------------------------------------------------------
>> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [14]PETSC ERROR: likely location of problem given in stack below
>> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [14]PETSC ERROR:       is given.
>> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [14]PETSC ERROR: Signal received!
>> [14]PETSC ERROR: ------------------------------------------------------------------------
>> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
>> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [15]PETSC ERROR: likely location of problem given in stack below
>> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [15]PETSC ERROR:       is given.
>> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [15]PETSC ERROR: Signal received!
>> [15]PETSC ERROR: ------------------------------------------------------------------------
>> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
>> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [14]PETSC ERROR: See docs/index.html for manual pages.
>> [14]PETSC ERROR: ------------------------------------------------------------------------
>> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>> [14]PETSC ERROR: ------------------------------------------------------------------------
>> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
>> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [15]PETSC ERROR: See docs/index.html for manual pages.
>> [15]PETSC ERROR: ------------------------------------------------------------------------
>> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>> [15]PETSC ERROR: ------------------------------------------------------------------------
>> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt
> 


From knepley at gmail.com  Sat Apr 28 17:45:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 28 Apr 2012 18:45:04 -0400
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
Message-ID: <CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>

On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> >-start_in-debugger noxterm -debugger_nodes 14
>
> All my cores are on the same machine, is this supposed to start a debugger
> on processor 14? or computer 14?
>

Neither. This spawns a gdb process on the same node as the process with MPI
rank 14. Then attaches gdb
to process 14.

    Matt


> I don't think I have x11 setup properly for the compute nodes, so x11
> isn't really an option.
>
> Thanks for the help.
>
> -Andrew
>
>
> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>
> > On Fri, 27 Apr 2012, Andrew Spott wrote:
> >
> >> I'm honestly stumped.
> >>
> >> I have some petsc code that essentially just populates a matrix in
> parallel, then puts it in a file.  All my code that uses floating point
> computations is checked for NaN's and infinities and it doesn't seem to
> show up.  However, when I run it on more than 4 cores, I get floating point
> exceptions that kill the program.  I tried turning off the exceptions from
> petsc, but the program still dies from them, just without the petsc error
> message.
> >>
> >> I honestly don't know where to go, I suppose I should attach a
> debugger, but I'm not sure how to do that for multi-processor code.
> >
> > assuming you have X11 setup properly from compute nodes you can run
> > with the extra option '-start_in_debugger'
> >
> > If X11 is not properly setup - and you'd like to run gdb on one of the
> > nodes [say node 14 where you see SEGV] - you can do:
> >
> > -start_in-debugger noxterm -debugger_nodes 14
> >
> > Or try valgrind
> >
> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
> >
> >
> > For debugging - its best to install with --download-mpich [so that its
> > valgrind clean] - and run all mpi stuff on a single machine - [usually
> > X11 works well from a single machine.]
> >
> > Satish
> >
> >>
> >> any ideas?  (long error message below):
> >>
> >> -Andrew
> >>
> >> [14]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> Exception,probably divide by zero
> >> [14]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >> [14]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> >> [14]PETSC ERROR: likely location of problem given in stack below
> >> [14]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> >> [14]PETSC ERROR:       is given.
> >> [14]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> >> [14]PETSC ERROR: Signal received!
> >> [14]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [14]PE[15]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> Exception,probably divide by zero
> >> [15]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >> [15]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> >> [15]PETSC ERROR: likely location of problem given in stack below
> >> [15]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> >> [15]PETSC ERROR:       is given.
> >> [15]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> >> [15]PETSC ERROR: Signal received!
> >> [15]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
> 09:28:45 CST 2012
> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >> [14]PETSC ERROR: See docs/index.html for manual pages.
> >> [14]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu
> named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> >> [14]PETSC ERROR: Libraries linked from
> /home/becker/ansp6066/local/petsc-3.2-p6/lib
> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> >> [14]PETSC ERROR: Configure options
> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
> --with-scalar-type=complex
> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
> --with-clanguage=cxx
> >> [14]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
> 09:28:45 CST 2012
> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >> [15]PETSC ERROR: See docs/index.html for manual pages.
> >> [15]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu
> named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> >> [15]PETSC ERROR: Libraries linked from
> /home/becker/ansp6066/local/petsc-3.2-p6/lib
> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> >> [15]PETSC ERROR: Configure options
> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
> --with-scalar-type=complex
> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
> --with-clanguage=cxx
> >> [15]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC
> ERROR: User provided function() line 0 in unknown directory unknown file
> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process
> 15[0]0:Return code = 0, signaled with Interrupt
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/12ad32b2/attachment.htm>

From jiangwen84 at gmail.com  Sat Apr 28 19:07:11 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Sat, 28 Apr 2012 20:07:11 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern(Hong Zhang)
Message-ID: <CAMJxm+AQVw5m_pSMPjFOavn2xe4RS+XHvczOMBvQQRAV875-0w@mail.gmail.com>

Reply to

> This is weird.  Try
> 1) increase work  space with
> -mat_mumps_icntl_14 50 (default is 20)
> 2) different matrix orderings with
> -mat_mumps_icntl_7 2 (or number from 0 to 6)
>
> Run your code with '-log_summary' and see which routine causes this huge
> difference.
>
> Hong
>
>
I just tested the problem according to what you suggested. I set icntl_14 =
50 and icntl_7 = 5 (METIS). The problem still persisted. The first solve
took 920 second and second solve took 215 second with same nonzero pattern
pc set up. I also attached the log_summary output file. Do you have any
further suggestion?  Thanks.

Regards,
Wen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/6b63daf8/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_summary
Type: application/octet-stream
Size: 4596 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/6b63daf8/attachment-0001.obj>

From andrew.spott at gmail.com  Sat Apr 28 19:07:04 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 18:07:04 -0600
Subject: [petsc-users]
 =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
Message-ID: <5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>

are there any tricks to doing this across ssh?

I've attempted it using the method given, but I can't get it to start in the debugger or to attach the debugger, the program just exits or hangs after telling me the error.

-Andrew

On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:

> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> >-start_in-debugger noxterm -debugger_nodes 14
> 
> All my cores are on the same machine, is this supposed to start a debugger on processor 14? or computer 14?
> 
> Neither. This spawns a gdb process on the same node as the process with MPI rank 14. Then attaches gdb
> to process 14.
> 
>     Matt
>  
> I don't think I have x11 setup properly for the compute nodes, so x11 isn't really an option.
> 
> Thanks for the help.
> 
> -Andrew
> 
> 
> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
> 
> > On Fri, 27 Apr 2012, Andrew Spott wrote:
> >
> >> I'm honestly stumped.
> >>
> >> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
> >>
> >> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.
> >
> > assuming you have X11 setup properly from compute nodes you can run
> > with the extra option '-start_in_debugger'
> >
> > If X11 is not properly setup - and you'd like to run gdb on one of the
> > nodes [say node 14 where you see SEGV] - you can do:
> >
> > -start_in-debugger noxterm -debugger_nodes 14
> >
> > Or try valgrind
> >
> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
> >
> >
> > For debugging - its best to install with --download-mpich [so that its
> > valgrind clean] - and run all mpi stuff on a single machine - [usually
> > X11 works well from a single machine.]
> >
> > Satish
> >
> >>
> >> any ideas?  (long error message below):
> >>
> >> -Andrew
> >>
> >> [14]PETSC ERROR: ------------------------------------------------------------------------
> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> >> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> >> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> >> [14]PETSC ERROR: likely location of problem given in stack below
> >> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
> >> [14]PETSC ERROR:       is given.
> >> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
> >> [14]PETSC ERROR: Signal received!
> >> [14]PETSC ERROR: ------------------------------------------------------------------------
> >> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> >> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> >> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> >> [15]PETSC ERROR: likely location of problem given in stack below
> >> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
> >> [15]PETSC ERROR:       is given.
> >> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
> >> [15]PETSC ERROR: Signal received!
> >> [15]PETSC ERROR: ------------------------------------------------------------------------
> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >> [14]PETSC ERROR: See docs/index.html for manual pages.
> >> [14]PETSC ERROR: ------------------------------------------------------------------------
> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> >> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> >> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
> >> [14]PETSC ERROR: ------------------------------------------------------------------------
> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >> [15]PETSC ERROR: See docs/index.html for manual pages.
> >> [15]PETSC ERROR: ------------------------------------------------------------------------
> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
> >> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
> >> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
> >> [15]PETSC ERROR: ------------------------------------------------------------------------
> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/62e01bf3/attachment.htm>

From knepley at gmail.com  Sat Apr 28 19:11:37 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 28 Apr 2012 20:11:37 -0400
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
Message-ID: <CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>

On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> are there any tricks to doing this across ssh?
>
> I've attempted it using the method given, but I can't get it to start in
> the debugger or to attach the debugger, the program just exits or hangs
> after telling me the error.
>

Is there a reason you cannot run this problem on your local machine with 4
processes?

   Matt


> -Andrew
>
> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>
> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> >-start_in-debugger noxterm -debugger_nodes 14
>>
>> All my cores are on the same machine, is this supposed to start a
>> debugger on processor 14? or computer 14?
>>
>
> Neither. This spawns a gdb process on the same node as the process with
> MPI rank 14. Then attaches gdb
> to process 14.
>
>     Matt
>
>
>> I don't think I have x11 setup properly for the compute nodes, so x11
>> isn't really an option.
>>
>> Thanks for the help.
>>
>> -Andrew
>>
>>
>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>
>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>> >
>> >> I'm honestly stumped.
>> >>
>> >> I have some petsc code that essentially just populates a matrix in
>> parallel, then puts it in a file.  All my code that uses floating point
>> computations is checked for NaN's and infinities and it doesn't seem to
>> show up.  However, when I run it on more than 4 cores, I get floating point
>> exceptions that kill the program.  I tried turning off the exceptions from
>> petsc, but the program still dies from them, just without the petsc error
>> message.
>> >>
>> >> I honestly don't know where to go, I suppose I should attach a
>> debugger, but I'm not sure how to do that for multi-processor code.
>> >
>> > assuming you have X11 setup properly from compute nodes you can run
>> > with the extra option '-start_in_debugger'
>> >
>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>> > nodes [say node 14 where you see SEGV] - you can do:
>> >
>> > -start_in-debugger noxterm -debugger_nodes 14
>> >
>> > Or try valgrind
>> >
>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>> >
>> >
>> > For debugging - its best to install with --download-mpich [so that its
>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>> > X11 works well from a single machine.]
>> >
>> > Satish
>> >
>> >>
>> >> any ideas?  (long error message below):
>> >>
>> >> -Andrew
>> >>
>> >> [14]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>> Exception,probably divide by zero
>> >> [14]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> >> [14]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> >> [14]PETSC ERROR: likely location of problem given in stack below
>> >> [14]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> >> [14]PETSC ERROR:       is given.
>> >> [14]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> >> [14]PETSC ERROR: Signal received!
>> >> [14]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [14]PE[15]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>> Exception,probably divide by zero
>> >> [15]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> >> [15]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> >> [15]PETSC ERROR: likely location of problem given in stack below
>> >> [15]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> >> [15]PETSC ERROR:       is given.
>> >> [15]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> >> [15]PETSC ERROR: Signal received!
>> >> [15]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>> 09:28:45 CST 2012
>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>> >> [14]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> >> [14]PETSC ERROR: Libraries linked from
>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> >> [14]PETSC ERROR: Configure options
>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>> --with-scalar-type=complex
>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>> --with-clanguage=cxx
>> >> [14]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>> 09:28:45 CST 2012
>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>> >> [15]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> >> [15]PETSC ERROR: Libraries linked from
>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> >> [15]PETSC ERROR: Configure options
>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>> --with-scalar-type=complex
>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>> --with-clanguage=cxx
>> >> [15]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC
>> ERROR: User provided function() line 0 in unknown directory unknown file
>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process
>> 15[0]0:Return code = 0, signaled with Interrupt
>> >
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/e4a76fe0/attachment-0001.htm>

From andrew.spott at gmail.com  Sat Apr 28 19:36:29 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 18:36:29 -0600
Subject: [petsc-users]
 =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
Message-ID: <63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>

When I attach debugger on error on the local machine, I get a bunch of lines like this one:

warning: Could not find object file "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)" - no debug information available for "cp-demangle.c".

then ". done"  And then nothing.  It looks like the program exits before the debugger can attach.  after a while I get this:

/Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151: No such file or directory
Unable to access task for process-id 76151: (os/kern) failure.

in the gdb window.  In the terminal window, I get

application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
[cli_3]: aborting job:

if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing, but everything else is the same.

any ideas?

-Andrew

On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:

> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> are there any tricks to doing this across ssh?
> 
> I've attempted it using the method given, but I can't get it to start in the debugger or to attach the debugger, the program just exits or hangs after telling me the error.
> 
> Is there a reason you cannot run this problem on your local machine with 4 processes?
> 
>    Matt
>  
> -Andrew
> 
> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
> 
>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>> >-start_in-debugger noxterm -debugger_nodes 14
>> 
>> All my cores are on the same machine, is this supposed to start a debugger on processor 14? or computer 14?
>> 
>> Neither. This spawns a gdb process on the same node as the process with MPI rank 14. Then attaches gdb
>> to process 14.
>> 
>>     Matt
>>  
>> I don't think I have x11 setup properly for the compute nodes, so x11 isn't really an option.
>> 
>> Thanks for the help.
>> 
>> -Andrew
>> 
>> 
>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>> 
>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>> >
>> >> I'm honestly stumped.
>> >>
>> >> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
>> >>
>> >> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.
>> >
>> > assuming you have X11 setup properly from compute nodes you can run
>> > with the extra option '-start_in_debugger'
>> >
>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>> > nodes [say node 14 where you see SEGV] - you can do:
>> >
>> > -start_in-debugger noxterm -debugger_nodes 14
>> >
>> > Or try valgrind
>> >
>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>> >
>> >
>> > For debugging - its best to install with --download-mpich [so that its
>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>> > X11 works well from a single machine.]
>> >
>> > Satish
>> >
>> >>
>> >> any ideas?  (long error message below):
>> >>
>> >> -Andrew
>> >>
>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>> >> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> >> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> >> [14]PETSC ERROR: likely location of problem given in stack below
>> >> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
>> >> [14]PETSC ERROR:       is given.
>> >> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
>> >> [14]PETSC ERROR: Signal received!
>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>> >> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>> >> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> >> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> >> [15]PETSC ERROR: likely location of problem given in stack below
>> >> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
>> >> [15]PETSC ERROR:       is given.
>> >> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
>> >> [15]PETSC ERROR: Signal received!
>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> >> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> >> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>> >> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>> >> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt
>> >
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/88752663/attachment.htm>

From knepley at gmail.com  Sat Apr 28 19:42:54 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 28 Apr 2012 20:42:54 -0400
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
Message-ID: <CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>

On Sat, Apr 28, 2012 at 8:36 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> When I attach debugger on error on the local machine, I get a bunch of
> lines like this one:
>
> warning: Could not find object file
> "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)"
> - no debug information available for "cp-demangle.c".
>

It looks like you build with autotools. That just makes things hard :)


> then ". done"  And then nothing.  It looks like the program exits before
> the debugger can attach.  after a while I get this:
>

You can use -debugger_pause 10 to make it wait 10s before continuing after
spawning the debugger. Make it long
enough to attach.

   Matt


> /Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151:
> No such file or directory
> Unable to access task for process-id 76151: (os/kern) failure.
>
> in the gdb window.  In the terminal window, I get
>
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
> [cli_3]: aborting job:
>
> if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing,
> but everything else is the same.
>
> any ideas?
>
> -Andrew
>
> On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:
>
> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> are there any tricks to doing this across ssh?
>>
>> I've attempted it using the method given, but I can't get it to start in
>> the debugger or to attach the debugger, the program just exits or hangs
>> after telling me the error.
>>
>
> Is there a reason you cannot run this problem on your local machine with 4
> processes?
>
>    Matt
>
>
>> -Andrew
>>
>> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>>
>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>
>>> >-start_in-debugger noxterm -debugger_nodes 14
>>>
>>> All my cores are on the same machine, is this supposed to start a
>>> debugger on processor 14? or computer 14?
>>>
>>
>> Neither. This spawns a gdb process on the same node as the process with
>> MPI rank 14. Then attaches gdb
>> to process 14.
>>
>>     Matt
>>
>>
>>> I don't think I have x11 setup properly for the compute nodes, so x11
>>> isn't really an option.
>>>
>>> Thanks for the help.
>>>
>>> -Andrew
>>>
>>>
>>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>>
>>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>>> >
>>> >> I'm honestly stumped.
>>> >>
>>> >> I have some petsc code that essentially just populates a matrix in
>>> parallel, then puts it in a file.  All my code that uses floating point
>>> computations is checked for NaN's and infinities and it doesn't seem to
>>> show up.  However, when I run it on more than 4 cores, I get floating point
>>> exceptions that kill the program.  I tried turning off the exceptions from
>>> petsc, but the program still dies from them, just without the petsc error
>>> message.
>>> >>
>>> >> I honestly don't know where to go, I suppose I should attach a
>>> debugger, but I'm not sure how to do that for multi-processor code.
>>> >
>>> > assuming you have X11 setup properly from compute nodes you can run
>>> > with the extra option '-start_in_debugger'
>>> >
>>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>>> > nodes [say node 14 where you see SEGV] - you can do:
>>> >
>>> > -start_in-debugger noxterm -debugger_nodes 14
>>> >
>>> > Or try valgrind
>>> >
>>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>>> >
>>> >
>>> > For debugging - its best to install with --download-mpich [so that its
>>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>>> > X11 works well from a single machine.]
>>> >
>>> > Satish
>>> >
>>> >>
>>> >> any ideas?  (long error message below):
>>> >>
>>> >> -Andrew
>>> >>
>>> >> [14]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>> Exception,probably divide by zero
>>> >> [14]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> >> [14]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSCERROR: or try
>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>> corruption errors
>>> >> [14]PETSC ERROR: likely location of problem given in stack below
>>> >> [14]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> >> [14]PETSC ERROR:       is given.
>>> >> [14]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> >> [14]PETSC ERROR: Signal received!
>>> >> [14]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [14]PE[15]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>> Exception,probably divide by zero
>>> >> [15]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> >> [15]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSCERROR: or try
>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>> corruption errors
>>> >> [15]PETSC ERROR: likely location of problem given in stack below
>>> >> [15]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> >> [15]PETSC ERROR:       is given.
>>> >> [15]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> >> [15]PETSC ERROR: Signal received!
>>> >> [15]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>> 09:28:45 CST 2012
>>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>>> >> [14]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>> 2012
>>> >> [14]PETSC ERROR: Libraries linked from
>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>> >> [14]PETSC ERROR: Configure options
>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>> --with-scalar-type=complex
>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>> --with-clanguage=cxx
>>> >> [14]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>> 09:28:45 CST 2012
>>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>>> >> [15]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>> 2012
>>> >> [15]PETSC ERROR: Libraries linked from
>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>> >> [15]PETSC ERROR: Configure options
>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>> --with-scalar-type=complex
>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>> --with-clanguage=cxx
>>> >> [15]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory
>>> unknown file
>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC
>>> ERROR: User provided function() line 0 in unknown directory unknown file
>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process
>>> 15[0]0:Return code = 0, signaled with Interrupt
>>> >
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/2bfe57b1/attachment-0001.htm>

From sean at mcs.anl.gov  Sat Apr 28 19:53:41 2012
From: sean at mcs.anl.gov (Sean Farley)
Date: Sat, 28 Apr 2012 19:53:41 -0500
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
	<CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
Message-ID: <CANH+MCjS_uqGCiN_cRtUg9BQKPuzLvCX6tSVUcEfBFQ7SVFSJw@mail.gmail.com>

>> When I attach debugger on error on the local machine, I get a bunch of
>> lines like this one:
>>
>> warning: Could not find object file
>> "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)"
>> - no debug information available for "cp-demangle.c".
>
>
> It looks like you build with autotools. That just makes things hard :)

What? Matt, stop spreading misinformation; autotools has *nothing* to
do with this. You're on a mac and should know better.

Andrew, you get this warnings because the package system (homebrew, in
your case) didn't generate and save the debug symbols for gcc-4.6.2. I
have been trying to add this to MacPorts (but failed) and I might try
to do this with homebrew but until then, you'll have to do something
like the following:

1) tell homebrew to build and destroot gcc
2) manually go to the destroot directory (wherever that is) and run
dsymutil on all the .dylib files. A command to do that is `find
/path/to/destroot -name '*.dylib' -exec dsymutil {} +`
3) continue the homebrew installation (hoping that it picks up the *.dSYM files

Of course, you could just ignore the warnings :-)

As for the other problems, you might want to learn how to launch this
in gdb from the terminal.

From andrew.spott at gmail.com  Sat Apr 28 19:59:03 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 18:59:03 -0600
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
	<CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
Message-ID: <CAPbJa2jMQssn1cm_UkW-YX+1ZwLDDXFLtBKPd8nYbKydOJSm3A@mail.gmail.com>

What makes it easier if autotools makes it hard?
On Apr 28, 2012 6:43 PM, "Matthew Knepley" <knepley at gmail.com> wrote:

> On Sat, Apr 28, 2012 at 8:36 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> When I attach debugger on error on the local machine, I get a bunch of
>> lines like this one:
>>
>> warning: Could not find object file
>> "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)"
>> - no debug information available for "cp-demangle.c".
>>
>
> It looks like you build with autotools. That just makes things hard :)
>
>
>> then ". done"  And then nothing.  It looks like the program exits before
>> the debugger can attach.  after a while I get this:
>>
>
> You can use -debugger_pause 10 to make it wait 10s before continuing after
> spawning the debugger. Make it long
> enough to attach.
>
>    Matt
>
>
>> /Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151:
>> No such file or directory
>> Unable to access task for process-id 76151: (os/kern) failure.
>>
>> in the gdb window.  In the terminal window, I get
>>
>> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
>> [cli_3]: aborting job:
>>
>> if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing,
>> but everything else is the same.
>>
>> any ideas?
>>
>> -Andrew
>>
>> On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:
>>
>> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>
>>> are there any tricks to doing this across ssh?
>>>
>>> I've attempted it using the method given, but I can't get it to start in
>>> the debugger or to attach the debugger, the program just exits or hangs
>>> after telling me the error.
>>>
>>
>> Is there a reason you cannot run this problem on your local machine with
>> 4 processes?
>>
>>    Matt
>>
>>
>>> -Andrew
>>>
>>> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>>>
>>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>>
>>>> >-start_in-debugger noxterm -debugger_nodes 14
>>>>
>>>> All my cores are on the same machine, is this supposed to start a
>>>> debugger on processor 14? or computer 14?
>>>>
>>>
>>> Neither. This spawns a gdb process on the same node as the process with
>>> MPI rank 14. Then attaches gdb
>>> to process 14.
>>>
>>>     Matt
>>>
>>>
>>>> I don't think I have x11 setup properly for the compute nodes, so x11
>>>> isn't really an option.
>>>>
>>>> Thanks for the help.
>>>>
>>>> -Andrew
>>>>
>>>>
>>>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>>>
>>>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>>>> >
>>>> >> I'm honestly stumped.
>>>> >>
>>>> >> I have some petsc code that essentially just populates a matrix in
>>>> parallel, then puts it in a file.  All my code that uses floating point
>>>> computations is checked for NaN's and infinities and it doesn't seem to
>>>> show up.  However, when I run it on more than 4 cores, I get floating point
>>>> exceptions that kill the program.  I tried turning off the exceptions from
>>>> petsc, but the program still dies from them, just without the petsc error
>>>> message.
>>>> >>
>>>> >> I honestly don't know where to go, I suppose I should attach a
>>>> debugger, but I'm not sure how to do that for multi-processor code.
>>>> >
>>>> > assuming you have X11 setup properly from compute nodes you can run
>>>> > with the extra option '-start_in_debugger'
>>>> >
>>>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>>>> > nodes [say node 14 where you see SEGV] - you can do:
>>>> >
>>>> > -start_in-debugger noxterm -debugger_nodes 14
>>>> >
>>>> > Or try valgrind
>>>> >
>>>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>>>> >
>>>> >
>>>> > For debugging - its best to install with --download-mpich [so that its
>>>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>>>> > X11 works well from a single machine.]
>>>> >
>>>> > Satish
>>>> >
>>>> >>
>>>> >> any ideas?  (long error message below):
>>>> >>
>>>> >> -Andrew
>>>> >>
>>>> >> [14]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>>> Exception,probably divide by zero
>>>> >> [14]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> >> [14]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSCERROR: or try
>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>>> corruption errors
>>>> >> [14]PETSC ERROR: likely location of problem given in stack below
>>>> >> [14]PETSC ERROR: ---------------------  Stack Frames
>>>> ------------------------------------
>>>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>> available,
>>>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the
>>>> function
>>>> >> [14]PETSC ERROR:       is given.
>>>> >> [14]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> >> [14]PETSC ERROR: Signal received!
>>>> >> [14]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [14]PE[15]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>>> Exception,probably divide by zero
>>>> >> [15]PETSC ERROR: Try option -start_in_debugger or
>>>> -on_error_attach_debugger
>>>> >> [15]PETSC ERROR: or see
>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSCERROR: or try
>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>>> corruption errors
>>>> >> [15]PETSC ERROR: likely location of problem given in stack below
>>>> >> [15]PETSC ERROR: ---------------------  Stack Frames
>>>> ------------------------------------
>>>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>> available,
>>>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the
>>>> function
>>>> >> [15]PETSC ERROR:       is given.
>>>> >> [15]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> >> [15]PETSC ERROR: Signal received!
>>>> >> [15]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>>> 09:28:45 CST 2012
>>>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>>>> >> [14]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>>> 2012
>>>> >> [14]PETSC ERROR: Libraries linked from
>>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>> >> [14]PETSC ERROR: Configure options
>>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>>> --with-scalar-type=complex
>>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>>> --with-clanguage=cxx
>>>> >> [14]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>>> 09:28:45 CST 2012
>>>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>>>> >> [15]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>>> 2012
>>>> >> [15]PETSC ERROR: Libraries linked from
>>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>> >> [15]PETSC ERROR: Configure options
>>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>>> --with-scalar-type=complex
>>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>>> --with-clanguage=cxx
>>>> >> [15]PETSC ERROR:
>>>> ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: User provided function() line 0 in unknown
>>>> directory unknown file
>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC
>>>> ERROR: User provided function() line 0 in unknown directory unknown file
>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process
>>>> 15[0]0:Return code = 0, signaled with Interrupt
>>>> >
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/63d66b53/attachment.htm>

From knepley at gmail.com  Sat Apr 28 20:10:45 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 28 Apr 2012 21:10:45 -0400
Subject: [petsc-users]
	=?windows-1252?q?floating_point_exception=85_but_on?=
	=?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAPbJa2jMQssn1cm_UkW-YX+1ZwLDDXFLtBKPd8nYbKydOJSm3A@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
	<CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
	<CAPbJa2jMQssn1cm_UkW-YX+1ZwLDDXFLtBKPd8nYbKydOJSm3A@mail.gmail.com>
Message-ID: <CAMYG4GnD88ockcsizhF6yUL3yuyQqM7+O78_+jatpWMbNXE4tA@mail.gmail.com>

On Sat, Apr 28, 2012 at 8:59 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> What makes it easier if autotools makes it hard?
>

This is a joke, but I do think autotools makes everything hard.


> On Apr 28, 2012 6:43 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>
>> On Sat, Apr 28, 2012 at 8:36 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>
>>> When I attach debugger on error on the local machine, I get a bunch of
>>> lines like this one:
>>>
>>> warning: Could not find object file
>>> "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)"
>>> - no debug information available for "cp-demangle.c".
>>>
>>
>> It looks like you build with autotools. That just makes things hard :)
>>
>>
>>> then ". done"  And then nothing.  It looks like the program exits before
>>> the debugger can attach.  after a while I get this:
>>>
>>
>> You can use -debugger_pause 10 to make it wait 10s before continuing
>> after spawning the debugger. Make it long
>> enough to attach.
>>
>
Did this work?

   Matt


>    Matt
>>
>>
>>> /Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151:
>>> No such file or directory
>>> Unable to access task for process-id 76151: (os/kern) failure.
>>>
>>> in the gdb window.  In the terminal window, I get
>>>
>>> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
>>> [cli_3]: aborting job:
>>>
>>> if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing,
>>> but everything else is the same.
>>>
>>> any ideas?
>>>
>>> -Andrew
>>>
>>> On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:
>>>
>>> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>>
>>>> are there any tricks to doing this across ssh?
>>>>
>>>> I've attempted it using the method given, but I can't get it to start
>>>> in the debugger or to attach the debugger, the program just exits or hangs
>>>> after telling me the error.
>>>>
>>>
>>> Is there a reason you cannot run this problem on your local machine with
>>> 4 processes?
>>>
>>>    Matt
>>>
>>>
>>>> -Andrew
>>>>
>>>> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>>>>
>>>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>>>
>>>>> >-start_in-debugger noxterm -debugger_nodes 14
>>>>>
>>>>> All my cores are on the same machine, is this supposed to start a
>>>>> debugger on processor 14? or computer 14?
>>>>>
>>>>
>>>> Neither. This spawns a gdb process on the same node as the process with
>>>> MPI rank 14. Then attaches gdb
>>>> to process 14.
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> I don't think I have x11 setup properly for the compute nodes, so x11
>>>>> isn't really an option.
>>>>>
>>>>> Thanks for the help.
>>>>>
>>>>> -Andrew
>>>>>
>>>>>
>>>>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>>>>
>>>>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>>>>> >
>>>>> >> I'm honestly stumped.
>>>>> >>
>>>>> >> I have some petsc code that essentially just populates a matrix in
>>>>> parallel, then puts it in a file.  All my code that uses floating point
>>>>> computations is checked for NaN's and infinities and it doesn't seem to
>>>>> show up.  However, when I run it on more than 4 cores, I get floating point
>>>>> exceptions that kill the program.  I tried turning off the exceptions from
>>>>> petsc, but the program still dies from them, just without the petsc error
>>>>> message.
>>>>> >>
>>>>> >> I honestly don't know where to go, I suppose I should attach a
>>>>> debugger, but I'm not sure how to do that for multi-processor code.
>>>>> >
>>>>> > assuming you have X11 setup properly from compute nodes you can run
>>>>> > with the extra option '-start_in_debugger'
>>>>> >
>>>>> > If X11 is not properly setup - and you'd like to run gdb on one of
>>>>> the
>>>>> > nodes [say node 14 where you see SEGV] - you can do:
>>>>> >
>>>>> > -start_in-debugger noxterm -debugger_nodes 14
>>>>> >
>>>>> > Or try valgrind
>>>>> >
>>>>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>>>>> >
>>>>> >
>>>>> > For debugging - its best to install with --download-mpich [so that
>>>>> its
>>>>> > valgrind clean] - and run all mpi stuff on a single machine -
>>>>> [usually
>>>>> > X11 works well from a single machine.]
>>>>> >
>>>>> > Satish
>>>>> >
>>>>> >>
>>>>> >> any ideas?  (long error message below):
>>>>> >>
>>>>> >> -Andrew
>>>>> >>
>>>>> >> [14]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>>>> Exception,probably divide by zero
>>>>> >> [14]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> >> [14]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSCERROR: or try
>>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>>>> corruption errors
>>>>> >> [14]PETSC ERROR: likely location of problem given in stack below
>>>>> >> [14]PETSC ERROR: ---------------------  Stack Frames
>>>>> ------------------------------------
>>>>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>> available,
>>>>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>> function
>>>>> >> [14]PETSC ERROR:       is given.
>>>>> >> [14]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>> >> [14]PETSC ERROR: Signal received!
>>>>> >> [14]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [14]PE[15]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
>>>>> Exception,probably divide by zero
>>>>> >> [15]PETSC ERROR: Try option -start_in_debugger or
>>>>> -on_error_attach_debugger
>>>>> >> [15]PETSC ERROR: or see
>>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSCERROR: or try
>>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>>>>> corruption errors
>>>>> >> [15]PETSC ERROR: likely location of problem given in stack below
>>>>> >> [15]PETSC ERROR: ---------------------  Stack Frames
>>>>> ------------------------------------
>>>>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>>>> available,
>>>>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the
>>>>> function
>>>>> >> [15]PETSC ERROR:       is given.
>>>>> >> [15]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>> >> [15]PETSC ERROR: Signal received!
>>>>> >> [15]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>>>> 09:28:45 CST 2012
>>>>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>>>>> >> [14]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>>>> 2012
>>>>> >> [14]PETSC ERROR: Libraries linked from
>>>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>>> >> [14]PETSC ERROR: Configure options
>>>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>>>> --with-scalar-type=complex
>>>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>>>> --with-clanguage=cxx
>>>>> >> [14]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11
>>>>> 09:28:45 CST 2012
>>>>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>>>>> >> [15]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a
>>>>> linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55
>>>>> 2012
>>>>> >> [15]PETSC ERROR: Libraries linked from
>>>>> /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>>> >> [15]PETSC ERROR: Configure options
>>>>> --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support
>>>>> --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0
>>>>> --with-scalar-type=complex
>>>>> --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a
>>>>> --with-clanguage=cxx
>>>>> >> [15]PETSC ERROR:
>>>>> ------------------------------------------------------------------------
>>>>> >> [15]PETSC ERROR: User provided function() line 0 in unknown
>>>>> directory unknown file
>>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC
>>>>> ERROR: User provided function() line 0 in unknown directory unknown file
>>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process
>>>>> 15[0]0:Return code = 0, signaled with Interrupt
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/7274b441/attachment-0001.htm>

From andrew.spott at gmail.com  Sat Apr 28 20:16:52 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 19:16:52 -0600
Subject: [petsc-users]
 =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <CAMYG4GnD88ockcsizhF6yUL3yuyQqM7+O78_+jatpWMbNXE4tA@mail.gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
	<CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
	<CAPbJa2jMQssn1cm_UkW-YX+1ZwLDDXFLtBKPd8nYbKydOJSm3A@mail.gmail.com>
	<CAMYG4GnD88ockcsizhF6yUL3yuyQqM7+O78_+jatpWMbNXE4tA@mail.gmail.com>
Message-ID: <2E191124-4A63-4E6D-94D4-0CB65682E944@gmail.com>

Yes it did? now I have a bunch of GDB windows open and ready, thanks.

-Andrew
On Apr 28, 2012, at 7:10 PM, Matthew Knepley wrote:

> On Sat, Apr 28, 2012 at 8:59 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> What makes it easier if autotools makes it hard?
> 
> 
> This is a joke, but I do think autotools makes everything hard.
>  
> On Apr 28, 2012 6:43 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
> On Sat, Apr 28, 2012 at 8:36 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> When I attach debugger on error on the local machine, I get a bunch of lines like this one:
> 
> warning: Could not find object file "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)" - no debug information available for "cp-demangle.c".
> 
> It looks like you build with autotools. That just makes things hard :)
>  
> then ". done"  And then nothing.  It looks like the program exits before the debugger can attach.  after a while I get this:
> 
> You can use -debugger_pause 10 to make it wait 10s before continuing after spawning the debugger. Make it long
> enough to attach.
> 
> Did this work?
> 
>    Matt
>  
>    Matt
>  
> /Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151: No such file or directory
> Unable to access task for process-id 76151: (os/kern) failure.
> 
> in the gdb window.  In the terminal window, I get
> 
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
> [cli_3]: aborting job:
> 
> if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing, but everything else is the same.
> 
> any ideas?
> 
> -Andrew
> 
> On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:
> 
>> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>> are there any tricks to doing this across ssh?
>> 
>> I've attempted it using the method given, but I can't get it to start in the debugger or to attach the debugger, the program just exits or hangs after telling me the error.
>> 
>> Is there a reason you cannot run this problem on your local machine with 4 processes?
>> 
>>    Matt
>>  
>> -Andrew
>> 
>> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>> 
>>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>> >-start_in-debugger noxterm -debugger_nodes 14
>>> 
>>> All my cores are on the same machine, is this supposed to start a debugger on processor 14? or computer 14?
>>> 
>>> Neither. This spawns a gdb process on the same node as the process with MPI rank 14. Then attaches gdb
>>> to process 14.
>>> 
>>>     Matt
>>>  
>>> I don't think I have x11 setup properly for the compute nodes, so x11 isn't really an option.
>>> 
>>> Thanks for the help.
>>> 
>>> -Andrew
>>> 
>>> 
>>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>> 
>>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>>> >
>>> >> I'm honestly stumped.
>>> >>
>>> >> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
>>> >>
>>> >> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.
>>> >
>>> > assuming you have X11 setup properly from compute nodes you can run
>>> > with the extra option '-start_in_debugger'
>>> >
>>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>>> > nodes [say node 14 where you see SEGV] - you can do:
>>> >
>>> > -start_in-debugger noxterm -debugger_nodes 14
>>> >
>>> > Or try valgrind
>>> >
>>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>>> >
>>> >
>>> > For debugging - its best to install with --download-mpich [so that its
>>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>>> > X11 works well from a single machine.]
>>> >
>>> > Satish
>>> >
>>> >>
>>> >> any ideas?  (long error message below):
>>> >>
>>> >> -Andrew
>>> >>
>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>>> >> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> >> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> >> [14]PETSC ERROR: likely location of problem given in stack below
>>> >> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
>>> >> [14]PETSC ERROR:       is given.
>>> >> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> >> [14]PETSC ERROR: Signal received!
>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>>> >> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> >> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> >> [15]PETSC ERROR: likely location of problem given in stack below
>>> >> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
>>> >> [15]PETSC ERROR:       is given.
>>> >> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> >> [15]PETSC ERROR: Signal received!
>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>>> >> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>> >> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>>> >> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>> >> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt
>>> >
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/48be9e38/attachment.htm>

From andrew.spott at gmail.com  Sat Apr 28 20:37:43 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Sat, 28 Apr 2012 19:37:43 -0600
Subject: [petsc-users]
 =?windows-1252?q?floating_point_exception=85_but_on?=
 =?windows-1252?q?ly_when_=3E4_cores_are_used=2E=2E=2E?=
In-Reply-To: <2E191124-4A63-4E6D-94D4-0CB65682E944@gmail.com>
References: <63EA8885-28E6-4083-A038-C1182C5D794B@gmail.com>
	<alpine.LFD.2.02.1204272020300.2752@asterix>
	<63662773-FF2B-4899-B36B-668CEF052E86@gmail.com>
	<CAMYG4GmPw-76z_gMJum9puEC2+04wWRGmSwXdxiDimQjP5gJVQ@mail.gmail.com>
	<5B6CACEF-33B3-4928-BEDA-7E06B9B549D7@gmail.com>
	<CAMYG4GnzPpYo7xnL4FJ87EE2L++oktfmNpnEG6ib0mqBBvFSug@mail.gmail.com>
	<63262BB6-4200-4FA1-97B2-806CA6AE4D42@gmail.com>
	<CAMYG4GmqNkEtJ=bH2tL4kRq32DqWb14ePLkPQ2m1NtPK6H5fGg@mail.gmail.com>
	<CAPbJa2jMQssn1cm_UkW-YX+1ZwLDDXFLtBKPd8nYbKydOJSm3A@mail.gmail.com>
	<CAMYG4GnD88ockcsizhF6yUL3yuyQqM7+O78_+jatpWMbNXE4tA@mail.gmail.com>
	<2E191124-4A63-4E6D-94D4-0CB65682E944@gmail.com>
Message-ID: <09F64899-8DA9-41AB-B743-C6F2E0331750@gmail.com>

I found my bug.

Thanks a lot everyone, I really appreciate the help.

-Andrew

On Apr 28, 2012, at 7:16 PM, Andrew Spott wrote:

> Yes it did? now I have a bunch of GDB windows open and ready, thanks.
> 
> -Andrew
> On Apr 28, 2012, at 7:10 PM, Matthew Knepley wrote:
> 
>> On Sat, Apr 28, 2012 at 8:59 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>> What makes it easier if autotools makes it hard?
>> 
>> 
>> This is a joke, but I do think autotools makes everything hard.
>>  
>> On Apr 28, 2012 6:43 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>> On Sat, Apr 28, 2012 at 8:36 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>> When I attach debugger on error on the local machine, I get a bunch of lines like this one:
>> 
>> warning: Could not find object file "/private/tmp/homebrew-gcc-4.6.2-HNPr/gcc-4.6.2/build/x86_64-apple-darwin11.3.0/libstdc++-v3/src/../libsupc++/.libs/libsupc++convenience.a(cp-demangle.o)" - no debug information available for "cp-demangle.c".
>> 
>> It looks like you build with autotools. That just makes things hard :)
>>  
>> then ". done"  And then nothing.  It looks like the program exits before the debugger can attach.  after a while I get this:
>> 
>> You can use -debugger_pause 10 to make it wait 10s before continuing after spawning the debugger. Make it long
>> enough to attach.
>> 
>> Did this work?
>> 
>>    Matt
>>  
>>    Matt
>>  
>> /Users/spott/Documents/Code/EnergyBasisSchrodingerSolver/data/ebss-input/basis_rmax1.00e+02_rmin1.00e-06_dr1.00e-01/76151: No such file or directory
>> Unable to access task for process-id 76151: (os/kern) failure.
>> 
>> in the gdb window.  In the terminal window, I get
>> 
>> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 3
>> [cli_3]: aborting job:
>> 
>> if I just "start_in_debugger"  I just don't' get the "MPI_Abort" thing, but everything else is the same.
>> 
>> any ideas?
>> 
>> -Andrew
>> 
>> On Apr 28, 2012, at 6:11 PM, Matthew Knepley wrote:
>> 
>>> On Sat, Apr 28, 2012 at 8:07 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>> are there any tricks to doing this across ssh?
>>> 
>>> I've attempted it using the method given, but I can't get it to start in the debugger or to attach the debugger, the program just exits or hangs after telling me the error.
>>> 
>>> Is there a reason you cannot run this problem on your local machine with 4 processes?
>>> 
>>>    Matt
>>>  
>>> -Andrew
>>> 
>>> On Apr 28, 2012, at 4:45 PM, Matthew Knepley wrote:
>>> 
>>>> On Sat, Apr 28, 2012 at 6:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>>> >-start_in-debugger noxterm -debugger_nodes 14
>>>> 
>>>> All my cores are on the same machine, is this supposed to start a debugger on processor 14? or computer 14?
>>>> 
>>>> Neither. This spawns a gdb process on the same node as the process with MPI rank 14. Then attaches gdb
>>>> to process 14.
>>>> 
>>>>     Matt
>>>>  
>>>> I don't think I have x11 setup properly for the compute nodes, so x11 isn't really an option.
>>>> 
>>>> Thanks for the help.
>>>> 
>>>> -Andrew
>>>> 
>>>> 
>>>> On Apr 27, 2012, at 7:26 PM, Satish Balay wrote:
>>>> 
>>>> > On Fri, 27 Apr 2012, Andrew Spott wrote:
>>>> >
>>>> >> I'm honestly stumped.
>>>> >>
>>>> >> I have some petsc code that essentially just populates a matrix in parallel, then puts it in a file.  All my code that uses floating point computations is checked for NaN's and infinities and it doesn't seem to show up.  However, when I run it on more than 4 cores, I get floating point exceptions that kill the program.  I tried turning off the exceptions from petsc, but the program still dies from them, just without the petsc error message.
>>>> >>
>>>> >> I honestly don't know where to go, I suppose I should attach a debugger, but I'm not sure how to do that for multi-processor code.
>>>> >
>>>> > assuming you have X11 setup properly from compute nodes you can run
>>>> > with the extra option '-start_in_debugger'
>>>> >
>>>> > If X11 is not properly setup - and you'd like to run gdb on one of the
>>>> > nodes [say node 14 where you see SEGV] - you can do:
>>>> >
>>>> > -start_in-debugger noxterm -debugger_nodes 14
>>>> >
>>>> > Or try valgrind
>>>> >
>>>> > mpiexec -n 16 valgrind --tool=memcheck -q ./executable
>>>> >
>>>> >
>>>> > For debugging - its best to install with --download-mpich [so that its
>>>> > valgrind clean] - and run all mpi stuff on a single machine - [usually
>>>> > X11 works well from a single machine.]
>>>> >
>>>> > Satish
>>>> >
>>>> >>
>>>> >> any ideas?  (long error message below):
>>>> >>
>>>> >> -Andrew
>>>> >>
>>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [14]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>>>> >> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>> >> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>> >> [14]PETSC ERROR: likely location of problem given in stack below
>>>> >> [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>>> >> [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>>> >> [14]PETSC ERROR:       INSTEAD the line number of the start of the function
>>>> >> [14]PETSC ERROR:       is given.
>>>> >> [14]PETSC ERROR: --------------------- Error Message ------------------------------------
>>>> >> [14]PETSC ERROR: Signal received!
>>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [14]PE[15]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>>>> >> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>> >> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>> >> [15]PETSC ERROR: likely location of problem given in stack below
>>>> >> [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>>> >> [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>>> >> [15]PETSC ERROR:       INSTEAD the line number of the start of the function
>>>> >> [15]PETSC ERROR:       is given.
>>>> >> [15]PETSC ERROR: --------------------- Error Message ------------------------------------
>>>> >> [15]PETSC ERROR: Signal received!
>>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>>>> >> [14]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> >> [14]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> >> [14]PETSC ERROR: See docs/index.html for manual pages.
>>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [14]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>>>> >> [14]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>> >> [14]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>> >> [14]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>>>> >> [14]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [14]TSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
>>>> >> [15]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>> >> [15]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>> >> [15]PETSC ERROR: See docs/index.html for manual pages.
>>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: /home/becker/ansp6066/local/bin/finddme on a linux-gnu named photon9.colorado.edu by ansp6066 Fri Apr 27 18:01:55 2012
>>>> >> [15]PETSC ERROR: Libraries linked from /home/becker/ansp6066/local/petsc-3.2-p6/lib
>>>> >> [15]PETSC ERROR: Configure run at Mon Feb 27 11:17:14 2012
>>>> >> [15]PETSC ERROR: Configure options --prefix=/home/becker/ansp6066/local/petsc-3.2-p6 --with-c++-support --with-fortran --with-mpi-dir=/usr/local/mpich2 --with-shared-libraries=0 --with-scalar-type=complex --with-blas-lapack-libs=/central/intel/mkl/lib/em64t/libmkl_core.a --with-clanguage=cxx
>>>> >> [15]PETSC ERROR: ------------------------------------------------------------------------
>>>> >> [15]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>>>> >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15[0]0:Return code = 0, signaled with Interrupt
>>>> >
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120428/5800d63c/attachment-0001.htm>

From behzad.baghapour at gmail.com  Mon Apr 30 02:09:32 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Mon, 30 Apr 2012 11:39:32 +0430
Subject: [petsc-users] How to skip checking in SNES
Message-ID: <CA+jOtD+2xGZxAkG1bSThUwFthGGceZ_D9efS2y912+JO0V8f2A@mail.gmail.com>

Dear All,

I want to turn off checking the convergence in SNES actually when the KSP
did not convergence due to low number of subspaces. I did the following:

PetscErrorCode solver::SNESConvTest( SNES snes, PetscInt it, PetscReal xn,
PetscReal gn, PetscReal f, SNESConvergedReason* res, void* ctx)
{
    *res =  SNES_CONVERGED_ITERATING;
    PetscFunctionReturn( 0 );
}

but I still receives the stopping alert:

Linear solve did not converge due to DIVERGED_ITS iterations 30
Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE

Then what is the proper way?

Regards,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/f43c412f/attachment.htm>

From knepley at gmail.com  Mon Apr 30 05:08:01 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Apr 2012 06:08:01 -0400
Subject: [petsc-users] How to skip checking in SNES
In-Reply-To: <CA+jOtD+2xGZxAkG1bSThUwFthGGceZ_D9efS2y912+JO0V8f2A@mail.gmail.com>
References: <CA+jOtD+2xGZxAkG1bSThUwFthGGceZ_D9efS2y912+JO0V8f2A@mail.gmail.com>
Message-ID: <CAMYG4Gnu4A3OTY7XjcSbNAG2yOJoQzmfSCXXBFCnCyPTU17o=A@mail.gmail.com>

On Mon, Apr 30, 2012 at 3:09 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Dear All,
>
> I want to turn off checking the convergence in SNES actually when the KSP
> did not convergence due to low number of subspaces. I did the following:
>
> PetscErrorCode solver::SNESConvTest( SNES snes, PetscInt it, PetscReal xn,
> PetscReal gn, PetscReal f, SNESConvergedReason* res, void* ctx)
> {
>     *res =  SNES_CONVERGED_ITERATING;
>     PetscFunctionReturn( 0 );
> }
>
> but I still receives the stopping alert:\
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESSetMaxLinearSolveFailures.html

   Matt


> Linear solve did not converge due to DIVERGED_ITS iterations 30
> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE
>
> Then what is the proper way?
>
> Regards,
> BehZad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/a2a958a7/attachment.htm>

From behzad.baghapour at gmail.com  Mon Apr 30 06:10:27 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Mon, 30 Apr 2012 15:40:27 +0430
Subject: [petsc-users] How to skip checking in SNES
In-Reply-To: <CAMYG4Gnu4A3OTY7XjcSbNAG2yOJoQzmfSCXXBFCnCyPTU17o=A@mail.gmail.com>
References: <CA+jOtD+2xGZxAkG1bSThUwFthGGceZ_D9efS2y912+JO0V8f2A@mail.gmail.com>
	<CAMYG4Gnu4A3OTY7XjcSbNAG2yOJoQzmfSCXXBFCnCyPTU17o=A@mail.gmail.com>
Message-ID: <CA+jOtDKuLUFY_mo4WLOD+E-3iex=9+BwqKorXsrw5v1EnKuLLA@mail.gmail.com>

It did work. Thanks a lot...

However, I am really confused with "convergence reasons" in order to
control KSP or SNES. Is there any note or example which I can follow to
understand?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/b4ebfb99/attachment.htm>

From knepley at gmail.com  Mon Apr 30 09:38:55 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Apr 2012 10:38:55 -0400
Subject: [petsc-users] How to skip checking in SNES
In-Reply-To: <CA+jOtDKuLUFY_mo4WLOD+E-3iex=9+BwqKorXsrw5v1EnKuLLA@mail.gmail.com>
References: <CA+jOtD+2xGZxAkG1bSThUwFthGGceZ_D9efS2y912+JO0V8f2A@mail.gmail.com>
	<CAMYG4Gnu4A3OTY7XjcSbNAG2yOJoQzmfSCXXBFCnCyPTU17o=A@mail.gmail.com>
	<CA+jOtDKuLUFY_mo4WLOD+E-3iex=9+BwqKorXsrw5v1EnKuLLA@mail.gmail.com>
Message-ID: <CAMYG4G=Jq5B+3wwCN0WaXXQmYXDNBWm=4eJmDFbGVC4+QM1Uyw@mail.gmail.com>

On Mon, Apr 30, 2012 at 7:10 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> It did work. Thanks a lot...
>
> However, I am really confused with "convergence reasons" in order to
> control KSP or SNES. Is there any note or example which I can follow to
> understand?
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/SNES/SNESConvergedReason.html

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/e97c8da5/attachment.htm>

From hzhang at mcs.anl.gov  Mon Apr 30 10:04:33 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Mon, 30 Apr 2012 10:04:33 -0500
Subject: [petsc-users] mumps solve with same nonzero pattern(Hong Zhang)
In-Reply-To: <CAMJxm+AQVw5m_pSMPjFOavn2xe4RS+XHvczOMBvQQRAV875-0w@mail.gmail.com>
References: <CAMJxm+AQVw5m_pSMPjFOavn2xe4RS+XHvczOMBvQQRAV875-0w@mail.gmail.com>
Message-ID: <CAGCphBvZiQXcCGbhvTNRMayvZo2VRKzZgMwKfjhjCD+8XTDtrA@mail.gmail.com>

Wen :

> Reply to
>
>> This is weird.  Try
>> 1) increase work  space with
>> -mat_mumps_icntl_14 50 (default is 20)
>> 2) different matrix orderings with
>> -mat_mumps_icntl_7 2 (or number from 0 to 6)
>>
>> Run your code with '-log_summary' and see which routine causes this huge
>> difference.
>>
> Why your '-log_summary' only gives
KSPSolve               4 1.0 2.2645e+03 1.0 0.00e+00 0.0 3.9e+04 3.6e+02
5.4e+01 96  0 27  0  9  96  0 27  0  9     0
PCSetUp                4 1.0 2.2633e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
3.4e+01 96  0  0  0  6  96  0  0  0  6     0
PCApply                4 1.0 1.1641e+00 1.0 0.00e+00 0.0 3.9e+04 3.6e+02
2.0e+01  0  0 27  0  3   0  0 27  0  3     0

I get
petsc-dev/src/ksp/ksp/examples/tutorials>mpiexec -n 2 ./ex2 -pc_type lu
-pc_factor_mat_solver_package mumps -log_summary
MatMult                2 1.0 1.6904e-04 1.0 4.44e+02 1.0 4.0e+00 5.6e+01
0.0e+00  0 47 25 13  0   0 47 33 13  0     5
MatSolve               2 1.0 3.8259e-03 1.0 0.00e+00 0.0 8.0e+00 1.9e+02
6.0e+00 10  0 50 84  7  11  0 67 87  9     0
MatLUFactorSym         1 1.0 2.9058e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
8.0e+00  7  0  0  0  9   8  0  0  0 11     0
MatLUFactorNum         1 1.0 2.0120e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
2.0e+00  5  0  0  0  2   6  0  0  0  3     0
...

I like to check these functions. In addition, have you tried other matrix
orderings?
Hong

>
>> Hong
>>
>>
> I just tested the problem according to what you suggested. I set icntl_14
> = 50 and icntl_7 = 5 (METIS). The problem still persisted. The first solve
> took 920 second and second solve took 215 second with same nonzero pattern
> pc set up. I also attached the log_summary output file. Do you have any
> further suggestion?  Thanks.
>
> Regards,
> Wen
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/d6f5b204/attachment.htm>

From jiangwen84 at gmail.com  Mon Apr 30 12:18:19 2012
From: jiangwen84 at gmail.com (Wen Jiang)
Date: Mon, 30 Apr 2012 13:18:19 -0400
Subject: [petsc-users] mumps solve with same nonzero pattern
Message-ID: <CAMJxm+DozmA0XKVqaQQGo6HEOuwjBJhb89TQr8YCHPXdEC2LWA@mail.gmail.com>

Hi Hong,

Thanks for your reply. I check the attachment sent last time and I miss
some very information. I attached the complete info with this email. Sorry
for this.

I also tried other matrix ordering like mat_mumps_icntl_7 2 and I got the
similar performance. I checked the configuration log file on that cluster,
they use --download-f-blas-lapack=1 instead of using optimal BLAS. Would
this be the problem which cause the poor performance? Actually the mumps
runs quite slow, at14 Gflop/s,  which is far from the machine's peak.

Thanks.
Wen

>
> Wen :
>
> > Reply to
> >
> >> This is weird.  Try
> >> 1) increase work  space with
> >> -mat_mumps_icntl_14 50 (default is 20)
> >> 2) different matrix orderings with
> >> -mat_mumps_icntl_7 2 (or number from 0 to 6)
> >>
> >> Run your code with '-log_summary' and see which routine causes this huge
> >> difference.
> >>
> > Why your '-log_summary' only gives
> KSPSolve               4 1.0 2.2645e+03 1.0 0.00e+00 0.0 3.9e+04 3.6e+02
> 5.4e+01 96  0 27  0  9  96  0 27  0  9     0
> PCSetUp                4 1.0 2.2633e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 3.4e+01 96  0  0  0  6  96  0  0  0  6     0
> PCApply                4 1.0 1.1641e+00 1.0 0.00e+00 0.0 3.9e+04 3.6e+02
> 2.0e+01  0  0 27  0  3   0  0 27  0  3     0
>
> I get
> petsc-dev/src/ksp/ksp/examples/tutorials>mpiexec -n 2 ./ex2 -pc_type lu
> -pc_factor_mat_solver_package mumps -log_summary
> MatMult                2 1.0 1.6904e-04 1.0 4.44e+02 1.0 4.0e+00 5.6e+01
> 0.0e+00  0 47 25 13  0   0 47 33 13  0     5
> MatSolve               2 1.0 3.8259e-03 1.0 0.00e+00 0.0 8.0e+00 1.9e+02
> 6.0e+00 10  0 50 84  7  11  0 67 87  9     0
> MatLUFactorSym         1 1.0 2.9058e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 8.0e+00  7  0  0  0  9   8  0  0  0 11     0
> MatLUFactorNum         1 1.0 2.0120e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 2.0e+00  5  0  0  0  2   6  0  0  0  3     0
> ...
>
> I like to check these functions. In addition, have you tried other matrix
> orderings?
> Hong
>
> >
> >> Hong
> >>
> >>
> > I just tested the problem according to what you suggested. I set icntl_14
> > = 50 and icntl_7 = 5 (METIS). The problem still persisted. The first
> solve
> > took 920 second and second solve took 215 second with same nonzero
> pattern
> > pc set up. I also attached the log_summary output file. Do you have any
> > further suggestion?  Thanks.
> >
> > Regards,
> > Wen
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/a7d3d921/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_summary_info
Type: application/octet-stream
Size: 18496 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/a7d3d921/attachment.obj>

From mirzadeh at gmail.com  Mon Apr 30 18:01:31 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 30 Apr 2012 16:01:31 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
	<CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
	<CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>
Message-ID: <CAKsPKw=895e54Xyuy1A1dQW1bHkAn0_WB5jF6amTBQFdvjuN1A@mail.gmail.com>

Hi,

Just a quick question. Following
petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c, it seems that I need to
call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation to be
able to preallocate for both MPI and Seq matrices. Does petsc automatically
chose the relevant function when the code is run in serial and parallel? In
other words, what is the effect
of MatMPIAIJSetPreallocation(MatSeqAIJSetPreallocation) when the code is
run in serial(parallel)?

I like how several functions are abstract and can be used both in serial
and parallel (like MatCreate). Is there a similar way to just call a single
"MatSetPreallocation" function?

Thanks,
Mohammad

On Wed, Apr 25, 2012 at 4:04 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Thanks Hong; that fixed the problem.
>
>
> On Wed, Apr 25, 2012 at 11:31 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>
>> Mohammad:
>>
>>>
>>> MatCreate(comm, &A);
>>> MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
>>> globalColumnSize);
>>> MatSetType(A, MATMPIAIJ);
>>> MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
>>> MatSetFromOptions(A);
>>> MatGetOwnershipRange(A, &rStart, &rEnd);
>>>
>>>
>>> This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
>>> dev. The only difference I can see is 1) the order of MatSetFromOptions and
>>> 2) I do not call MatSeqAIJSetPreallocation which I think I do not need
>>> anyway. Is there something I'm doing wrong?
>>>
>>
>> MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
>> If user set mattype at runtime, MatSetFromOptions() picks it and set the
>> type accordingly. SetPreallocation()
>> will be called after the type is set.
>>
>> Hong
>>
>>>
>>> Mohammd
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/f09c61f5/attachment-0001.htm>

From andrew.spott at gmail.com  Mon Apr 30 18:49:42 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 30 Apr 2012 17:49:42 -0600
Subject: [petsc-users] Solving a very simple time step problem:
Message-ID: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>

I want to solve a very simple equation:

u_t = F(t) u

Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t) is a time dependent scalar).

But I'm not sure how to go about doing this using the TS context.

I don't have a Jacobian that I need to be worried about, so should I be doing:

TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);

Where:
myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
{
	//Create temporary matrix A = H_0 + a(t) H'
	//then do F = A u
}

Or should I be doing something else?

Thanks for the help, unfortunately, it looks like the documentation on TS in the manual isn't accurate.

-Andrew

From knepley at gmail.com  Mon Apr 30 18:57:20 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 30 Apr 2012 19:57:20 -0400
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAKsPKw=895e54Xyuy1A1dQW1bHkAn0_WB5jF6amTBQFdvjuN1A@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
	<CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
	<CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>
	<CAKsPKw=895e54Xyuy1A1dQW1bHkAn0_WB5jF6amTBQFdvjuN1A@mail.gmail.com>
Message-ID: <CAMYG4Gk0UFP3RtrnayvZxJd5jgXR-hZ+9u=VafKTxai-bFPw6Q@mail.gmail.com>

On Mon, Apr 30, 2012 at 7:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi,
>
> Just a quick question. Following
> petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c, it seems that I need to
> call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation to be
> able to preallocate for both MPI and Seq matrices. Does petsc automatically
> chose the relevant function when the code is run in serial and parallel? In
> other words, what is the effect
> of MatMPIAIJSetPreallocation(MatSeqAIJSetPreallocation) when the code is
> run in serial(parallel)?
>
> I like how several functions are abstract and can be used both in serial
> and parallel (like MatCreate). Is there a similar way to just call a single
> "MatSetPreallocation" function?
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatXAIJSetPreallocation.html

   Matt


> Thanks,
> Mohammad
>
> On Wed, Apr 25, 2012 at 4:04 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Thanks Hong; that fixed the problem.
>>
>>
>> On Wed, Apr 25, 2012 at 11:31 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>>
>>> Mohammad:
>>>
>>>>
>>>> MatCreate(comm, &A);
>>>> MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
>>>> globalColumnSize);
>>>> MatSetType(A, MATMPIAIJ);
>>>> MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
>>>> MatSetFromOptions(A);
>>>> MatGetOwnershipRange(A, &rStart, &rEnd);
>>>>
>>>>
>>>> This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
>>>> dev. The only difference I can see is 1) the order of MatSetFromOptions and
>>>> 2) I do not call MatSeqAIJSetPreallocation which I think I do not need
>>>> anyway. Is there something I'm doing wrong?
>>>>
>>>
>>> MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
>>> If user set mattype at runtime, MatSetFromOptions() picks it and set the
>>> type accordingly. SetPreallocation()
>>> will be called after the type is set.
>>>
>>> Hong
>>>
>>>>
>>>> Mohammd
>>>>
>>>
>>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/954c2b12/attachment.htm>

From bsmith at mcs.anl.gov  Mon Apr 30 20:05:32 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 30 Apr 2012 20:05:32 -0500
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <CAMYG4Gk0UFP3RtrnayvZxJd5jgXR-hZ+9u=VafKTxai-bFPw6Q@mail.gmail.com>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
	<CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
	<CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>
	<CAKsPKw=895e54Xyuy1A1dQW1bHkAn0_WB5jF6amTBQFdvjuN1A@mail.gmail.com>
	<CAMYG4Gk0UFP3RtrnayvZxJd5jgXR-hZ+9u=VafKTxai-bFPw6Q@mail.gmail.com>
Message-ID: <64C9EAE0-84FC-4488-89E9-7D22AD58778C@mcs.anl.gov>


On Apr 30, 2012, at 6:57 PM, Matthew Knepley wrote:

> On Mon, Apr 30, 2012 at 7:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:
> Hi,
> 
> Just a quick question. Following petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c, it seems that I need to call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation to be able to preallocate for both MPI and Seq matrices. Does petsc automatically chose the relevant function when the code is run in serial and parallel?

    Yes, it uses the relevant one and ignores any not relevant. 

    This is a common trick in PETSc. You can think of the calls as methods specific to a particular subclass of the Mat class. PETSc automatically uses all the methods that are appropriate for the particular subclass and ignores all the other ones.

    Barry



> In other words, what is the effect of MatMPIAIJSetPreallocation(MatSeqAIJSetPreallocation) when the code is run in serial(parallel)? 
> 
> I like how several functions are abstract and can be used both in serial and parallel (like MatCreate). Is there a similar way to just call a single "MatSetPreallocation" function?
> 
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatXAIJSetPreallocation.html
> 
>    Matt
>  
> Thanks,
> Mohammad
> 
> On Wed, Apr 25, 2012 at 4:04 PM, Mohammad Mirzadeh <mirzadeh at gmail.com> wrote:
> Thanks Hong; that fixed the problem.
> 
> 
> On Wed, Apr 25, 2012 at 11:31 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
> Mohammad:
> 
> MatCreate(comm, &A);
> MatSetSizes(A, localRowSize, localColumnSize, globalRowSize, globalColumnSize);
> MatSetType(A, MATMPIAIJ);
> MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
> MatSetFromOptions(A);
> MatGetOwnershipRange(A, &rStart, &rEnd);
> 
> 
> This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not dev. The only difference I can see is 1) the order of MatSetFromOptions and 2) I do not call MatSeqAIJSetPreallocation which I think I do not need anyway. Is there something I'm doing wrong?
>  
> MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
> If user set mattype at runtime, MatSetFromOptions() picks it and set the type accordingly. SetPreallocation()
> will be called after the type is set.
> 
> Hong
> 
> Mohammd
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From mirzadeh at gmail.com  Mon Apr 30 22:05:18 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Mon, 30 Apr 2012 20:05:18 -0700
Subject: [petsc-users] error in petsc-dev
In-Reply-To: <64C9EAE0-84FC-4488-89E9-7D22AD58778C@mcs.anl.gov>
References: <CAKsPKwkC1UB6MWCJHG8YfaDftVYYo7VX_AaYiuEBJwe05b0seQ@mail.gmail.com>
	<CAGCphBt+JzKzYrf9CdwybkK1kLQK4OGr6sxz53s2i4K4MdCRfA@mail.gmail.com>
	<CAKsPKw=BxqixbLM26O+rWJhKHnTYpsuxE_BvVXBG7gmOORUiyA@mail.gmail.com>
	<CAKsPKwkF-x6=EgPpgxmb2zrws0-Q74Sr_wHj9mL24h3Jqg1sqA@mail.gmail.com>
	<CAGCphBuJf3s9gCEteub_+7-DRTHGCGEjAuuUDdt0nXQCS61LAQ@mail.gmail.com>
	<CAKsPKwnKSZZSQYb3Q07p8C43zsSeUFi8oS1pcgcRpbiFeCSLCw@mail.gmail.com>
	<CAGCphBs8tiNWh5ZD0cAWMyd0ybUjDmnCSu-BWJ2Rp+3FN6fUKg@mail.gmail.com>
	<CAKsPKwk8RitZMu54ma+iHN=HfctBFkHsK-=Ju_p-H87pR=2aOA@mail.gmail.com>
	<CAGCphBss8X_FLyDzmyxJDyor8Ezf=rJ7+k3+t24vRwGS8DvwSg@mail.gmail.com>
	<CAKsPKwn8tioFAeEKm_aw7gXk3rw=PArm+YcLBdhHg4Q=esUknw@mail.gmail.com>
	<CAKsPKw=895e54Xyuy1A1dQW1bHkAn0_WB5jF6amTBQFdvjuN1A@mail.gmail.com>
	<CAMYG4Gk0UFP3RtrnayvZxJd5jgXR-hZ+9u=VafKTxai-bFPw6Q@mail.gmail.com>
	<64C9EAE0-84FC-4488-89E9-7D22AD58778C@mcs.anl.gov>
Message-ID: <CAKsPKwkRJF9APJrEQq15AFfT9vVXHMmiQ+=m0z7sS68sdSLEVw@mail.gmail.com>

Barry, Matt,

Thank you both.

Mohammad

On Mon, Apr 30, 2012 at 6:05 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Apr 30, 2012, at 6:57 PM, Matthew Knepley wrote:
>
> > On Mon, Apr 30, 2012 at 7:01 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
> > Hi,
> >
> > Just a quick question. Following
> petsc-dev/src/ksp/ksp/examples/tutorials/ex2.c, it seems that I need to
> call both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation to be
> able to preallocate for both MPI and Seq matrices. Does petsc automatically
> chose the relevant function when the code is run in serial and parallel?
>
>     Yes, it uses the relevant one and ignores any not relevant.
>
>    This is a common trick in PETSc. You can think of the calls as methods
> specific to a particular subclass of the Mat class. PETSc automatically
> uses all the methods that are appropriate for the particular subclass and
> ignores all the other ones.
>
>    Barry
>
>
>
> > In other words, what is the effect of
> MatMPIAIJSetPreallocation(MatSeqAIJSetPreallocation) when the code is run
> in serial(parallel)?
> >
> > I like how several functions are abstract and can be used both in serial
> and parallel (like MatCreate). Is there a similar way to just call a single
> "MatSetPreallocation" function?
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatXAIJSetPreallocation.html
> >
> >    Matt
> >
> > Thanks,
> > Mohammad
> >
> > On Wed, Apr 25, 2012 at 4:04 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
> > Thanks Hong; that fixed the problem.
> >
> >
> > On Wed, Apr 25, 2012 at 11:31 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
> > Mohammad:
> >
> > MatCreate(comm, &A);
> > MatSetSizes(A, localRowSize, localColumnSize, globalRowSize,
> globalColumnSize);
> > MatSetType(A, MATMPIAIJ);
> > MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz);
> > MatSetFromOptions(A);
> > MatGetOwnershipRange(A, &rStart, &rEnd);
> >
> >
> > This (even without MatSetType(A, MATMPIAIJ);) works with 3.2-p6 but not
> dev. The only difference I can see is 1) the order of MatSetFromOptions and
> 2) I do not call MatSeqAIJSetPreallocation which I think I do not need
> anyway. Is there something I'm doing wrong?
> >
> > MatSetFromOptions() must be called before MatMPIAIJSetPreallocation().
> > If user set mattype at runtime, MatSetFromOptions() picks it and set the
> type accordingly. SetPreallocation()
> > will be called after the type is set.
> >
> > Hong
> >
> > Mohammd
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120430/d97f0d2a/attachment.htm>